cs.CL Archives

ReplicationBench evaluates AI agents on end-to-end astrophysics research paper replication. For each paper, we supply the manuscript, dataset, and execution metadata, and a complete set of author-written tasks. We evaluate LLM agents in multi-turn code execution environments and conduct both automatic and expert human grading of traces. -- cs.CL

AI - Data - Apps - Cybernetics, Citizen Science / Open Science, Education and Outreach, Genomics, Proteomics, Bioinformatics, Mapping, Geodesy, Cartography, Bathymetry, Nomenclature, Systematics, Natural History, Status Report

ReplicationBench: Can AI Agents Replicate Astrophysics Research Papers?

by Keith CowingOctober 31, 2025October 31, 2025

Frontier AI agents show increasing promise as scientific research assistants, and may eventually be useful for extended, open-ended research workflows.

Left: The original Arecibo message intended to be reconstructed, but sent as a linear stream from the radiotelescope in Arecibo, Puerto Rico. The 1,679 bits are meant to be arranged into 23 columns of 73 rows, 23 and 73 being two prime numbers which when multiplied together equal 1,679. Right: If the stream is instead arranged into 23 rows and 73 columns, the original visual interpretations of the message are scrambled, which may result in a figure that is closer to being statistically random. What we show is that the message is still there, concealed, and can be deciphered by algorithmic deconvolution. -- cs.CL

AI - Data - Apps - Cybernetics, Astronomy & Telescopes, Intelligence & Language, Interstellar, SETI & Technosignatures, Status Report

Decoding Geometric Properties in Non-Random Data from First Information-Theoretic Principles

by Keith CowingNovember 29, 2024November 29, 2024

Based on the principles of information theory, measure theory, and theoretical computer science, we introduce a univariate signal deconvolution method with a wide range of applications to coding theory, particularly […]

Status Report

Tricorder Tech: A Highly Capable Language Model Locally On Your Phone

by Keith CowingJuly 1, 2024June 27, 2026

Editor’s note: you can never seem to have too much access to data when you are on an Away Team out in the middle of nowhere. The further afield you […]