Coding-agents can replicate scientific machine learning papers

Abstract: Scientific machine learning papers typically make computational claims, e.g., that the relative mean square error is less than 5% or that the 95% predictive credible interval covers the test data. A coding agent can be prompted to replicate those claims from paper materials alone, but the prompt does not by itself reliably preserve progress or check whether generated evidence supports the paper's claims. We introduce Paper-replication, a workflow that makes each selected paper claim a target with recorded evidence, and implement it as a coding-agent skill. The workflow makes the agent record those targets, reconstruct the paper's method, run computational experiments, link generated outputs to provenance and comparisons with the paper's claims, record where matched evidence appears in the replication report, and pass validation checks before completion. We evaluate Paper-replication on twelve independent runs across four scientific machine learning papers. All twelve workspaces pass the completion gate, and all 158 recorded targets are matched with report coverage. Even in this completed workspace state, repeated runs differ in how papers are divided into targets, in numerical fidelity to the source papers, in elapsed replication time, in the number of intermediate executions replaced before final evidence is accepted, and in the rules used to accept evidence. Paper-replication makes completion depend on workspace evidence and validation checks rather than on the agent's final message.

• The paper presents a novel replication protocol that rigorously decomposes and validates computational claims from ML papers using traceable evidence bundles.
• It employs persistent workspaces and manifest records to track and verify method reconstruction, code execution, and experimental provenance.
• The evaluation confirms 95% numeric fidelity across replicates while revealing variability in target decomposition and acceptance criteria.

Coding Agents for Scientific Paper Replication: A Technical Analysis of Paper-Replication

Introduction and Background

The paper "Coding-agents can replicate scientific machine learning papers" (2607.02134) introduces a workflow—Paper-replication—designed to rigorously evaluate the ability of coding agents to replicate computational claims from scientific machine learning papers, starting solely from the paper and referenced data. The work offers a granular framework for decomposition of paper-level claims, validation of agent-generated evidence, and persistent tracking of progress within a structured workspace. This approach is motivated by the persistent challenges in scientific reproducibility, including the lack of code, unspecified randomness, stochastic procedure sensitivity, and underreporting of empirical details.

Paper-replication departs from conventional code-generation or result-matching protocols by making each selected scientific claim a "target" whose acceptance requires explicit evidence: method reconstruction, runnable code, provenance, validation checks, and report coverage. The process is operationalized as a coding-agent skill with persistent state and external validation, rather than relying solely on prompt instructions or agent self-report.

Paper-Replication Workflow and Implementation

Central to the Paper-replication workflow is the target-evidence contract. Each computational claim selected for replication (target) is accompanied by a requirement for traceable, multi-faceted evidence, including:

• The agent's precise reconstruction of the method section relevant to each target.
• Provenance for each experiment execution, including configuration, code, seeds, and references to paper content.
• Comparison evidence using claim-specific acceptance rules (scalar, structural, distributional, or visual).
• Documentation of inclusion in the formal replication report.

Completion is strictly defined through validation of the above evidence bundles and is a state of the workspace, not an agent-internal assertion.

The workflow introduces persistent workspace records:

• A manifest encoding paper materials, hashes, environment, and policy constraints (e.g., prohibiting author code).
• Reproduction matrices with explicit status and dependency tracking for each target.
• Intermediate experiment and provenance logs, recording all attempted, failed, or superseded executions.

Validation consists of multiple layers: hash-based checks to prevent undetected reuse of paper assets, consistency checks between workspace targets and generated outputs, and enforcement of target-specific acceptance rules. Only when all targets have accepted evidence and the formal report includes all matched work does Paper-replication consider the process complete.

Case Study Evaluation and Key Results

The evaluation applies Paper-replication to twelve independent replications (with three repeated runs per paper) on four diverse machine learning papers: PIFT, PINN-I, PINN-II, and SINDy. Each run starts from identical materials but proceeds independently, leveraging GPT-5.4 Codex as the coding agent and a skill implementation enforcing the Paper-replication protocol.

A summary of per-run outcomes, including elapsed time, final target count, and tracked execution paths, demonstrates the persistent, inspectable, and variable characteristics of recorded evidence.

Figure 1: Per-run workspace evidence for the twelve case-study runs, highlighting variance in effort, decomposition, and correction paths while maintaining completion consistency.

Target Decomposition and Variation

Analysis of the final target counts reveals that while some papers (PINN-I, SINDy) have stable decomposition across runs, others (PIFT, PINN-II) display substantial run-to-run differences in how claims are split into discrete targets. The Gamma-Poisson model quantifies these differences. The decomposition ratios reach as high as 3.1 for PIFT.

Figure 2: Target coverage analysis demonstrating per-paper decomposition ratios and between-run variability, especially for PIFT and PINN-II.

Numeric Fidelity and Acceptance Rules

The scalar analysis, based on thirteen numeric anchors (e.g., error metrics, coefficient errors) with paper-specified thresholds, shows 95% of replications matching within the pre-established criteria. Variations up to an order of magnitude are observed among runs on individual targets due to stochastic and implementation sensitivities rather than failure of the workflow. Importantly, targets are matched according to the agent's acceptance rule, which is claim-specific and sometimes less strict than standardized thresholds enforced in a separate analysis.

Figure 3: Distribution of scalar discrepancies relative to paper values, per-paper headroom below threshold, and residual run-to-run variation illustrating variability in replicated numeric precision.

Effort, Correction, and Judgment Variation

Elapsed replication time varies across and within papers, with PINN subtasks being the most labor-intensive, likely due to repeated optimization and greater correction overhead. Correction work (superseded executions) predominates in complex PINN replications, reflecting the iterative nature of hypothesis testing and adjustment required for successful evidence matching.

Figure 4: (a) Posterior distribution over replication effort; (b) Correction load, highlighting the dominant role of iterative correction in the PINN cases.

Acceptance rule classification for targets diverges across runs, especially in the PINN papers, where identical claims are sometimes judged with scalar and elsewhere with structural acceptance. This underscores the subjective elements introduced even under a formalized workflow.

Figure 5: Agreement rates in acceptance rule type selection across repeated runs for each paper, with lowest consensus in PINN-II.

Discussion and Implications

The formalization of paper replication as a workspace-anchored, target-level evidence task enabled observable, quantitatively tractable, and auditable replications of complex scientific machine learning claims. The results highlight not only strong completion rates but also variability in decomposition, numeric fidelity, and judgment. These findings challenge any simplistic use of "reproducibility" as a binary metric and underline the necessity for persistent records that capture process heterogeneity, correction histories, and acceptance rule subjectivity.

Paper-replication's approach directly addresses reward hacking and ambiguous standards cited in recent discussions on reproducibility [manheim2018categorizing, weng2024rewardhacking]. By explicit separation of specification, execution, provenance, and acceptance, the protocol reduces both agent self-delusion and over-optimistic interpretation of plausible-looking outputs.

Beyond replicability, this work signals a direction for automated evaluation and auditing of scientific claims, providing a substrate for reproducibility benchmarking and meta-research on agentic science. The agent's ability to recover, hypothesize, and test missing details positions coding agents as both research assistants and audit instruments for open science infrastructures.

Future developments may expand in several dimensions:

• Broader evaluation across more scientific domains and paper archetypes, increasing statistical power for comparative or diagnostic studies.
• Integration with improved LLM agentic memory and cross-episode tracking, closing gaps in long-horizon reasoning and hypothesis updating.
• Development of standardized, extensible acceptance-rule taxonomies and objective thresholding to further codify subtle claims.
• Automated benchmarking for the effect of prompt structure, workflow skill ablations, and compute resource variability on reproducibility outcomes.

Conclusion

Paper-replication advances the rigor and inspectability of coding-agent scientific paper replication by architecting workflows in which computational claims are decomposed into explicit targets, each requiring traceable, validated evidence and comprehensive provenance. Empirical analysis demonstrates that while the workflow robustly drives completion, variation in process and fidelity persists, fundamentally shaped by both agentic interpretation and the indeterminacy of the original papers. The study advocates for systematic recording of replication workflows and sets a foundation for auditable, reproducible scientific computation mediated by advanced coding agents.