Sanity Checks for Agentic Data Science

Abstract: Agentic data science (ADS) pipelines have grown rapidly in both capability and adoption, with systems such as OpenAI Codex now able to directly analyze datasets and produce answers to statistical questions. However, these systems can reach falsely optimistic conclusions that are difficult for users to detect. To address this, we propose a pair of lightweight sanity checks grounded in the Predictability-Computability-Stability (PCS) framework for veridical data science. These checks use reasonable perturbations to screen whether an agent can reliably distinguish signal from noise, acting as a falsifiability constraint that can expose affirmative conclusions as unsupported. Together, the two checks characterize the trustworthiness of an ADS output, e.g. whether it has found stable signal, is responding to noise, or is sensitive to incidental aspects of the input. We validate the approach on synthetic data with controlled signal-to-noise ratios, confirming that the sanity checks track ground-truth signal strength. We then demonstrate the checks on 11 real-world datasets using OpenAI Codex, characterizing the trustworthiness of each conclusion and finding that in 6 of the datasets an affirmative conclusion is not well-supported, even though a single ADS run may support one. We further analyze failure modes of ADS systems and find that ADS self-reported confidence is poorly calibrated to the empirical stability of its conclusions.

• The paper introduces lightweight, PCS-grounded consistency checks to systematically evaluate ADS outputs under controlled perturbations.
• The paper demonstrates that agentic systems often yield unstable conclusions due to brittle data cleaning, model selection biases, and poor calibration.
• The paper's empirical results on synthetic and real datasets emphasize the urgent need for routine falsifiability and stability checks in autonomous data analysis.

Sanity Checks for Agentic Data Science: A Technical Review

Motivation and Problem Statement

The proliferation of Agentic Data Science (ADS) systems, notably those leveraging powerful LLMs such as OpenAI Codex, has enabled “end-to-end” automation of statistical analysis and inferential workflows. While these agentic systems lower barriers to entry and accelerate exploratory data analysis, the veridicality of their conclusions—how much they can be trusted to reliably distinguish signal from noise—remains underexplored. This work recognizes that ADS systems are highly prone to drawing falsely affirmative or unstable conclusions due to brittle data-cleaning, unsafe model selection, sycophancy to user intent, and statistical miscalibration. Standard approaches to uncertainty and significance testing are largely inadequate in these settings.

Methodology: PCS-Grounded Sanity Checks

This paper proposes lightweight, actionable consistency checks for ADS outputs grounded in the Predictability-Computability-Stability (PCS) framework. The central premise is that robust, reality-tested scientific inferences must be stable under a suite of reasonable perturbations across all stages of data science—problem specification, metadata, data, and model selection. Accordingly, two formally distinct types of perturbations are defined:

• PCS perturbations: Metadata and prompt-level modifications that should not affect a conclusion grounded in true signal (e.g., anonymizing/shuffling feature names, adding irrelevant covariates, introducing confirmatory/dissenting prompt statements).
• Null-defining perturbations: Structured randomizations (e.g., shuffling feature values) that decouple the outcome from all predictors, simulating a noise regime with no underlying association by construction.

ADS agents are then systematically evaluated on perturbed and unperturbed variants of the input, and their responses—elicited as integer-valued agreement scores on a 0 (strong “No”) to 100 (strong “Yes”) scale—are aggregated to form empirical distributions.

Two complementary sanity checks emerge:

1. Yes Check: A one-sided bootstrap hypothesis test to assess if the ADS’s mean response on the original dataset is significantly greater than neutral (50), controlling for the variance induced by PCS perturbations.
2. Overlap Check: Quantifies the statistical separation between the ADS response distribution on real vs. null data via the overlap coefficient (OVL). Low OVL implies the agent can distinguish signal from noise; high OVL indicates susceptibility to spurious associations.

These checks yield a two-dimensional classification that partitions agentic outputs into “trustworthy affirmations”, “data-ungrounded affirmations”, “weak evidence”, and “unsupported” regimes.

Experimental Evaluation and Empirical Findings

The authors conduct extensive validation using both synthetic and real datasets. On synthetic tabular data with systematically varied signal-to-noise ratios, the PCS sanity checks demonstrate high sensitivity to true effect size. As the proportion of variance explained (PVE) increases, the “yes” rate rises and OVL drops, confirming the theoretical mapping between signal strength and agentic veridicality.

On eleven real-world datasets from the BLADE benchmark, the method exposes critical instabilities:

• Only 5/11 datasets yield stable, data-grounded positive ADS conclusions (passing both checks).
• 5/11 datasets show no reliable signal, with high overlap and means below the affirmative threshold.
• In one case (caschools), the ADS consistently affirms the hypothesis (mean Likert 55.8, $p=0.002$), but with high distributional overlap (OVL=0.280). This exposes the danger of relying solely on point estimates or significance without cross-verification.
• A crucial finding is that single-run evaluations of an ADS pipeline can yield misleadingly optimistic results—a phenomenon directly traceable to pipeline instability and the compounding of upstream perturbations.

In ablation studies and finer-grained analysis using the BLADE harness, the conclusion step of the pipeline is identified as the most volatile to PCS perturbation, while independent variable selection remains robust. Sycophancy, confirmation bias (in response to leading statements), and model shopping/p-hacking are reproduced as failure modes.

Remarkably, the study finds poor calibration between agentic self-reported “confidence” and empirical outcome stability. Agents tend to hedge or overstate certainty, with confidence only weakly correlated with empirical exceedance, especially under null settings (Spearman $\rho=0.18$ under the null). This finding undermines the utility of introspective uncertainty estimation within LLM-based ADS.

Human vs. Agentic Analyst Variability

To contextualize ADS instability, agentic outputs are compared to human analyst responses on the canonical “soccer red cards” case (Silberzahn et al., 2018). Human analysts exhibit greater response variance, illustrating that substantial analytic multiplicity (the “garden of forking paths” problem) is a property of the data science task itself, not merely an LLM artifact. However, agents display more clustered bimodal scoring, related to the deterministic nature of their optimization procedures.

Theoretical and Practical Implications

Key claims:

• The PCS sanity checks empirically track ground-truth signal and robustly demarcate between veridical and spurious ADS conclusions across real and synthetic domains.
• Positive conclusions from cutting-edge ADS systems, including Codex, are frequently not well-supported, and single-run output is unreliable—highlighting an urgent need for routine empirical falsifiability and stability checks in any autonomous data analysis pipeline.
• Self-reported agentic confidence provides no substitute for external perturbation-based verification.
• Existing multiverse and classical significance tests, while informative, are either unscalable or insufficiently decisive for agentic pipelines.

The approach’s robustness is validated by bootstrap calibration studies; its flexibility allows practitioners to specify domain-adapted null-defining perturbations, yielding application-relevant significance thresholds.

Practically, this framework establishes a path toward operationalized evaluation standards for ADS deployment in science, regulatory, and high-stakes data analysis settings. It further enables automated, actionable filtering of agentic findings by their empirical trustworthiness.

Limitations and Future Directions

Main limitations arise from:

• Potential pretraining exposure of ADS models to benchmark datasets, and reliance on single or few LLM instantiations.
• Computational overhead imposed by multiple runs under perturbation, which may be alleviated as LLM costs decrease or via early-stopping techniques.
• Necessity for domain-specific null-defining perturbations that preserve realistic covariate structures.

Promising future research avenues include broader agent benchmarking, extension to hypothesis-generating or human-in-the-loop science tasks, and integration of the sanity check framework into fully autonomous scientific discovery pipelines.

Conclusion

This work presents a critical, technically rigorous assessment of ADS pipeline trustworthiness and introduces a practical, systematically validated methodology for empirically vetting agentic data science outputs. By anchoring veridicality in the Predictability-Computability-Stability paradigm and operationalizing it into routine, interpretable checks, the paper equips ADS practitioners and researchers with the necessary tools for falsifiability and stability in automated data analysis. The framework is both immediately actionable and extendable to future developments in agentic AI for science.