CausalDS: Benchmarking Causal Reasoning in Data-Science Agents

Abstract: LLMs increasingly act as integrated data-science agents, combining abstract reasoning with advanced tool use. Yet the relevant benchmark landscape largely divides into symbolic causal reasoning benchmarks without realistic data analysis or data analysis benchmarks without a principled causal data-generating structure. Furthermore, existing causal evaluation datasets are often restricted to curated examples from existing sources, with diversity coming from limited templatized variations rather than from systematic generation of novel synthetic causal structures. We introduce CausalDS, a benchmark for evaluating causal reasoning in agentic data-science workflows. Each benchmark instance is a scene consisting of a sampled structural causal model (SCM) with generated observational data and an accompanying synthetic natural-language story grounded in a realistic domain. We optionally ground the composition of the benchmark components in empirical distributions obtained from real-world datasets, thus retaining empirical structure while reducing the "causal parrot" risk through completely synthetic generation. From each scene, we then derive tasks spanning all three of Pearl's rungs, with typical data-science prediction tasks appearing as Rung 1. Most tasks include a data science coding component, where the model typically needs to use several tools to arrive at the final answer due to the frequent presence of imperfect observations, which are generated by an observation model. Additionally, recognizing when a question admits no warranted answer and abstaining is treated as a first-class scored outcome. The benchmark thus jointly evaluates symbolic causal reasoning, data science, uncertainty quantification, abstention, and tool use/coding.

• The paper presents a novel benchmark integrating symbolic causal reasoning with quantitative analysis to evaluate data-science agents.
• It employs synthetic SCMs with realistic measurement errors and abstention protocols to assess identification, effect estimation, and counterfactual reasoning.
• Empirical results reveal significant performance gaps in uncertainty calibration and tool use between frontier and open-weight models.

CausalDS: Benchmarking Causal Reasoning in Data-Science Agents

Motivation and Benchmark Design

The CausalDS benchmark fills a significant gap in the evaluation of LLM-based data-science agents by systematically testing all key axes required for practical causal data analysis: symbolic causal reasoning, quantitative data analysis, uncertainty quantification, abstention recognition (when estimation is not warranted), and robust tool use. Prior evaluations tend to address only narrow slices—symbolic graph reasoning divorced from data; data-analysis without a defined underlying causal structure; or curated real-world scenarios that risk semantic contamination (“causal parrot” failure). CausalDS instead constructs fully synthetic, yet empirically grounded, scenes: each instance comprises a hidden structural causal model (SCM) with generated observational data, natural-language domain stories, a realistic measurement layer (including noise and proxies of latent variables), and a broad task suite derived from the full Pearl hierarchy (associational, interventional, counterfactual).

Each scene includes:

• A sampled, potentially complex DAG with grounded variable names and partial seeding from CauseNet to amplify semantic realism.
• An SCM spanning both binary and continuous mechanisms, with parameterizations blended from empirical biochemical corpora and synthetic augmentations.
• An explicit separation between conceptual causal variables and the observation layer; the latter introduces bundled noisy measurements, exposing agents to realistic inference and identification stressors.
• A 3-axis task taxonomy spanning identification, effect estimation, prediction, counterfactual reasoning, graph recovery, and bias diagnostics—each with rigorously defined scoring rules and deterministic, abstention-sensitive evaluation protocols.

Synthetic Scene Generation and Observation Model

CausalDS scene generation forefronts both diversity and empirical grounding. Motif sampling (chain, fork, confounder, IV, etc.) is integrated with anchor-based grafting, producing composite graphs that mimic the combinatorial structures found in real applied work. Crucially, measurement error and partial observability are not limited to outcomes, but can target any node, leveraging bundle-based observation layers parameterized by Fisher information diagnostics to ensure task recoverability remains within tractable but nontrivial bounds.

Figure 2: Per-scene recoverability diagnostic for measurement bundles, showing the range of information available to the agent under proxy-noised observation variants.

The observation layer is distinguished from proximal identification frameworks (cf. [tchetgen_introduction_2024], [miao_identifying_2018])—no new identification routes are supplied by auxiliary measurements: the SCM is fixed and identifiability is always defined at the level of the conceptual DAG.

Free-Form Mapping and Rich Task Suite

Variable mapping and scenario narration are handled via LLM-augmented, auditor-controlled loops with iterative semantic and structural validation. Scene instantiation includes an LLM-generated story and file-backed dataset, which the agent then interacts with using a sandboxed execution environment (mini-swe-agent-style). Tasks are auto-derived across families and rungs: the same DAG supports prediction, association, causal effect estimation, identification queries, counterfactuals (e.g., ETT, NDE, NIE), and mediation, thus tightly coupling symbolic reasoning and code-based estimation.

Evaluation, Scoring, and Baselines

Evaluation is multi-axis and robust to the confounding of correctness, calibration, and abstention. Deterministic answer parsing and abstention-sensitive routing enforce that models are rewarded for both returning correct answers and appropriately abstaining on non-identifiable queries.

The composite CausalDSScore combines pass rate (pooled over abstention and content), normalized point and interval errors on continuous outputs, and $F_1$ losses on set/graph recovery, with cuts at each axis. Composite rank and pass@k/pess@k-like metrics quantify both single- and repeated-attempt reliability.

Empirical Results

The headline exam tested six models (three frontier closed, three large open-weight) on a 100-scene, empirically grounded composition. Explicit differences emerged:

• Symbolic graph recovery and identification are universally easy for all models.
• Performance splits appear downstream, at the intersection of epistemic axes and tool use:
  • Frontier models (Claude Opus 4.8, Gemini 3.1 Pro, GPT-5.5) dominate in abstention accuracy, uncertainty calibration, and tool-use economy.
  • Open models (Qwen 3.6 35B, Kimi K2.6, Gemma 4 26B) lag on abstention (often over-committing answers where abstention is warranted) and efficiency.
  • 

  • Figure 4: Tokens per task versus CausalDSScore across all evaluated models, illustrating the efficiency-quality frontier.

  • 

  • Figure 6: Pass Rate by rung and model, summarizing the dissociation of performance across Pearl's hierarchy.

• GPT-5.5 stands out as an example of discrepancy between pass rate and true quality: despite a high content pass rate, it exhibits the worst calibration (as measured by S_{NR}) among frontiers, reflecting high uncertainty miscalibration and catastrophic errors on difficult continuous tasks.
• Observation-layer hardness (proxy/proxy_hard) nonlinearly increases failure rates and error magnitudes, especially for open models, exposing a real-world relevant vulnerability to imperfect data.
• Abstention accuracy is the key separator; content accuracy saturates for most families, but the ability to declare a query non-identifiable when appropriate remains an advanced and sparsely acquired capability.
• Interval coverage on uncertainty tasks is universally overconfident: empirical $95\%$ confidence intervals cover as little as $20-71\%$ of true parameters, with only the best models showing some calibration improvement (cf. also [hobor_bayesian_2026], [epstein_llms_2025], [renda_openestimate_2025]).
• Repeated-attempt pass@k metrics reveal strong within-model stochasticity: open-weight models can occasionally match frontier one-shot rates across $k=3$ attempts, but reliability (pass^k) remains low, with most failures clustered in epistemic tasks (uncertainty/abstention).

Diagnostic Ablations

• Ablation across matched scenes with increasing measurement error demonstrates that estimation failures split between outright non-answers (due to context or tool limitations) and severe magnitude errors, especially on the hardest proxy_hard variant.
• Variable mapping audits indicate that model correctness is stable to variable renaming and narrative swapping for strong models, underscoring the resistance to surface-level contamination in this synthetic framework.

Contextualization With Prior Work

CausalDS integrates dimensions that have remained partitioned in previous benchmarks:

• Synthetic data-agent benchmarks (DSBench, BLADE, DiscoveryBench) lack explicit causal generative structure.
• Causal graph Q&A benchmarks (CLadder, CLEAR, CausalGraph2LLM, CausalBench) are purely symbolic, lacking file-backed data or measurement error.
• Real-world study benchmarks (CauSciBench, CausalReasoningBenchmark) are empirically rich but risk contamination and only rarely present non-identifiability or abstention targets.
• Interactive agent/experiment environments (CausaLab, CausalGame) focus on mechanism recovery, not data-backed effect estimation or uncertainty calibration.

CausalDS uniquely realizes a synthetic, audit-verifiable, empirically compositional setting with deliberate stress-testing of identification, abstention, and tradeoffs in tool-using causal data science.

Implications and Future Directions

CausalDS empirically validates that high-level symbolic causal reasoning (graph structure, identification) does not guarantee capability on the full spectrum of agentic causal data science. With strong numerical results showing clear model stratification along axes of uncertainty calibration, abstention, and tool-use efficiency, the benchmark both quantifies and diagnoses systematic failure modes—especially overconfidence and tendency to hallucinate answers when abstention is warranted.

From a theoretical perspective, the CausalDS findings challenge the assumption that scale alone (“emergence”) suffices for well-rounded causal inference: the epistemic axes require explicit, targeted capability. Practically, improved model development should focus on robust abstention-aware frameworks, enhanced uncertainty elicitation, and data-aware strategies for managing measurement error and latent confounding.

The benchmark’s compositional nature supports further extension—harder scenes, more diverse motif mixes, targeted ablation exams—and invites systematic study of intervention, counterfactual, and abstention calibration under adversarial data conditions.

Conclusion

CausalDS sets a new bar for evaluating agentic causal reasoning in LLMs, tying together symbolic, statistical, and epistemic competencies in a rigorously controlled, empirically anchored, and algorithmically auditable framework. The strong separation of models across all axes—particularly for abstention and interval uncertainty—indicates persistent gaps in current frontier and open-weight agents. CausalDS will be a central resource for both advancing agentic causal inference and diagnosing the most pressing areas for improvement in the construction of LLM-based data-science agents.

---

References: (2607.08093), [jin_cladder_2023], [tchetgen_introduction_2024], [hobor_bayesian_2026], [epstein_llms_2025], [renda_openestimate_2025], [jin_large_2023], [jin_large_2024], [yang_causalab_2026], [jin_can_2023], [sawarni_causalreasoningbenchmark_2026], [acharya_causcibench_2026], [lee_econcausal_2026]