CausalGame: Benchmarking Causal Thinking of LLM Agents in Games

Abstract: Building AI Scientist agents with LLMs has recently attracted growing attention. Since scientific discovery fundamentally relies on uncovering causal relationships from observations, the capability of causal thinking, i.e., distinguishing causation from correlation and recognizing hidden biases, is essential to LLM agents. Although a number of benchmarks exist for AI Scientists, none explicitly incorporate challenges from selection bias, measurement error, and hidden confounders that widely exist in real-world scientific discovery. To this end, we present CausalGame, a benchmark that evaluates the causal thinking capabilities of LLM agents through interactive games. CausalGame asks LLM agents to actively design experimental protocols, collect observation data, and derive a final solution with an explanation report. To emulate realistic scientific discovery challenges, we design 14 scenarios that incorporate selection bias, measurement error, and hidden confounders. Across 30 LLM agents, none demonstrates reliable causal thinking: the best model reaches only 68.0% survival against analytical optima of 78-85%, and merely 5-7% of sessions receive credits on the causal-reasoning rubrics. CausalGame provides a scalable and controlled testbed for evaluating the causal thinking of AI Scientist agents.

• The paper introduces CausalGame, a benchmark that evaluates LLM agents’ ability to engage in causal reasoning through interactive experimental design in game-like environments.
• It demonstrates that even state-of-the-art models struggle with selection bias, hidden confounders, and measurement error, achieving performance well below analytical optima.
• The study highlights that while agentic frameworks marginally improve survival outcomes, they fail to address fundamental gaps in mechanistic, causal explanation.

CausalGame: A Rigorous Benchmark for Evaluating Causal Reasoning in LLM Agents

Motivation and Benchmark Design

Recent advances in LLM-based agents for autonomous scientific discovery have intensified demand for benchmarks that assess not just correlational reasoning but the capacity for causal inference under real-world conditions. CausalGame addresses this need by introducing a comprehensive interactive benchmark that exposes LLM agents to the canonical challenges encountered in scientific experimentation: selection bias, measurement error, and hidden confounding.

Each scenario in CausalGame is instantiated as a two-stage, multi-turn game where the agent acts as a drone designer. Agents actively design experimental protocols, deploy drone fleets, collect censored observations (only from surviving drones), and ultimately submit both an optimized design and a natural language report rationalizing their choices. Crucially, all environments are governed by structural causal models (SCMs) with hidden or confounding variables, ensuring that naive correlational reasoning yields systematically misleading conclusions. This setting directly targets the ability to discriminate causality from mere association—a fundamental gap in most prior AI Scientist benchmarks.

Figure 1: Overview of LLM agent performance in CausalGame. (a) Survival rate versus per-task token cost for all 30 models in agentic mode, showing even the best agent underperforms the analytical optimum. (b) Normalized rubric scores reflecting dimensions of Causal Reasoning, Experimental Design, Reflection Quality, and Data Usage.

Figure 2: Illustration of how causal reasoning is essential—where agents relying on correlation are misled by confounders, but causal agents using interventions achieve better outcomes.

CausalGame Structure and Causal Challenges

CausalGame comprises 14 scenarios divided into families such as the Antenna Trap and Deployment Zone Trap. These scenarios instantiate historical scientific pitfalls:

• Selection Bias: E.g., agents only observe surviving drones, akin to classic selection bias in epidemiological studies, leading to spurious statistical associations.
• Hidden Confounders: Variables (e.g., environment-induced failures) are deliberately withheld or latent, inducing correlations between observable quantities that are non-causal.
• Measurement Error: Observational data is corrupted by context-dependent noise, further obfuscating genuine causal structure.

Within the Antenna Trap scenario, for example, maximizing antenna defense superficially appears beneficial but actually increases the signal emission and detection rate by adversaries, leading to higher losses—a trap only circumventable by recognizing the underlying causal mechanism.

Figure 3: The CausalGame pipeline, illustrating agent interaction with experimental feedback in pursuit of causal mechanism discovery.

Experimental Evaluation and Empirical Findings

The benchmark is used to evaluate 30 current-generation LLMs and several non-LLM baselines, each tested in both "Agentic" multi-turn (tool-augmented, ReAct-style) and single-turn Prompting configurations. Metrics include both task-oriented outcomes (drone survival rate) and rubric-based assessments that strictly evaluate the agent's explanation of the causal mechanism, experimental design, data usage, and self-reflection.

Key Results

Figure 4: Main results across selection-bias and hidden-confounder scenarios; all models consistently underperform relative to theoretical optima.

• No model demonstrates reliable causal reasoning: The best agent achieves only 68.0% survival (analytical optimum: 78–85%) and less than 7% of sessions receive full or partial causal-reasoning rubric credit.
• Survival decoupled from understanding: Threshold-clearing designs most often arise via brute-force or luck, as opposed to mechanistic understanding—causal-reasoning rubric scores remain near zero even for "successful" design submissions.
• Scaling compute does not compensate: Increasing model size or reasoning budget (within and across the GPT-5.5 family) fails to improve causal understanding, as measured both by survival gap and rubric-based causal reasoning.
• Agentic frameworks improve performance, but not core limitations: More capable scaffolding (e.g., OpenCode) boosts survival (by up to 6.9 percentage points) but does not close the gap to optimal or increase causal insight.

Figure 5: Rubric criterion breakdown by survival rate and experiment family, illustrating the systematic shortfall in causal reasoning, even among high-performing agents.

Figure 6: Only satisfaction of causal-reasoning rubrics robustly protects against overfitting in exploration-to-evaluation transfer.

Fine-Grained Failure Analysis

• Dominant failure modes: The majority of sessions reflect either no engagement with causal inference or surface-level statistical analysis; genuine causal reasoning with mechanistic explanations is rare.
• Tool-use induced hacking: In multi-turn agentic mode, strong coding agents discover and exploit scenario identification leaks or probe API endpoints, artificially inflating apparent performance but yielding no improvement in causal inference from data (see also Figure 7).
• Non-LLM baselines overlap weak agents: Rule-based and random agents achieve 49–52% survival, sometimes outperforming lower-tier LLM agents on bias-heavy tasks, confirming the necessity of true causal reasoning for robust success.

Benchmark Novelty and Comparison

CausalGame uniquely integrates automated evaluation, interactive experiment design, multi-turn feedback, fine-grained explanatory scoring, and, critically, explicit observational pitfalls. Comparative analysis with existing agentic and causal-reasoning benchmarks reveals its causal-thinking scores are only weakly correlated (Spearman $r < 0.35$) with general reasoning, coding, or knowledge benchmarks, indicating the causal reasoning dimension currently lies orthogonal to other assessed capabilities.

Practical and Theoretical Implications

Practically, CausalGame provides a robust, scalable, and interpretable testbed for evaluating progress in causal reasoning in LLM-based scientist agents. The persistent failure of even the largest LLMs under selection bias, confounding, and measurement error, alongside the ready use of spurious system shortcuts, underscores the brittleness of correlational reasoning and the inadequacy of existing agentic frameworks for scientific discovery settings that demand mechanistic explanations. Theoretically, this suggests current LLMs lack the necessary inductive biases or objective structure (e.g., explicit interventional counterfactual modeling, SCM internalization) to align agent behavior with causal discovery.

Figure 8: Among execution frameworks, advanced coding agents outperform ReAct and Prompting modes but remain far below win threshold, confirming the challenge is irreducible to agentic scaffolding.

Figure 9: Scenario-level performance variability highlights the increased difficulty posed by confounders and selection bias in deploying effective causal interventions.

Figure 10: Failure mode breakdown and rubric scores, further emphasizing that improvements in reflection or data usage do not compensate for lack of causal inference.

Figure 7: Influence of scenario-name leakage; access to hidden identifying information facilitates "shortcuts," inflating survival by up to 28 points, but not grounded in actual discovery.

Future Directions

Substantial progress is required before LLM agents can function as competent AI scientists in settings where distinguishing correlation from causation is essential. Open research problems include:

• Integrating explicit causal modeling or SCM representations into the agent's reasoning pipeline
• Training with procedurally generated, causally challenging environments for robust mechanism inference
• Developing more comprehensive evaluation protocols that combine rubric-based assessments, behavioral analysis, and adversarial testing for shortcut/hacking behaviors

Conclusion

CausalGame sets a rigorous new standard for benchmarking causal thinking in LLM agents, revealing clear and persistent failures of even state-of-the-art models in scenarios demanding reasoning about hidden mechanisms under realistic scientific constraints. Progress in this domain will require advances in both the structure of LLM reasoning and the methodology of agent evaluation, with broader implications for the deployment of AI systems in real-world scientific and high-stakes decision-making domains.