METER: Evaluating Multi-Level Contextual Causal Reasoning in Large Language Models

Abstract: Contextual causal reasoning is a critical yet challenging capability for LLMs. Existing benchmarks, however, often evaluate this skill in fragmented settings, failing to ensure context consistency or cover the full causal hierarchy. To address this, we pioneer METER to systematically benchmark LLMs across all three levels of the causal ladder under a unified context setting. Our extensive evaluation of various LLMs reveals a significant decline in proficiency as tasks ascend the causal hierarchy. To diagnose this degradation, we conduct a deep mechanistic analysis via both error pattern identification and internal information flow tracing. Our analysis reveals two primary failure modes: (1) LLMs are susceptible to distraction by causally irrelevant but factually correct information at lower level of causality; and (2) as tasks ascend the causal hierarchy, faithfulness to the provided context degrades, leading to a reduced performance. We belive our work advances our understanding of the mechanisms behind LLM contextual causal reasoning and establishes a critical foundation for future research. Our code and dataset are available at https://github.com/SCUNLP/METER .

• The paper introduces METER, a unified benchmark assessing discovery, intervention, and counterfactual causal reasoning within a single narrative context.
• The paper finds that LLM performance declines with increased causal complexity, with counterfactual tasks showing a significant gap compared to human accuracy.
• The paper employs token saliency tracing, demonstrating that explicit evidence marking in prompts improves accuracy and offers pathways for future enhancements.

METER: A Framework for Multi-Level Contextual Causal Reasoning Evaluation in LLMs

Motivation and Benchmark Construction

Contextual causal reasoning—deductive inference based on specific, situated natural language context—is an indispensable capability for robust AI, especially for domains where nuanced, context-specific causal chains determine outcomes, such as medical or legal applications. Prior benchmarks have typically assessed LLMs under fragmented settings, often focusing only on individual causal tasks (discovery, intervention, or counterfactual reasoning) or mixing different tasks using unrelated contexts, which obfuscates cross-level performance analysis and leads to biased assessment.

METER addresses these limitations by introducing a systematically structured benchmark in which each instance is a unified narrative from which three types of causal queries are derived, corresponding to Pearl’s Causal Ladder: discovery (association), intervention (manipulation), and counterfactual (retrospective deviation). The dataset consists of 4,145 entries, each paired with three multiple-choice questions (one per causal level), constructed with careful data de-contamination and multilayered human validation.

Figure 1: Illustration of the contextual causal reasoning process METER targets, exemplifying association, intervention, and counterfactual queries from a unified context.

METER’s data pipeline combines automated event description expansion (to overcome sparse trigger-based annotations in existing datasets), prompt-based question/answer/distractor generation with LLMs, and thorough manual curation and error categorization, resulting in high-quality, context-grounded items. Distractors are systematically categorized (contradictory, unfounded, causal reversal, irrelevant fact) to diagnose specific failure modes.

Distinguishing Causal Reasoning Paradigms

METER is positioned as an evidence-driven, context-bound evaluation, sharply distinct from “commonsense” (parametric knowledge-based) and “formal” (symbol-manipulation or graphical model-based) causal reasoning paradigms. As established in the related work and exemplified cases, contextual causal reasoning requires dynamic, prompt-dependent extraction and application of causal chains, immune to simple harvesting of “static” world knowledge or the application of predefined causal calculus.

Experimental Evaluation and Key Findings

Diverse LLMs—including both closed-source (GPT-4o, Gemini3-Flash, Gemini3-Pro, GPT-5) and open-source (Qwen3 family, Llama-3), and models specifically tuned for reasoning—are comprehensively evaluated under four prompting strategies: zero-shot, few-shot, zero-shot CoT, few-shot CoT.

Core empirical findings:

• Performance degrades sharply as causal complexity increases: LLMs approach human-level accuracy on discovery tasks but performance drops consistently for intervention (on average, –15.78%) and further for counterfactual (–26.27%) queries.
• Reasoning-optimized models maintain stronger performance at higher causal levels: For example, Gemini3-Pro, GPT-5, and Qwen3-Next-Thinking outperform their instruction-tuned counterparts on intervention and counterfactual, but the gap is most pronounced in high-complexity tasks.
• Prompting impacts vary non-monotonically: Few-shot learning improves higher-level causal reasoning, while CoT prompts may benefit some models (e.g., Llama-3) yet degrade performance in models with strong intrinsic causal chains (e.g., GPT-4o).
• Scaling is critical for complex causal reasoning tasks: Discovery capability saturates by 4–8B parameters; by contrast, intervention and counterfactual reasoning see steady gains through 32B+.
• Human performance exceeds all tested LLMs, especially for counterfactuals (human: 91% vs. best LLM: ~77%).

Figure 2: LLM performance comparison across all three causal levels under a unified context. Performance gaps widen markedly as task complexity increases.

Error Pattern Analysis

Through the explicit use of categorized distractors, the error patterns can be precisely charted:

• Causal discovery errors are dominated by distractions from contextually irrelevant but factually correct statements. The strongest error class in low-level tasks is “Irrelevant Fact.”
• At higher causal levels, faithfulness to the specific context erodes. Errors shift towards “Unfounded Statement”—hallucinated or de-contextualized answers—and “Contradictory Statement” (options that conflict with the premises/questions).
• Causal reversal errors (choosing an effect as cause or vice versa) are non-dominant but still contribute meaningfully, especially when explicit causal directionality is unclear.

Mechanistic Interpretability: Information Flow Tracking

A salient contribution of the work is the internal probing of LLMs using token-to-token saliency tracing. By mapping saliency between semantically annotated input segments—namely, evidence (E), non-evidence (N), question (Q), selected option (O), and the output position (T)—the study traces where models aggregate information when making causal inferences.

Figure 3: Layer-wise information flow dynamics for Qwen3-8B, highlighting context-evidence utilization in different causal levels and its degradation in complex reasoning.

• Causal discovery relies on early-layer aggregation of context evidence. High E→O saliency is observed in shallow transformer layers, indicating effective grounding.
• Intervention and counterfactual tasks trigger a drop in E→O saliency and heightened reliance on Q→O, indicating models shift toward internal priors or parametric knowledge, failing to utilize prompt-specific facts. This detachment mechanistically explains the empirically observed hallucinations and inconsistency errors.
• Attention masking interventions confirm the functional necessity of context-evidence flow (E→O) for causal discovery, but find negligible effect for higher-level tasks, causally validating the observed behavior.

Figure 4: Impact of masking evidence-to-option attention flow in shallow model layers; Causal Discovery accuracy collapses, while higher-level tasks remain insensitive.

Implications and Future Directions

Theoretical: The layered breakdown in evidence aggregation underpins clear mechanistic limitations in current LLM architectures for multi-level contextual causal reasoning. The progression from context-anchored extraction (discovery) to prior-anchored simulation (counterfactual) exposes a gap between parametrically trained models and human-style, narrative-grounded causal inference.

Practical: For high-stakes and regulatory-critical applications (e.g., medicine, law, science policy), this study cautions against the unqualified deployment of LLMs for advanced causal tasks, given their strong tendency towards hallucinated, unfaithful, or logically inconsistent responses outside of explicit context cues.

Eliciting improvement: The experiment demonstrating that explicit marking of evidence spans in the prompt directly enhances both information flow and answer accuracy (+2.5% to +4.6%) motivates further work on supervised fine-tuning, RL-based objective shaping, or dedicated architectural modules for focused context utilization in causal reasoning.

Generalizability/limitations: Mechanistic analysis is constrained to open-weight models. Extension to closed-source foundation models awaits further research access. Data contamination remains a theoretical risk, although strong filtering mitigates this.

Conclusion

METER establishes the state of the art in benchmarking multi-level contextual causal reasoning in LLMs, providing granular, unified-context evaluation and mechanistic insight into the strengths and limitations of current paradigms. The gap between LLM and human performance, especially at counterfactual and intervention levels, is technologically explainable as a deficiency of evidence binding and dynamic causal chain simulation. The METER dataset and methodology define a critical foundation for research on training strategies, model architectures, and evaluation settings to drive future advances in the causal reasoning capacity of foundation models.