Causal Judge Evaluation: Calibrated Surrogate Metrics for LLM Systems

Abstract: LLM-as-judge evaluation has become the de facto standard for scaling model assessment, but the practice is statistically unsound: uncalibrated scores can invert preferences, naive confidence intervals on uncalibrated scores achieve near-0% coverage, and importance-weighted estimators collapse under limited overlap despite high effective sample size (ESS). We introduce Causal Judge Evaluation (CJE), a framework that fixes all three failures. On n=4,961 Chatbot Arena prompts (after filtering from 5k), CJE achieves 99% pairwise ranking accuracy at full sample size (94% averaged across configurations), matching oracle quality, at 14x lower cost (for ranking 5 policies) by calibrating a 16x cheaper judge on just 5% oracle labels (~250 labels). CJE combines three components: (i) AutoCal-R, reward calibration via mean-preserving isotonic regression; (ii) SIMCal-W, weight stabilization via stacking of S-monotone candidates; and (iii) Oracle-Uncertainty Aware (OUA) inference that propagates calibration uncertainty into confidence intervals. We formalize the Coverage-Limited Efficiency (CLE) diagnostic, which explains why IPS-style estimators fail even when ESS exceeds 90%: the logger rarely visits regions where target policies concentrate. Key findings: SNIPS inverts rankings even with reward calibration (38% pairwise, negative Kendall's tau) due to weight instability; calibrated IPS remains near-random (47%) despite weight stabilization, consistent with CLE; OUA improves coverage from near-0% to ~86% (Direct) and ~96% (stacked-DR), where naive intervals severely under-cover.

• The paper introduces CJE, a framework that uses calibrated surrogate metrics to overcome preference inversion and invalid confidence intervals in LLM evaluation.
• It employs isotonic regression and a two-stage calibration process along with weight stabilization to reliably map judge scores to ground-truth outcomes.
• Empirical results demonstrate near-oracle ranking accuracy and restored uncertainty coverage, significantly reducing evaluation cost while ensuring scalability.

Authoritative Summary: Causal Judge Evaluation — Calibrated Surrogate Metrics for LLM Systems

Introduction and Motivation

Evaluating LLM policies at scale against expensive "oracle" ground-truth labels (human preference, expert audits, deployment KPIs) is cost-prohibitive, motivating the widespread use of LLMs themselves as "judges"—cheap surrogates trained or prompted to assign scores to model outputs. Standard practice relies on correlating judge scores with oracle targets, assuming this suffices for reliable comparative evaluation. However, this approach suffers from three systematic statistical failures: preference inversion (uncalibrated scores can misalign optimization with true quality), invalid confidence intervals (empirical coverage near zero), and catastrophic failure of standard off-policy evaluation (OPE) methods due to limited sample overlap.

"Causal Judge Evaluation" (CJE) presents a unified framework addressing these failures, delivering calibrated surrogate-based evaluation of LLM systems with rigorous coverage, bias detection, and scalable amortization across policies.

Figure 1: CJE pipeline overview: a small oracle slice trains a calibration model mapping judge scores $S$ to true labels $Y$, enabling low-cost evaluation on bulk responses by propagating calibrated surrogates.

Methodological Contributions

CJE's methodological design follows a single principle: Design-by-Projection (DbP). Each module encodes justified knowledge by projecting statistical objects onto closed convex sets in Hilbert space, minimizing variance subject to validity constraints.

1. Reward Calibration (AutoCal-R)

AutoCal-R fits a mean-preserving mapping from judge score $S$ to oracle outcome $Y$, using isotonic regression and, when necessary, a flexible two-stage fallback incorporating covariates (e.g., response length). By preserving the oracle-slice mean, calibration prevents preference inversion and supports both monotonic and non-monotonic relationships observed empirically.

Figure 2: AutoCal-R: monotone (isotonic) versus two-stage calibration—two-stage fits flexible, covariate-dependent patterns while retaining mean-honesty.

2. Weight Stabilization (SIMCal-W)

Naïve importance weighting (IPS/SNIPS) exhibits extreme variance under policy shift because logged actions barely cover target policy support. SIMCal-W projects ratios onto the cone of $S$-monotone, unit-mean weights and stacks candidate projections to minimize influence-function variance. Empirical majorization results in dramatic effective sample size (ESS) improvement, though, crucially, reliability remains limited by support overlap.

Figure 3: SIMCal-W stabilizes importance weights across policies—raw ratios exhibit 130+ orders of magnitude spread, projection restores ESS.

3. Oracle-Uncertainty-Aware (OUA) Inference

Standard CIs ignore calibration uncertainty, catastrophically under-cover when oracle budgets are small. CJE applies delete-one-oracle-fold jackknife, propagating first-stage calibration variance into total variance, restoring coverage rates near the nominal level even at low oracle fractions.

Figure 4: Uncertainty decomposition for direct+cov: at 5% oracle, calibration uncertainty dominates total variance—OUA inference is essential.

4. Policy-wise Transport Diagnostics

CJE formalizes a transport test: for each target policy $\pi'$, mean residual $E_{\pi'}[Y-f(S,X)]$ is tested. Calibration is safe to re-use only if residual mean is zero—empirically detecting adversarial or drifted policies with systematic bias.

Figure 5: Policy-wise mean transport test: adversarial/unhelpful policy fails transport, revealing a $-0.31$ bias.

5. Direct Versus Off-Policy Methods

Experiments show direct evaluation using surrogate-calibrated fresh samples outperforms (or matches) doubly robust (DR) estimators and logs-only IPS variants in low overlap regimes. DR does not dominate when propensities are unreliable; its IPS component adds noise rather than information. Coverage-Limited Efficiency (CLE) diagnostics formalize this: high ESS alone is insufficient; support overlap (TTC) and distributional affinity are necessary.

Empirical Results

Extensive benchmarking on 4,961 Chatbot Arena prompts (5 candidate policies, including adversarial variations) supports CJE's effectiveness:

• Ranking Accuracy: Direct+cov achieves 94.3% pairwise accuracy; DR and stacked-DR provide similar accuracy with near-nominal CI coverage.
• Cost Efficiency: Calibrating on just 5% oracle labels ($\sim$250 labels) yields 99% ranking accuracy at 14$\times$ lower cost for the five-policy suite.
• Uncertainty & Coverage: CJE's OUA intervals restore empirical CI coverage from 0% (uncalibrated) to 85–96%, depending on estimator and oracle fraction.
• Weight Behavior: SIMCal-W boosts ESS by 4–3000$\times$ but CLE diagnostics show that IPS remains unreliable under poor logger coverage of target regions.

  Figure 6: CJE outputs calibrated policy value estimates with valid uncertainty—direct+cov achieves near-oracle accuracy and coverage.

  

  Figure 7: Sample size planning: MDE contours for direct+cov guiding design for statistical power under budget constraints.

  

  Figure 8: Two-stage calibration enables flexible, covariate-aware reward mapping—first spline on judge score, then isotonic projection preserves mean.

Theoretical Insights

CJE formalizes efficiency improvement via model restriction: each justified constraint encoded (e.g., monotonicity, mean-one weights, surrogate sufficiency) shrinks the tangent space, lowering the efficiency bound of estimation variance (Bickel et al. 1993). Influence-function (IF) stacking further decreases variance among regular estimators. Notably, richer judges guarantee improved efficiency by Blackwell monotonicity.

CLE diagnostics provide explicit sharp lower bounds on IPS variance in terms of coverage ($\alpha/\sqrt{\beta}$) and shape mismatch ($\chi^2$ divergence), demonstrating why weight stabilization alone is inadequate under support misspecification.

Practical Implications

• Practitioner Guidance: Always use OUA intervals; default to direct+two-stage calibration when possible; use CLE diagnostics to gate IPS/DR estimation; validate calibration transport across policies (policy-wise residual tests).
• Resource Allocation: Square-root law for optimal budget allocation balances surrogate scoring and oracle labeling given fixed cost constraints, with empirical validation confirming cost-effectiveness.
• Robustness: CJE's refusal-level gates report only ranking when identification is compromised (e.g., out-of-range calibration regions or drift).
• Engineering: Minimal architecture for deployment; persistent diagnostics, reproducible gates, and privacy features are specified.

Limitations and Future Directions

CJE assumes oracle ground-truth aligns with end-user objectives; calibration must generalize across time, distribution, and subgroups. Diagnostic gates and transparency mitigate risks, but further research is needed on selection-aware inference, robust isotonic calibration under heavy tails, active labeling, sequential agent evaluation, and fairness-aware constraints.

Conclusion

CJE substantially advances surrogate-based, low-cost, statistically sound evaluation of LLM systems by marrying mean-preserving calibration, variance-optimal weight stabilization, calibration-aware uncertainty quantification, and formal transport diagnostics. It supersedes default judge-based evaluation and unreliable off-policy methods by enabling confident, audit-ready deployment of scalable evaluation pipelines across diverse policies and environments.

Related Work and Theoretical Positioning

CJE integrates and generalizes threads from surrogate-assisted causal inference [KallusMao2020Surrogates], probability calibration [Ayer1955PAVA, ZadroznyElkan2002], off-policy evaluation [BangRobins2005], and recent stabilization and calibration for OPE [vanDerLaanLinCaroneLuedtke2025StabIPW, vanDerLaanLuedtkeCarone2024DRCalibration], and is complementary to recent efforts on bias mitigation and calibration of automatic LLM judges ("CalibraEval" [Li2025CalibraEval], "LLM-as-a-Judge Evaluations Reporting" (Lee et al., 26 Nov 2025)). Its separation of efficiency and identification under surrogacy offers an extensible paradigm for cost-effective, statistically principled evaluation in real-world ML deployments.

---

Conclusion

CJE constitutes a rigorous, principled solution to surrogate-based LLM evaluation, demonstrably overcoming core statistical and operational failures endemic in current practice. Its framework, theory, and pipeline are broadly relevant for scalable, reliable, and cost-efficient assessment of large models—especially critical as systems are deployed in high-stakes and fast-evolving contexts.