How to Correctly Report LLM-as-a-Judge Evaluations

Abstract: LLMs are increasingly used as evaluators in lieu of humans. While scalable, their judgments are noisy due to imperfect specificity and sensitivity of LLMs, leading to biased accuracy estimates. Although bias-correction methods exist, they are underutilized in LLM research and typically assume exact knowledge of the model's specificity and sensitivity. Furthermore, in general we only have estimates of these values and it is not well known how to properly construct confidence intervals using only estimates. This work presents a simple plug-in framework that corrects such bias and constructs confidence intervals reflecting uncertainty from both test and calibration dataset, enabling practical and statistically sound LLM-based evaluation. Additionally, to reduce uncertainty in the accuracy estimate, we introduce an adaptive algorithm that efficiently allocates calibration sample sizes.

• The paper introduces a bias correction framework that adjusts for specificity and sensitivity errors in LLM evaluations.
• It develops statistically valid confidence intervals incorporating uncertainties from both test and calibration data.
• The study proposes an adaptive calibration strategy to optimally allocate sample data, reducing bias and annotation costs.

Correct Statistical Practices for LLM-as-a-Judge Evaluation

Motivation and Problem Overview

The widespread substitution of human evaluation with LLM-based automated judgment in NLP and AI tasks offers scalability and efficiency. However, the inherent noise in LLM judges, stemming from their imperfect specificity ($q_0$) and sensitivity ($q_1$), induces systematic bias in commonly reported accuracy metrics. The naive estimator—typically the raw fraction of "correct" labels assigned by the LLM—yields a biased estimate of true model accuracy with respect to human ground-truth. This bias can be positive or negative depending on the underlying error profile of the judge and the region of ground-truth accuracy. Consequently, both overstatement and understatement of system-level model performance can occur, contaminating benchmarking, model selection, and longitudinal comparison.

Figure 1: Illustration of the bias in the naive LLM-as-a-Judge estimator $\hat{p}$ and the effectiveness of bias correction across the space of true accuracy $\theta$.

Bias Characterization and Statistical Correction Framework

The paper formalizes the bias as a function of the judge’s error rates, using parameters $q_0$ (specificity) and $q_1$ (sensitivity). The expected observed proportion $\mathbb{E}[\hat{p}]$ deviates from true accuracy $\theta$ except in the limiting case of perfect judges ($q_0 = q_1 = 1$). Specifically:

$$\mathbb{E}[\hat{p}] = (q_0 + q_1 - 1)\theta + (1 - q_0)$$

This relationship reveals a critical threshold at $\theta^* = (1 - q_0)/(2 - q_0 - q_1)$, at which the bias changes sign—overestimating $\theta$ for smaller values, underestimating at higher ones.

The authors raise a central issue: direct reporting of $\hat{p}$ is statistically invalid unless the LLM judge's confusion profile is perfectly known and invertible. While calibrating for $q_0$ and $q_1$ is standard in prevalence estimation [rogan1978estimating], such corrections are largely absent from LLM-based evaluation literature. Moreover, when $q_0$ and $q_1$ are themselves only estimated (through a finite calibration set), previous work failed to address proper confidence-interval construction that incorporates all relevant sources of uncertainty.

To resolve this, the paper proposes a plug-in estimator:

$$\hat{\theta} = \frac{\hat{p} + \hat{q}_0 - 1}{\hat{q}_0 + \hat{q}_1 - 1}$$

where the calibration sample yields empirical estimates $\hat{q}_0$ and $\hat{q}_1$. This adjustment is shown to be, for large enough calibration size, uniformly lower in absolute bias than the naive estimator.

Uncertainty Quantification and Confidence Interval Construction

A major technical contribution is the construction of statistically valid confidence intervals for $\theta$ that jointly incorporate uncertainty from both test data (estimation of $\hat{p}$) and calibration data (estimation of $\hat{q}_0$, $\hat{q}_1$). The resulting interval has the form:

$$\tilde{\theta} + d\tilde{\theta} \pm z_\alpha \sqrt{ \frac{ \tilde{p}(1 - \tilde{p})/\tilde{n} + (1 - \tilde{\theta})^2 \tilde{q}_0(1 - \tilde{q}_0)/\tilde{m}_0 + \tilde{\theta}^2 \tilde{q}_1(1 - \tilde{q}_1)/\tilde{m}_1 }{ (\tilde{q}_0 + \tilde{q}_1 - 1)^2 } }$$

where the tilde denotes "add-two" Wald-adjusted proportions [agresti2000simple]. This accounts for finite sample bias and improves coverage, and the authors advise truncating values to the $[0,1]$ interval as necessary.

Figure 2: The relationship between calibration dataset size and the confidence interval width under various test set accuracies shows diminishing returns and guides allocation strategy.

Adaptive Calibration and Optimal Sample Allocation

Another practical issue addressed is how to allocate the calibration budget. The authors find that the two components—calibration for specificity ($m_0$ for negatives) and calibration for sensitivity ($m_1$ for positives)—contribute asymmetrically to variance depending on the empirical accuracy and LLM error rates. They derive a closed-form optimal allocation ratio for $(m_0, m_1)$ under asymptotic analysis:

$$m_0^* \approx (1/\hat{p} - 1) \sqrt{\kappa} \cdot m_1 \quad \text{where} \ \kappa = \frac{1-\hat{q}_0}{1-\hat{q}_1}$$

This is implemented in an adaptive pilot-based procedure that collects initial calibration data, estimates the variance ratio across conditions, and allocates the remainder optimally to minimize the final confidence interval width.

Figure 3: Monte Carlo validation demonstrates unbiasedness of the adjusted estimate, high coverage of the proposed interval, and interval length reduction from adaptive allocation.

Empirical Validation

Extensive Monte Carlo simulations show that the bias-adjusted estimator $\hat{\theta}$ recovers the true accuracy across the spectrum of $\theta$, while the naive estimator’s bias can be large and of either sign. Empirical coverage of the proposed intervals holds at the desired nominal levels. The adaptive allocation strategy achieves consistently shorter intervals for fixed calibration effort, thus reducing the total annotation burden.

Implications and Future Considerations

Practically, these corrections are essential for sound benchmarking in any context using automated LLM evaluation, such as model-vs-model comparison, judging improvements, or automated leaderboard construction. The continuing evolution of LLM architectures and corresponding judge models with shifting specificity/sensitivity profiles amplifies the need for statutorily sound reporting procedures, particularly as differences in judged gains may be within the margin of error induced by bias and uncertainty in the judge itself.

Theoretically, this work extends prevalence adjustment literature—traditionally situated in epidemiology—to the AI evaluation paradigm, offering a data-driven methodology for fair and transparent reporting. Future extensions could consider multi-label classification, dependency across samples, or joint estimation when the same judge model is used for both calibration and testing datasets, as well as Bayesian formulations for small data regimes.

The algorithm and codebase enable direct integration of this framework in LLM evaluation pipelines. The methods presented provide a robust statistical foundation for LLM-as-a-Judge evaluation moving forward.

Conclusion

This work establishes robust, statistically principled estimators and confidence intervals for LLM-as-a-Judge evaluations. By providing bias adjustment and proper uncertainty quantification that reflect the dual noise sources from test and calibration data, along with guidance on sample allocation, it enables more reliable and interpretable use of LLM-based model evaluation. Adoption of these methods will enhance the rigor of future reporting and facilitate trustworthy comparison in empirical LLM research.

Reference:

"How to Correctly Report LLM-as-a-Judge Evaluations" (2511.21140)