Ask the Right Comparison:Bias-Aware Bayesian Active Top-$k$ Ranking with LLM Judges

Abstract: LLMs are increasingly used as cheap, scalable judges that compare candidate outputs pairwise -- to rank responses, select models, or triage papers. Yet LLM judges are both noisy and systematically biased: they favor verbose or well-formatted answers and exhibit position effects, so simply aggregating their votes recovers a ranking of presentation, not of true quality. We study the practical goal of identifying the \topk{} items under a fixed comparison budget, and make two contributions. First, we cast judging as Bayesian inference over latent quality with explicit, judge-specific bias covariates (verbosity, position), regularized by a shrinkage prior so that the data decide which biases a given judge actually exhibits. Second, we introduce a \topk-aware active acquisition rule that chooses the next comparison to maximally reduce uncertainty about \topk{} \emph{membership}, rather than about the full ranking. On a controlled benchmark with known ground-truth quality, judged by sixteen real LLMs spanning open and proprietary families (Llama, Qwen, Phi-4, GPT-4o-mini/5.1/5.5, Gemini, DeepSeek, and Claude Haiku/Sonnet/Opus), naive aggregation plateaus at a wrong \topk{} on biased judges regardless of budget, while our bias-aware model recovers it; \topk-aware acquisition reaches this ceiling with far fewer comparisons than round-robin or a global-uncertainty (D-optimal) rule. Bias is real but heterogeneous and capability-dependent: cheap and mid-tier judges carry a strong verbosity bias that our model corrects (lifting recall from $\sim$$0.5$--$0.6$ to $0.84$--$1.0$), whereas the frontier judges we tested show little bias and already rank accurately, so bias-aware modeling changes little there.

• The paper introduces a Bayesian Bradley–Terry model augmented with bias covariates to explicitly separate true quality from presentation-driven bias.
• It employs a top‑k‑aware adaptive acquisition function that strategically prioritizes comparisons near the decision boundary to optimize evaluation cost.
• Empirical results show significant recall improvements and cost efficiencies, particularly for mid-tier LLM judges with pronounced presentation biases.

Bias-Aware Bayesian Active Top-$k$ Ranking with LLM Judges: A Technical Overview

Introduction and Motivation

The use of LLMs as automated judges for pairwise comparison has become standard for ranking and evaluating natural language outputs due to cost and scalability advantages. However, empirical evidence demonstrates that LLM judges exhibit both stochastic noise and systematic biases—especially a pronounced preference for verbose or well-formatted answers and order-based position effects. Simply aggregating LLM comparison votes based on win counts leads to systematic misranking, conflating true underlying quality with superficial presentation features. This presents a significant challenge for applications where high-precision identification of the top-$k$ items is required under a fixed query budget.

Figure 1: Budget-limited active top-$k$ selection workflow with explicit bias modeling and targeted acquisition.

Methodology

Explicitly Modeling Judge Bias

The central technical contribution is a Bayesian Bradley–Terry model augmented with observable bias covariates ($x$) for each item—such as response verbosity and a position effect—and corresponding judge-specific coefficients ($c$ for verbosity, $\kappa$ for position). This enables explicit separation of presentation-driven bias from genuine latent quality $\theta$:

$$\Pr(y=1 \mid a,b) = \sigma\left(\theta_a - \theta_b + c^\top(x_a - x_b) + \kappa\right),$$

where $\sigma(\cdot)$ is the logistic function. A Gaussian shrinkage prior on $c$ and $k$0 ensures data-driven regularization—critically, if a judge is unbiased, the bias terms are nullified automatically, ensuring safe model application even without prior knowledge of the judge's characteristics.

Top-$k$1-Aware Adaptive Acquisition

To optimize under a strict comparison budget, the active learning component employs an acquisition function that prioritizes pairs expected to maximally reduce uncertainty about top-$k$2 membership, rather than the global ranking. Quantitatively, for each candidate pair $k$3, the utility score is:

$k$4

where $k$5 is the current estimated probability that item $k$6 is in the top-$k$7, and $k$8 is the binary entropy. This focuses the comparison budget on items near the top-$k$9 boundary, efficiently resolving ambiguous cases.

Empirical Results

Bias Prevalence and Correction

On a controlled benchmark with ground-truth item quality and artificially decoupled verbosity, sixteen real LLM judges of differing capability (including Llama, Qwen, GPT-4o-mini/5.1/5.5, Gemini, Claude, DeepSeek, and Phi) were assessed for ranking fidelity and bias separation. Naive aggregation (i.e., using the standard Bradley–Terry without explicit bias terms) consistently plateaued at an incorrect top-$k$0 due to confounding between verbosity and perceived quality (up to +0.40 recall error on biased judges). In contrast, the bias-aware model cleanly separated verbosity effects and recovered the true top-$k$1 set, achieving recall improvements up to +0.40.

Figure 2: Verbosity-induced confounding in naive aggregation: lower-quality verbose items supplant terse top items in the naive top-$k$2.

Bias magnitude was highly heterogeneous and inversely correlated with LLM competence. Bias-aware debiasing provided pronounced gains on mid-tier/cheaper LLMs with substantial verbosity bias (e.g., Llama-3.1-8B, Qwen-2.5), but offered negligible changes for frontier models (e.g., GPT-5.1, Claude-Opus) with essentially zero detectable bias, confirming safe deployment in unknown scenarios due to the adaptive shrinkage prior.

Budget Efficiency of Top-$k$3-Aware Acquisition

The top-$k$4-aware acquisition rule achieved higher recall at every budget compared to round-robin and global D-optimality baselines, particularly for small and mid-range budgets.

Figure 3: Top-$k$5 recall as a function of pairwise comparison budget, highlighting sample efficiency of the active rule.

Budget allocations were concentrated on comparisons near the top-$k$6 boundary, as desired, whereas naive baselines waste resources resolving uninformative contrasts.

Figure 4: Distribution of comparison effort: the active acquisition method intensifies sampling near the decision frontier.

Ablation confirms the unique contribution of the boundary-targeting term in the acquisition function: removing it leads to substantial degradation in sample efficiency, showing it is not merely a generic information-gathering strategy.

Figure 5: Ablation of the acquisition score demonstrates necessity of each component for sample-efficient top-$k$7 discovery.

External Validation: Nuanced Effects of Debiasing

Validation on LLMBar demonstrates that length-based correction increases gold agreement precisely when length is a spurious feature, but harms performance in settings where length is genuinely predictive of quality (e.g., SummEval).

Figure 6: On adversarial LLMBar subsets with presentation traps, length correction recovers ground-truth ordering; unaffected when length is non-spurious.

A cost-quality tradeoff further reveals that correcting bias in a cheap judge can match frontier judge performance at a fraction (1/20th) of the cost at high but imperfect recall thresholds.

Figure 7: Cost–quality frontier: debiased cheap-LLM performance matches, at far lower cost, that of an unbiased expensive LLM judge up to very high recall.

Contextual Dependence and Limitations

The effectiveness of the method critically depends on separating spurious presentation features from legitimate quality signals in the underlying dataset. When the covariate genuinely correlates with true quality (as in SummEval or Nectar), unwarranted correction reduces recall; a practical gating mechanism leveraging a small set of trusted pairwise labels can decide whether to enable debiasing dynamically.

Bias prevalence is judge-specific and not a function of LLM family alone—the direction and magnitude of both verbosity and position biases varied significantly across evaluated models (Figure 8).

Figure 8: The gain from bias-aware debiasing concentrates on judges that are both competent and non-neutral in verbosity bias; frontier judges require no correction.

Theoretical Analysis

Naive Bradley–Terry aggregation, when facing systematic bias orthogonal to ground-truth quality, is formally inconsistent for top-$k$8 recovery; this confounding persists regardless of sample size. The bias-aware model is identifiable under mild randomization and coverage assumptions, and the acquisition function is heuristically justified for maximal entropy reduction over the relevant top-$k$9 boundary, rather than the global item order.

Practical and Theoretical Implications

The findings directly inform LLM-based evaluation policy at scale for research and production:

• Safe Debiasing for Black-box Judges: The adaptive shrinkage mechanism ensures that debiasing can be applied universally, with negligible risk on unbiased judges and significant gains where bias is present.
• Sample-Efficient Evaluation Pipelines: Top-$x$0-aware adaptive querying achieves superior recall with reduced cost, enabling viable deployment in resource-constrained environments.
• Dynamic Debiasing Activation: The proposed trusted-anchor gating addresses the challenge of feature legitimacy heterogeneity across datasets, preventing performance degradation in settings where presentation features signal true quality.

On the theoretical front, this work strengthens the argument that explicit, probabilistic modeling of systematic bias is mandatory for rigorous automated evaluation with LLMs, especially as model diversity and application heterogeneity grow.

Future Directions

A salient extension is multi-fidelity ranking—escalating undecidable cases from cheap, biased judges to expensive, near-unbiased models, with robust modeling of per-judge bias. The observed failure of naive judge mixing highlights the need for future work on hierarchical or federated bias-credit assignment across heterogeneous annotators.

Conclusion

Explicit, Bayesian modeling of judge-specific, presentation-driven bias—coupled with active, boundary-aware acquisition—enables high-fidelity, cost-efficient identification of top-$x$1 items using LLM-as-a-judge protocols. This approach is essential for avoiding systematic ranking errors due to bias and for achieving reliable evaluation at scale, particularly with inexpensive, mid-tier LLMs. Debiasing delivers maximum benefit in precisely the practical regime where automated evaluation is most attractive, and its downside is controlled when unneeded. Future developments should focus on multi-fidelity systems and dynamic debiasing policies responsive to task- and judge-specific characteristics.