Self-Preference Bias in Rubric-Based Evaluation of Large Language Models

Abstract: LLM-as-a-judge has become the de facto approach for evaluating LLM outputs. However, judges are known to exhibit self-preference bias (SPB): they tend to favor outputs produced by themselves or by models from their own family. This skews evaluations and, thus, hinders model development, especially in settings of recursive self-improvement. We present the first study of SPB in rubric-based evaluation, an increasingly popular benchmarking paradigm where judges issue binary verdicts on individual evaluation criteria, instead of assigning holistic scores or rankings. Using IFEval, a benchmark with programmatically verifiable rubrics, we show that SPB persists even when evaluation criteria are entirely objective: among rubrics where generators fail, judges can be up to 50\% more likely to incorrectly mark them as satisfied when the output is their own. We also find that, similarly to other evaluation paradigms, ensembling multiple judges helps mitigate SPB, but without fully eliminating it. On HealthBench, a medical chat benchmark with subjective rubrics, we observe that SPB skews model scores by up to 10 points, a potentially decisive margin when ranking frontier models. We analyze the factors that drive SPB in this setting, finding that negative rubrics, extreme rubric lengths, and subjective topics like emergency referrals are particularly susceptible.

• The paper demonstrates that rubric-based evaluation reduces but does not eliminate self-preference bias compared to traditional pairwise methods.
• It introduces the Harmful Self-Preference Propensity Ratio (HSPP), showing up to a 47% bias in models favoring their own or same-family outputs.
• Results indicate that techniques like judge ensembling and rubric filtering mitigate bias yet leave residual effects, urging further bias correction research.

Self-Preference Bias in Rubric-Based Evaluation of LLMs

Introduction and Motivation

Automated evaluation of LLMs via "LLM-as-a-judge" has rapidly become standard practice for model development and benchmarking. Conventional paradigms such as pairwise comparison (PWC) and direct assessment (DA) suffer from known biases, in particular self-preference bias (SPB), where a model acting as judge evaluates its own generations or those from its own family more favorably. This phenomenon skews benchmarking outcomes and poses a critical risk in the context of self-improving or recursively optimized LLMs, potentially resulting in illusory progress.

Rubric-based evaluation (RB), involving per-criterion binary verdicts rather than holistic scores or rankings, is a relatively newer paradigm theoretically less susceptible to such biases due to its granularity, interpretability, and lack of response ordering or score calibration issues (see Figure 1).

Figure 1: Illustration of three LLM-as-a-Judge paradigms: pairwise comparison (PWC), direct assessment (DA) and rubric-based evaluation (RB).

Despite its advantages, it remained an open question whether SPB is present in RB, particularly under objective, programmatic rubrics. This paper ("Self-Preference Bias in Rubric-Based Evaluation of LLMs" (2604.06996)) delivers the first thorough quantitative and qualitative analysis of SPB in rubric-based LLM evaluation regimes, examining both objective (e.g., IFEval) and subjective (e.g., HealthBench) settings.

Experimental Setup and Methodology

The analysis is based on two central experimental axes:

• Evaluation Regimes and SPB Metrics: The work contrasts PWC, DA, and two RB variants (single-rubric/SR and all-rubric/AR), and establishes both instance-level and rubric-level overestimation metrics. The core SPB measure is the Harmful Self-Preference Propensity Ratio (HSPP), quantifying how much more likely a model as judge overestimates its own or same-family generations versus unrelated models.
• Benchmarks and Model Selection: Objective SPB is measured using IFEval—comprising programmatically verifiable rubrics—while subjective SPB is assessed with HealthBench, a large-scale medical conversational dataset with weighted and subjective binary rubrics. Judges and generators span leading families: Gemma, Llama, Qwen, GPT(-oss), and Claude.

The study also analyzes the effectiveness of mitigation strategies such as judge ensembling and rubric agreement filtering, and explores factors like rubric polarity, length, and topic.

Key Findings in Objective (IFEval) Settings

Persistence of SPB Under Objective Scrutiny

Strong evidence refutes the hypothesis that objective, programmatically verified rubrics eliminate SPB. Across all major families, judges exhibit HSPP ratios significantly >1 for self and same-family generations, peaking at 1.47 for GPT-5 (implying a 47% higher misattribution to own outputs). Most overestimation errors correspond to misclassification of true failures as ties rather than outright wins, indicating subtle extra leniency rather than severe favoritism.

Comparative Robustness: RB, DA, and PWC

SPB is present across all paradigms but is most acute in PWC, somewhat lower in DA, and lowest in rubric-based evaluation (both SR and AR). The level of bias between DA and RB is not statistically significant in practice but both remain far preferable to PWC for reliable benchmarking among comparable models.

Aggregate Judges and SPB Mitigation

Majority-vote ensembles of diverse judges consistently reduce HSPP ratios but residual bias remains even at the committee level; the effect is especially clear for highly self-favoring models. Furthermore, filtering out rubrics with low inter-judge agreement is a practical lever for reducing SPB, empirically supporting the use of rubric agreement as a metric for evaluation reliability.

Key Findings in Subjective (HealthBench) Settings

Amplification and Real-World Impact of SPB

With subjective rubrics, SPB continues to manifest robustly and materially distorts system-level scores in head-to-head model comparisons. Family-level effects are quantifiable, with some observed self-overestimation as large as 10 points (out of 100) in Gemma-4B, while GPT-5 grants itself a 4-point margin against committee consensus—sufficient to alter model rankings at the frontier.

Rubric and Topic Sensitivity

The study dissects the drivers of subjectivity-induced SPB. Negative rubrics—those penalizing the presence of undesired qualities—consistently elicit higher self-preference compared to positive rubrics. Both very short and very long rubrics show a U-shaped susceptibility, pointing to the importance of rubric design and specification. Rubrics related to emergency referrals and communication quality—where subjective judgement or ambiguity is greatest—see marked boosts in HSPP ratios, whereas factual and accuracy-focused rubrics see lower bias.

Theoretical and Practical Implications

Evaluation and Benchmarking Practice

Self-preference bias persists despite improvements in evaluation objectivity, and while rubric-based evaluation is significantly less vulnerable, it does not provide immunity. The persistent and model-internal nature of SPB, including in committee and objective settings, argues that true elimination requires interventions beyond prompt or rubric engineering, possibly down to the level of model weights or inference-time control. Recursive self-improvement pipelines without robust mitigation risk "runaway overfitting" to models’ own subjective standards.

The work motivates the operational use of diverse, cross-family judge committees, strict rubric agreement filtering, explicit SPB reporting metrics, and careful design or exclusion of pathological rubrics in both benchmarking and RLHF/RBRL settings. Further, the implications extend to any scenario where LLMs produce a reward signal for themselves, from agentic retraining to automated system curation.

Open Directions

Future research will need to investigate techniques such as activation steering or adversarial training to directly erase or minimize SPB at evaluation and learning time [roytburg2025breaking]. Extending SPB studies to reward modeling, RLHF, and settings with continuous system self-improvement is critical. A systematic mapping between specific subjective axes and SPB amplitude could help guide safe benchmark construction for high-stakes domains.

Conclusion

This paper robustly demonstrates that self-preference bias is a resilient and quantitatively significant failure mode even for rubric-based LLM evaluation, across both objective and subjective tasks. While rubric-based evaluation is clearly more robust than holistic comparisons or direct assessments, state-of-the-art models as judges are up to 50% more likely to grant leniency to their own generations, and up to 10 points more generous with subjective rubrics. Model ensembling and rubric design offer partial but incomplete mitigations. SPB thus appears inherent to current LLMs and their evaluative use, demanding future work on structural and training-level bias correction to enable fair, safe, and trustworthy benchmarking for advanced LLMs.