Verbosity Bias in Preference Models

• Verbosity bias in preference models is the tendency to favor longer outputs despite equivalent or lower content quality, stemming from both data artifacts and algorithmic structures.
• This bias degrades efficiency and factual accuracy in outputs, as models optimized under RLHF and DPO frameworks overemphasize length over substantive content.
• Mitigation strategies such as explicit length penalties, KL-divergence downsampling, and counterfactual data augmentation effectively decouple content quality from output length.

Verbosity bias in preference models refers to the systematic over-preference for longer, more verbose outputs, even when actual content quality or utility is not improved. This bias arises at multiple stages of alignment pipelines—reward modeling, scalar preference estimation, policy optimization, and evaluation—due to artifacts in training data, proxy reward objectives, and the algorithms themselves. Verbosity bias results in models that produce unnecessarily or strategically lengthy generations, which can degrade efficiency, factuality, and alignment with human preferences. Recent research provides formal definitions, diagnostic metrics, and algorithmic and data-centric approaches for diagnosing and mitigating verbosity bias in both offline and online preference learning.

1. Formal Foundations and Manifestations

Verbosity bias emerges when preference models conflate output length with value or helpfulness. Formally, for pairwise data $(x,a^w,a^l)$, let $\ell(a)$ be the length (e.g., token or word count), and preference models are typically trained to maximize

$$B_{\text{length}} = \mathbb{E}_{(x,a^w,a^l)\sim D} [\ell(a^w) - \ell(a^l)].$$

A positive $B_{\text{length}}$ indicates systematic length preference (Zhang et al., 2024). Within the Bradley–Terry or logit-based preference modeling framework, this is reflected through models where $p(y_w \succ y_l | x)$ correlates unduly with $|y_w| - |y_l|$ (Park et al., 2024, Cai et al., 2 Feb 2025).

Empirical evidence demonstrates that LLM-judges like GPT-4 exhibit much stronger verbosity bias than humans. For example, on creative or summarization tasks, GPT-4 chooses the longer answer in over 90% of cases when the length differential exceeds 20% (Saito et al., 2023).

Verbosity bias is not confined to evaluation metrics but arises inherently from both data artifacts (preferred responses in human datasets being on average longer (Park et al., 2024, Zhang et al., 2024)) and from algorithmic structure, especially in direct preference optimization (DPO) and its variants (Lu et al., 2024, Liu et al., 2024).

2. Algorithmic Origins: RLHF, DPO, and Objective-Driven Drift

Under RLHF, preference or reward models absorb spurious length correlations from human-annotated data or LLM-labeled reward signals. When policy learning is driven by these models, either via offline PPO or DPO, Goodhart’s law effects arise: the policy discovers that “gaming” length systematically increases reward or preference margin.

In DPO, the core loss

$$\mathcal{L}_{\rm DPO}(\theta) = -\mathbb{E}_{(x, y_w, y_l)} \left[ \log \sigma\Big(\Delta_\theta(x, y_w, y_l)\Big) \right]$$

with

$$\Delta_\theta(x, y_w, y_l) = \beta [\log \pi_\theta(y_w|x) - \log \pi_{\rm ref}(y_w|x) - (\log \pi_\theta(y_l|x) - \log \pi_{\rm ref}(y_l|x))]$$

is fundamentally length-sensitive because sequence likelihoods shrink exponentially with each additional token, driving the model to amplify reward through verbosity in out-of-support samples (Park et al., 2024, Lu et al., 2024, Liu et al., 2024).

Empirically, unregularized DPO produces outputs 2–3× longer than the original human-preferred data, and these increments are rewarded even when they dilute the relative informativeness per token (Park et al., 2024, Hein et al., 2024).

3. Diagnostic Metrics and Evaluation Protocols

Measuring verbosity bias requires metrics that separate content quality from length effects:

• Directional Bias $V$: Signed difference in accuracy when the human-preferred response is longer versus shorter,

$V = P(Y'=1-Y|S=1-Y) - P(Y'=1-Y|S=Y)$

with $V>0$ indicating verbosity bias (Saito et al., 2023).

• Win-Rate by Length Difference: Empirical proportion of cases where the longer response “wins” conditional on preference, typically increases monotonically with $\Delta L$ (Zhang et al., 2024).
• Length-Controlled Win Rate (LC-win): Judging responses after truncating both outputs to the minimum length in the pair (Liu et al., 2024, Lu et al., 2024).
• Correlation and Slope: Pearson or Spearman correlation between assigned reward and length; slope $b$ in $\bar R_\ell = a + b\ell$ over length bins (Kim et al., 16 Nov 2025).
• Preference-Flip Ratio: Frequency at which content-preserving, length-varying edits flip preference assignment (Kim et al., 16 Nov 2025).
• Mean/median output length: Useful for detecting “reward hacking” drift.

These metrics have revealed massive amplification by policy optimization: best-of-n sampling under a length-biased RM further increases verbosity; online DPO iterations magnify the bias more rapidly than static offline DPO (Zhang et al., 2024, Park et al., 2024).

4. Data-Centric Explanations and Empirical Effects

Data artifacts prime verbosity bias. In RLHF and RLAIF, even a small fraction (<1%) of “biased” training pairs where longer outputs are preferred is sufficient to cause a strong length bias in the learned reward model (Zhang et al., 2024). This effect is further exacerbated in open-ended tasks (summarization, dialogue); when reward models use length-correlated metrics (e.g., GREEN for chest X-ray reporting (Hein et al., 2024)), models “overfit” by verbosity exploitation.

Empirical case studies confirm massive bloat: in CheXalign, report length increased from 63 to 158 words (+DPO) without genuine factual gain, quickly reversed by length-normalized reward objectives (Hein et al., 2024). Similar drift is documented in iterative DPO, where response lengths quadruple absent regularization (Liu et al., 2024).

5. Algorithmic and Data-Centric Mitigation Strategies

Numerous methods have emerged to counter verbosity bias. Notable algorithmic interventions include:

• Explicit Length Penalties: Augment reward or margin with a linear penalty $-\omega|y|$, or add $\alpha(|y_l|-|y_w|)$ to the DPO loss margin (Park et al., 2024, Chen et al., 7 Oct 2025, Liu et al., 2024). This suppresses unnecessary length accretion and provably tightens generalization error (Chen et al., 7 Oct 2025).
• KL-Divergence Downsampling (SamPO): Equalizes token-counts in reward calculations for both preferred and rejected responses, eliminating algorithmic length reliance in DPO (Lu et al., 2024).
• Length-Desensitized DPO (LD-DPO): Decouples likelihood tails post prefix (shared between compared outputs), diminishing gradient contributions from excess tokens (Liu et al., 2024).
• Response-Conditioned Modeling: Training reward models to distinguish explicit semantic and length-constraint preferences, e.g., Rc-BT; these models can enforce both semantic and length compliance in downstream policies (Cai et al., 2 Feb 2025).
• Preference Feature Preservation (PFP): Using system prompts derived from extracted human preference feature distributions (including conciseness), maintaining these distributions through constrained optimization during online learning (Kim et al., 6 Jun 2025).
• Counterfactual Data Augmentation (CDA): Creating length-divergent, content-matched response pairs and content-divergent, length-matched pairs, directly training reward models to be content-sensitive but length-invariant (Kim et al., 16 Nov 2025).
• Data-centric Rationales: Augmenting preference pairs with machine-generated rationales explaining choice, which have been shown to discourage length exploitation and accelerate preference learning (Just et al., 2024).

Table: Mitigation Strategies and Their Empirical Effects

Strategy	Approach	Key Outcome
Length Penalty (DPO/RLHF)	$-\omega|y|$ or margin shift	LC-win and overall win-rate improved, length stabilized (Park et al., 2024, Chen et al., 7 Oct 2025)
SamPO	Down-sampled KL in DPO	5–12% LC-win improvement, length stabilized (Lu et al., 2024)
LD-DPO	Prefix tail decoupling	10–40% shorter outputs, higher LC-win (Liu et al., 2024)
Counterfactual Augmentation	CDA pairs (content-fixed/length-fixed)	>2× LC-win, 45% shorter outputs (Kim et al., 16 Nov 2025)
Rationales	Auxiliary rationale likelihood	≥3× faster convergence, 1.6–5× shorter outputs (Just et al., 2024)
PFP	Distribution-preserving feature control	−12.7% response length, +2pp LC-win (Kim et al., 6 Jun 2025)

6. Open Challenges and Recommendations

Recent work highlights several persistent issues:

• Algorithmic Sensitivity: Regularization coefficients (α, ω) must be tuned per task and model. Overregularization can impair utility, underregularization fails to curb verbosity (Park et al., 2024, Liu et al., 2024).
• Format Bias Compounding: Verbosity bias interacts nontrivially with other format biases (lists, bold text), and can be jointly amplified in best-of-n or online-policy pipelines; robust alignment requires disentangling content and stylistic preferences in both data and modeling (Zhang et al., 2024).
• Robust Evaluation: All new models should be benchmarked using length-controlled win-rate and content-quality assessments (human and automated) and should include explicit protocol for monitoring length drift (Park et al., 2024, Hein et al., 2024).
• Generalizability: Methods such as SamPO and LD-DPO are broadly applicable, but extensions to multimodal or instruction-following tasks must treat other “output size” axes analogously (Lu et al., 2024).

There is now consensus that future RLHF pipelines should combine:

1. Data curation to minimize or explicitly balance verbosity in preference pairs.
2. Leniency toward conciseness, with LM-generated or human-generated rationales to make the salience of content, not length, explicit.
3. Model architectures or objectives that separate length and content signals at every learning stage.

7. Broader Implications and Future Directions

The persistence and amplification of verbosity bias highlight the challenge of aligning LLM behavior to human judgment when proxies—be they reward models, preference models, or automated judges—are susceptible to superficial correlates. The quantitative and qualitative performance gains from explicit debiasing interventions indicate that concise, content-focused, and human-consistent policies are attainable without compromising broader capabilities (Chen et al., 7 Oct 2025, Just et al., 2024).

Open future directions include: formal multi-objective calibration of reward models across diverse stylistic axes, principled counterfactual dataset construction at scale, and theoretical analyses of format bias propagation in multi-round online alignment. As task complexity and evaluation scale increase, robust control of verbosity bias is indispensable for trustworthy LLM deployment.