Automatically Finding Reward Model Biases

Abstract: Reward models are central to LLM post-training. However, past work has shown that they can reward spurious or undesirable attributes such as length, format, hallucinations, and sycophancy. In this work, we introduce and study the research problem of automatically finding reward model biases in natural language. We offer a simple approach of using an LLM to iteratively propose and refine candidate biases. Our method can recover known biases and surface novel ones: for example, we found that Skywork-V2-8B, a leading open-weight reward model, often mistakenly favors responses with redundant spacing and responses with hallucinated content. In addition, we show evidence that evolutionary iteration outperforms flat best-of-N search, and we validate the recall of our pipeline using synthetically injected biases. We hope our work contributes to further research on improving RMs through automated interpretability methods.

• The paper's main contribution is an evolutionary search pipeline that automatically identifies reward model biases by contrasting RM rewards with LLM judge preferences.
• It employs hypothesis generation, counterfactual rewriting, and multi-objective optimization to quantify bias strength and statistically filter undesirable attributes.
• By uncovering known and novel biases in state-of-the-art models, the methodology offers actionable insights for improving alignment and ensuring safer LLM outputs.

Automated Discovery of Reward Model Biases in LLMs

Problem Statement and Motivation

Reward models (RMs) are foundational in the post-training of LLMs, particularly for aligning model outputs with human preferences via RLHF. Systematic failures of these reward models have emerged, with evidence showing that RMs can inadvertently favor undesirable textual attributes (e.g., verbosity, certain formatting choices, hallucinated content, sycophancy). Such preferences can lead to reward hacking, undermining alignment by incentivizing LLMs toward outputs that are preferred by the RM but not by humans. The paper proposes the research problem of automatically discovering reward model biases—systematic and undesirable RM preferences expressible in natural language.

Method: Evolutionary Search Pipeline for Bias Discovery

The core contribution is a black-box, evolutionary pipeline for uncovering RM biases, motivated by multi-objective optimization. The pipeline operates on two explicit criteria: (1) attributes that are rewarded by the RM, and (2) simultaneously disfavored by a robust LLM-as-judge, intended as a proxy for aligned human preference.

The pipeline comprises several stages:

1. Hypothesis Generation: An LLM examines sampled responses to user prompts (from diverse LLMs), together with their RM-assigned scores, and proposes candidate biased attributes that are more prevalent in high-reward outputs.
2. Counterfactual Pair Construction: For each attribute, minimal rewrite operations (using distinct LLM rewriters) are performed on generated responses to create pairs that differ only in the presence/absence of the candidate attribute.
3. Bias Quantification: The RM and the LLM judge are queried on these pairs to estimate the RM bias strength (mean reward difference) and judge winrate (fraction of pairwise preferences for the biased attribute). An attribute is marked as a bias if $R(A) > 0$ (RM prefers) and $J(A) < 0.5$ (judge disfavored).
4. Evolutionary Iteration: The most promising candidate attributes (near the Pareto frontier in the two-objective space) are mutated via LLMs, generating semantically related variants. This population is filtered, deduplicated, and iterated over multiple rounds to encourage discovery of rare and diverse biases.
5. Validation and Filtration: Statistical testing with bonferroni corrections is applied to select only significant and undesirable attributes, using a held-out validation set and multiple independent rewriting models to ensure robustness.

   Figure 1: Schematic of the evolutionary search pipeline, where each circle represents a population of candidate biases at different stages.

Implementation Details

• User Prompt Generation: Rather than relying on public datasets (which suffer from toxicity, lack of coverage, etc.), the authors handcraft 20 distinct prompt topics spanning diverse LLM use-cases and utilize LLMs to synthesize prompt clusters for each topic.
• Attribute Validation: Attribute presence is judged using an LLM or simple regex. Counterfactual pairs are produced by prompting various LLMs for minimal edits.
• Metrics: RM bias strength is computed as the expected per-prompt reward difference between attribute- and non-attribute-containing responses; judge bias as the preference winrate, forming the basis for multi-objective selection.
• Rewriter Diversity: To increase robustness, three separate LLMs from major providers are used for the rewrite step, and empirical analysis confirms that their rewrites are meaningfully correlated yet not identical (Figures 7, 8).

Figure 2: Significant positive correlation in bias estimation between distinct rewriter models confirms the robustness of the pipeline.

Figure 3: Q-Q plots demonstrate that reward distributions across rewriters track each other, supporting invariance.

Main Results: Biases Uncovered in Skywork-V2-8B

Applied to the state-of-the-art Skywork-V2-8B RM, a leading open-weight reward model, the pipeline recovers known formatting biases and surfaces novel, non-trivial biases, such as:

• Preference for responses with redundant whitespace (triple spaces between words, sometimes as a result of typo artifacts).
• Rewarding hallucinated quoted content and fabricated details in plausible-seeming but fictitious events.
• Favoring explanatory or hedging statements, seemingly “harmless” safety suggestions (e.g., always suggesting consultation with a mental health professional for distressing topics).
• Strong preferences induced by sycophantic or safety-washing templates.
• Subtle biases dependent on prompt topic: list formatting is only preferred in "how-to" prompts, not general queries.

  Figure 4: Example where the RM mistakenly prefers redundant whitespace, disagreeing with the LLM judge.

  

Figure 5: Pareto plot of validated candidate biases—bottom-right (more RM bias, less LLM judge bias) indicates more severe bias. Points from evolutionary search (multiple iterations) are denser and further on the Pareto frontier.

Notably, the evolutionary pipeline discovers a broader set of diverse, more severe biases than a single-pass “best-of-N” hypothesis approach, supported both visually by dominance in the Pareto space and numerically via a diversity-adjusted Pareto metric.

Figure 6: Even very subtle, low-effect injected biases are uncovered by the pipeline, indicating high recall.

Empirical Evaluation and Validation

• Recall Testing: Controlled injections of biases with varying prevalence and signal-to-noise ratios demonstrate that the LLM-based hypothesis generation is effective—correct attributes are surfaced even when their presence is low and noisy.
• Rewriter Consistency: Pairwise metric analysis confirms that bias measurements are stable across distinct rewriter LLMs.
• Counterfactual Validity: Manual rubric-based scoring and secondary classification tasks validate that the attribute rewrites are both precise and interpreted as intended by LLMs.

Figure 7: Rubric-based evaluation of counterfactual pairs—vast majority achieve near-perfect minimality and attribute localization.

Figure 8: Automatic semantic comparison between LLM-classified attributes and ground-truth confirms high rewrite fidelity.

Theoretical and Practical Implications

The evolutionary search pipeline represents a scalable, data- and domain-agnostic framework for automatically auditing reward model biases with minimal human involvement. Unlike prior work, it does not constrain its search to pre-specified attributes or failure modes, enabling the detection of unexpected, non-trivial RM preferences that may propagate to deployed LLMs. The demonstrated methodological robustness—cross-prompt, cross-model, statistically filtered—attests to its viability for model evaluation pipelines.

Practically, the work urges for routine automated RM audits to be incorporated into development cycles, especially as reward models are increasingly deployed as alignment proxies and in preference data synthesis (2602.15222). The discovered biases have direct implications for alignment failure, safety interventions, and can guide both mechanistic interpretability and data curation efforts.

Limitations and Future Directions

Key limitations include reliance on synthetic prompt clusters (not all real-world user distributions are captured), cost constraints imposed by judge LLM API calls, and the possibility that discovered RM biases may not always be exploited by LLMs optimized against RMs due to distributional gaps. Further, rewrites cannot guarantee full disentanglement of the candidate attribute from correlated features. Expanding to more realistic prompt datasets, integrating more agentic bias-finding agents, and investigating reward model class-invariant or causal bias measurement are logical next steps.

Conclusion

This work introduces a novel, scalable methodology for uncovering systematic, natural-language-describable biases in reward models by leveraging an iterative evolutionary search pipeline. It robustly surfaces both known and previously unrecognized biases in state-of-the-art RMs, suggesting that even high-performing open models remain susceptible to subtle, undesirable failure modes. The generality and automation of the approach position it as a crucial tool for future RM and LLM development, and offer actionable targets for bias mitigation and interpretability research.