Rethinking Reward Model Evaluation: Are We Barking up the Wrong Tree?

Abstract: Reward Models (RMs) are crucial for aligning LLMs with human preferences. Currently, the evaluation of RMs depends on measuring accuracy against a validation set of manually annotated preference data. Although this method is straightforward and widely adopted, the relationship between RM accuracy and downstream policy performance remains under-explored. In this work, we conduct experiments in a synthetic setting to investigate how differences in RM measured by accuracy translate into gaps in optimized policy performance. Our findings reveal that while there is a weak positive correlation between accuracy and downstream performance, policies optimized towards RMs with similar accuracy can exhibit quite different performance. Moreover, we discover that the way of measuring accuracy significantly impacts its ability to predict the final policy performance. Through the lens of the Regressional Goodhart effect, we recognize that accuracy, when used for measuring RM quality, can fail to fully capture the potential RM overoptimization. This underscores the inadequacy of relying solely on accuracy to reflect their impact on policy optimization.

• The paper reveals a weak positive correlation between RM accuracy and downstream policy performance, questioning the effectiveness of current metrics.
• Methodological variations in measuring accuracy lead to significant performance differences among policies optimized for similar reward models.
• By applying Regressional Goodhart’s effect, the study demonstrates how external variables dilute the predictive power of accuracy in guiding model alignment.

The paper "Rethinking Reward Model Evaluation: Are We Barking up the Wrong Tree?" discusses the critical role of Reward Models (RMs) in aligning LLMs with human preferences. The authors focus on the traditional method of evaluating RMs, which involves measuring accuracy against a manually annotated preference validation set. They argue that this evaluation approach is limited and does not fully account for the relationship between RM accuracy and the performance of the downstream policies that are optimized using these RMs.

The key insights from the paper include:

1. Weak Correlation: The authors find a weak positive correlation between RM accuracy and downstream policy performance. This suggests that while higher accuracy might indicate better alignment with human preferences, it does not necessarily correspond to improved performance in policy application.
2. Performance Variability: Policies optimized for RMs with comparable accuracy levels can show significant differences in performance. This implies that accuracy alone does not capture all the nuances that affect the quality and effectiveness of RMs in practical implementations.
3. Impact of Accuracy Measurement: The way accuracy is measured significantly influences its predictive power regarding the final policy performance. This casts doubt on the adequacy of using accuracy as a standalone metric for RM evaluation.
4. Goodhart's Effect: Through the framework of Regressional Goodhart's effect, the study highlights the influence of exogenous variables that affect the relationship between RM quality (as measured by accuracy) and the capability of the resulting policy models. This suggests that external factors can skew the perceived effectiveness of RMs in guiding policy optimization.

The authors emphasize that relying solely on accuracy to evaluate RMs is insufficient. They suggest that understanding and addressing external variables and the broader context in which RMs operate is essential for improving their evaluation and, consequently, the effectiveness of policies optimized using these models. This work encourages a re-examination of current evaluation practices and calls for more comprehensive metrics that better capture the complexity of RM application in LLMs.