JudgeBench: Evaluating LLM Judges

• JudgeBench is a comprehensive benchmark suite designed to objectively evaluate LLM-based judges by testing their ability to distinguish factually correct responses from subtle errors.
• It employs a rigorous pipeline combining automated checks and dual LLM verification across diverse domains including knowledge, reasoning, mathematics, and coding.
• Benchmark results reveal significant performance gaps in LLM judges, underscoring the challenges of achieving reliable, objective evaluation in complex response scenarios.

JudgeBench is a benchmark suite for evaluating LLM-based judges, focusing on the objective assessment of challenging response pairs across knowledge tasks, reasoning, mathematics, and code verification. Developed to address deficiencies in earlier benchmarks dominated by human preference or superficial correctness metrics, JudgeBench applies a rigorous construction pipeline for generating high-difficulty response pairs with objectively verifiable ground-truth, providing a reliable substrate for the quantitative evaluation and comparison of LLM “judge” models (Tan et al., 2024, Li et al., 23 Apr 2025).

1. Definition and Purpose

JudgeBench is a purpose-built test suite designed to measure the reliability and limits of LLM-based judges—the automated agents tasked with adjudicating the correctness of model-generated answers to demanding problems. The central premise is the evaluation of models not on generic preference alignment, but on their capacity to distinguish outputs that are factually and logically correct from those that harbor subtle errors. JudgeBench operationalizes this by presenting each judge with (A, B) response pairs distilled from upstream ground-truth datasets, with labeling that reflects verifiable correctness rather than subjective labeler preference (Tan et al., 2024).

2. Dataset Construction and Composition

JudgeBench draws response pairs from three principal sources:

• MMLU-Pro: College-level multiple-choice questions spanning diverse knowledge domains.
• LiveBench: Reasoning and mathematics tasks, including Big-Bench Hard, AMC12, USAMO, and zebra puzzles.
• LiveCodeBench: Challenging coding problems from platforms such as LeetCode, CodeForces, and AtCoder.

The construction pipeline is as follows:

1. Sample $k\geq4$ responses per question from a strong LLM (e.g., GPT-4o) under greedy decoding.
2. Conduct correctness checks via both automated means (e.g., regex for multiple-choice, test suite execution for code) and an auxiliary LLM-based verifier (GPT-4o-mini), discarding inconsistent or ambiguous cases.
3. Exclude questions where all responses are equivalent (all correct or all wrong).
4. Randomly select one correctly-verified and one incorrectly-verified candidate response to form a binary labeled pair.

This results in a total of 350 adversarial-style pairs, stratified as shown:

Category	Number of Pairs
Knowledge	154
Reasoning	98
Mathematics	56
Coding	42

There is no train/dev/test split; JudgeBench is a single holdout evaluation set (Tan et al., 2024, Li et al., 23 Apr 2025).

3. Labeling, Gold Standard, and Pair Format

Each JudgeBench instance is labeled as consisting of one objectively correct and one objectively incorrect response. Labeling strictly uses dataset ground-truth and dual-verification: correctness requires the candidate to pass both the automated check and LLM-based secondary verification, eliminating reliance on subjective human preference labels. Pairs where verifiers disagree or where neither response can be unequivocally labeled are omitted.

The data is presented as tuples: (pair_id, question, answer₁, answer₂, label ∈ {A, B})

For coding items (drawn from LiveCodeBench), unit test suites or canonical output matching are the decisive criteria; for other domains, answer format normalization ensures consistency of automated assessment (Tan et al., 2024).

4. Evaluation Protocol and Metrics

The benchmark employs a double-trial protocol to minimize position bias:

1. Each judge is presented first with (A, B), then with (B, A).
2. Both trials must correctly prefer the ground-truth winner (or allow a tie in one trial without contradiction) to score as accurate.

The principal metric is accuracy: $$\mathrm{Acc} = \frac{1}{N} \sum_{i=1}^{N} \mathbf{1}[\hat{y}_i = y_i]$$ where $N$ is the total number of pairs, $\hat{y}_i$ the judge model verdict, and $y_i$ the ground-truth label.

This protocol penalizes inconsistent or position-sensitive judgments, thus providing a robust measure of verifier-level discriminative ability. Reporting is subdivided by domain for detailed error analysis (Tan et al., 2024).

5. Benchmarking Results and Analysis

JudgeBench’s results reveal a significant gap between the performance of current LLM-based judges and the demands of objective verification:

• Prompted Judges: GPT-4o with a vanilla (AlpacaFarm-style) prompt achieves only 44.2% (Knowledge), 48.0% (Reasoning), 66.1% (Math), 61.9% (Coding), and 50.9% (Overall)—barely above random guessing except in mathematics.
• Arena-Hard (LMSYS pipeline): Boosts performance modestly (overall 56.6%), with math as the easiest category.
• Fine-Tuned Judges: Preference-optimized models such as PandaLM (LLaMA-7B) can perform below random in knowledge and reasoning due to context truncation and ties.
• Reward Models: Skywork-Reward-Gemma-2-27B and InternLM2-20B-Reward achieve 60–64% overall, competitive with top commercial models.
• Multi-Agent Systems: ChatEval (two GPT-4o agents debating) performs subpar (∼34% overall).
• Top Baseline: The o1-preview model reaches 75.4% overall, with 85.7% in math and coding.

Judges consistently struggle with tasks requiring knowledge or subtle reasoning, while mathematical correctness is relatively more tractable (Tan et al., 2024).

Model	Knowledge	Reasoning	Math	Coding	Overall
GPT-4o (vanilla)	44.2%	48.0%	66.1%	61.9%	50.9%
Arena-Hard (GPT-4o)	50.7%	54.1%	75.0%	59.5%	56.6%
Skywork Reward (Gemma)	59.7%	66.3%	83.9%	50.0%	64.3%

Ablation confirms that the ability to verify is nearly as difficult as generation itself; model judge accuracy closely tracks the accuracy of model solvers (Tan et al., 2024).

6. Meta-Evaluation on JudgeBench

Recent frameworks apply meta-judging, leveraging multiple LLMs and comprehensive rubrics to further evaluate or select trustworthy LLM-judge outputs (Li et al., 23 Apr 2025). For instance, a three-stage pipeline (rubric construction, multi-agent scoring, threshold filtering) yields a 15.55% increase in precision over raw (unfiltered) judgments and 8.37% over the best single-agent baseline. Weighted averaging, majority voting, and "panel discussion" (inter-agent debate) strategies are explored for meta-judge aggregation.

In these studies, the precision metric is:

$$\text{Precision} = \frac{\text{TP}}{\text{TP} + \text{FP}}$$

where TP = true positives (correct meta-judgments), FP = false positives. The preferred aggregation approach is contextually dependent (e.g., reasoning questions favor averaging, mathematics favor debate-based discussion), but overall, multi-agent frameworks enhance the reliability of judgment selection (Li et al., 23 Apr 2025).

7. Applications, Usage, and Impact

JudgeBench serves as a canonical benchmark for:

• Comparing prompted, fine-tuned, or reward-modeled LLM judges.
• Stress-testing judges on tasks that defeat simple human-preference or instruction-following baselines.
• Measuring improvements from prompting, model scaling, or meta-judging pipelines.

Recommended usage involves downloading the suite from the official repository and evaluating judge models via double-trial protocol, reporting overall and per-domain accuracy. The data and protocol have been widely adopted in subsequent LLM-judge and meta-judge research.

By providing an adversarial, objectively labeled, and domain-diverse testbed, JudgeBench fills a critical methodological gap: tracking progress toward verifier-level, robust, and scalable model evaluation in a landscape where LLMs themselves are increasingly used as autonomous scientific and technical critics (Tan et al., 2024, Li et al., 23 Apr 2025).