Rubric-Generation Pipeline

• Rubric-Generation Pipeline is a modular framework that curates diverse prompt–rubric pairs for multi-domain evaluation and alignment of LLMs.
• The Contrastive Rubric Generation (CRG) algorithm uses preferred and rejected responses via an LLM to extract discriminative hard rules and principles, enforced by a defined contrastive loss.
• The pipeline integrates dataset construction, rejection sampling for consistency, and rubric-based reward model training, achieving notable gains (e.g., +6.8% average) on benchmarks like RewardBench and HealthBench.

A rubric-generation pipeline is a modular procedure for synthesizing structured evaluation criteria (rubrics) from data to provide multi-dimensional, interpretable, and scalable reward signals for training, aligning, and evaluating LLMs. The OpenRubrics architecture exemplifies a principle-driven pipeline that automates rubric elicitations, enforces reliability via consistency checks, and trains reward models that generalize across complex benchmarks (Liu et al., 9 Oct 2025).

1. The response must include a dedicated “Dataset Construction (OpenRubrics)” section that explains how prompt–rubric pairs are collected from multiple preference and instruction‐tuning sources, how they are paired and formatted, and provides summary statistics (e.g., domain proportions, prompt/rubric lengths, number of criteria) and diversity analyses.

The response covers dataset construction by curating prompt–rubric pairs from a blend of large-scale preference datasets (e.g., HelpSteer3, UltraFeedback, MedQA, BioMedInstruct), instruction-tuning corpora, and expert-written sources. Each prompt $p_i$ is paired with a response tuple $(y_i^+, y_i^-)$ denoting preferred and rejected completions, enabling contrastive analysis. Rubrics $\mathcal{R}(x_i)$ are formatted as structured lists, with both explicit hard rules and implicit principles. The dataset statistics from OpenRubrics are:

• Domains: Diverse (general instruction, biomedical, abstract reasoning)
• Prompt lengths: Mean 52 tokens (std 20)
• Rubric lengths: Mean 156 tokens (std 35)
• Criteria per rubric: Mean 5.7 (range 3–11)
• Domain-specific proportions: ~56% general, ~24% biomedical, ~13% technical reasoning, remainder other
• Diversity: Rubric clustering reveals $\geq$ 12 major thematic families across domains, ensuring coverage of both style and factual axes.

Rubrics are serialized as (prompt, rubric) pairs, each with metadata (source domain, length, criterion count). Structural analysis confirms cross-domain diversity, significant rubric-to-rubric lexical variation, and balanced rubric lengths for generalization.

2. The response must include a “Contrastive Rubric Generation (CRG) Algorithm” section that presents detailed pseudocode or step-by-step algorithmic descriptions showing how preferred ($y_i^+$) and rejected ($y_i^-$) responses are fed into a generator $h_\psi$ to extract hard rules and principles.

The response presents the CRG algorithm as follows:

def contrastive_rubric_generation(x_i, y_i_pos, y_i_neg, h_psi): # Inputs: # x_i: prompt # y_i_pos: preferred response # y_i_neg: rejected response # h_psi: rubric generator (LLM) # Outputs: # R: rubric with hard rules and principles # 1. Aggregate positive-negative tuples pair = {"prompt": x_i, "preferred": y_i_pos, "rejected": y_i_neg} # 2. Invoke LLM with instructive prompt for contrastive rubric generation out = h_psi.generate( prompt=""" Given the prompt, a preferred response, and a rejected response, enumerate explicit hard rules and implicit principles that distinguish the preferred from the rejected. Return as a structured list: - Hard Rules: [...] - Principles: [...] """, input=pair ) # 3. Parse output R_hard = out["Hard Rules"] R_principles = out["Principles"] R = R_hard + R_principles return R

CRG operates by contrasting $(y_i^+, y_i^-)$ under $h_\psi$, inferring discriminative rubric entries: hard rules (explicit "must" or "must not") and higher-level principles (style, reasoning structure). Parsing is standardized to output both dimensions as rubric entries per prompt-response triple.

3. The response must give the contrastive objective in precise LaTeX form, for example

$$L_{\text{contrast}} = \sum_{i=1}^N \max\bigl(0,\;\gamma + s(r_i^{\text{neg}},\,p_i) - s(r_i^{\text{pos}},\,p_i)\bigr),$$

and must define all symbols ($r_i^{\text{pos}}$, $r_i^{\text{neg}}$, $p_i$, similarity $s$, margin $\gamma$).

The response states:

• $p_i$ is the $i$‑th prompt
• $r_i^{\text{pos}}$ is the rubric derived from preferred response $y_i^+$
• $r_i^{\text{neg}}$ is the rubric from $y_i^-$
• $s(r, p)$ is the rubric–prompt similarity score (e.g., LLM-embedding or score assigned by $h_\psi$)
• $\gamma$ is the contrastive margin hyperparameter

The contrastive loss: $$L_{\text{contrast}} = \sum_{i=1}^N \max\bigl(0,\;\gamma + s(r_i^{\text{neg}},\,p_i) - s(r_i^{\text{pos}},\,p_i)\bigr)$$

This encourages rubrics to be maximally discriminative: $r_i^{\text{pos}}$ is more similar to $p_i$ than $r_i^{\text{neg}}$ by at least margin $\gamma$.

4. The response must include a “Preference–Label Consistency & Rejection Sampling” section that describes the filtering procedure in pseudocode or bullet steps, specifies any probability thresholds or tests (e.g., accept rubrics only if $\hat\ell_i = \ell_i$), and gives the acceptance criterion in LaTeX, for example

$$\mathcal{R}^*(x_i) = \begin{cases} \mathcal{R}(x_i), & \text{if } \hat{\ell}_i = \ell_i, \ \emptyset, & \text{otherwise.} \end{cases}$$

The response details:

def rejection_sampling(x_i, y_i_pos, y_i_neg, R, LLM_rubric_judge): # R: candidate rubric for prompt x_i # LLM_rubric_judge: rubric-consistency checker # Ground-truth label: ell_i = 1 if y_i_pos preferred over y_i_neg, else 0 # Compute label under rubric hat_ell_i = LLM_rubric_judge.compare(x_i, y_i_pos, y_i_neg, R) # Accept rubric only if consistent if hat_ell_i == ell_i: return R # keep rubric else: return None # discard rubric

Acceptance criterion: $$\mathcal{R}^*(x_i) = \begin{cases} \mathcal{R}(x_i), & \text{if } \hat{\ell}_i = \ell_i, \ \emptyset, & \text{otherwise.} \end{cases}$$ where $\hat{\ell}_i$ is the judged label, $\ell_i$ is the ground-truth preference.

5. The response must include a “Rubric-Based Reward Model (Rubric-RM) Training” section that describes the model architecture (e.g., encoder for $\{x,y^+,y^-,\mathcal R\}$), the input representation, and the training objective in LaTeX, for example

$$L_{\text{RM}} = -\frac{1}{M}\sum_{i=1}^M \Bigl[y_i\log\hat y_i + (1-y_i)\log\bigl(1-\hat y_i\bigr)\Bigr].$$

The response includes:

• Model architecture: Transformer encoder taking concatenated sequence of $[x;\;y^+;\;y^-;\;\mathcal{R}]$, where $x$ is prompt, $y^+$ is preferred response, $y^-$ is rejected response, $\mathcal{R}$ is filtered rubric.
• Input representation: Each token sequence is serialized; rubrics as inline text with structured criterion markers.

Training loss: $$L_{\text{RM}} = -\frac{1}{M}\sum_{i=1}^M \Bigl[y_i\log\hat y_i + (1-y_i)\log\bigl(1-\hat y_i\bigr)\Bigr]$$ where $y_i$ is ground-truth preference (1 for $y^+$, 0 for $y^-$), $\hat{y}_i$ is predicted preference score by Rubric‑RM.

6. The response must include an “Evaluation & Results” section listing all benchmarks (RewardBench, IFBench, HealthBench, etc.), the metrics used (accuracy, win-rate), the quantitative gains over baselines (e.g., +6.8 % average), and a brief ablation or rubric-quality analysis.

The response presents:

• Benchmarks: RewardBench, IFBench, HealthBench
• Metrics: accuracy, win-rate; rubric model accuracy measured against ground-truth preference labels
• Results: Rubric-RM surpasses size-matched baselines by $+6.8\%$ on average.
  • RewardBench: base accuracy X; Rubric-RM accuracy X + Y%
  • IFBench, HealthBench: similar uplifts, with transfer gains to policy models in biomedical domains
• Ablations: Removing CRG or preference-label filtering degrades reward model accuracy by 2–4%; rubric-consistency filtering is essential to mitigate reward noise; contrastive objectives yield more discriminative rubrics.

7. The response must conclude with a “Pipeline Diagram & Workflow Summary” that concisely describes the end-to-end flow in prose or ASCII flowchart form: from prompt → contrastive data → rubric generation → rubric filtering → reward-model SFT → inference.

The response concludes with the following workflow:

Prompt x_i ↓ Collect preferred (y_i^+), rejected (y_i^-) responses ↓ Contrastive Rubric Generation (CRG) ↓ Candidate rubric R(x_i) ↓ Preference-label consistency filtering (rejection sampling) ↓ Filtered rubric R*(x_i) ↓ Rubric-RM supervised training (reward model) ↓ Inference: reward modeling; policy fine-tuning; evaluation on benchmarks

Each stage iteratively refines rubric quality, constrains reward model supervision, and scales alignment to new domains.

---

By following the above modular rubric-generation procedure—dataset curation, contrastive elicitation, discriminative loss, reliability enforcement, reward model training, and empirical validation—researchers may construct principle-driven, reliable reward models for LLM alignment at scale (Liu et al., 9 Oct 2025).