RMCB: Confidence Estimation Benchmark

• RMCB is a framework that assesses the reliability of reasoning models by measuring discrimination (AUROC) and calibration (ECE) in multi-step outputs.
• It segments reasoning traces into coherent chunks with annotated correctness, enabling detailed evaluation of both intermediate results and final answers.
• Empirical results reveal trade-offs among various estimation methods, emphasizing the role of reflective, ‘slow thinking’ strategies in improving model calibration.

Reasoning Model Confidence estimation Benchmark (RMCB) is a standardized framework for evaluating the reliability of large reasoning models, particularly their ability to quantify and communicate the uncertainty in their multi-step outputs. RMCB operationalizes confidence estimation in the context of chain-of-thought (CoT) reasoning, providing both comprehensive datasets and rigorous metrics to assess discrimination (the ability to rank correct and incorrect responses) and calibration (the alignment of verbalized confidence with empirical correctness) for high-stakes tasks across law, medicine, finance, mathematics, and general reasoning domains (Khanmohammadi et al., 13 Jan 2026). This benchmark systematically exposes the limitations of conventional methods relying solely on self-reports or token-level proxies, and delivers a testbed for next-generation confidence estimation architectures, training objectives, and evaluation protocols.

1. Motivation and Definitional Scope

RMCB was developed in response to the observation that existing large reasoning models frequently issue high self-confidence on incorrect answers and low confidence on correct ones, undermining their deployability in high-risk settings. Previous evaluation protocols—centered on classification or simple generation—do not scale to structured, long-form outputs with intrinsic dependencies and intermediate results (Damani et al., 22 Jul 2025, Khanmohammadi et al., 13 Jan 2026). RMCB aims to address three key axes:

• Constructing a large benchmark with ground-truth correctness labels at both final answer and intermediate step levels
• Comparing modern representation-based confidence estimation methods, including sequential, graph-based, and text-based models
• Quantifying persistent trade-offs between discrimination (AUROC) and calibration (ECE), and identifying whether any method achieves state-of-the-art on both

The benchmark explicitly supports both qualitative confidence (e.g., persistence under reconsideration) and quantitative measures (e.g., self-reported numerical or categorical scores, token-level probabilities), expanding the scope beyond simple answer-level correctness (Pawitan et al., 2024).

2. Benchmark Construction: Datasets, Trace Segmentation, Annotation

RMCB comprises a corpus of 347,496 reasoning traces from six open-weight LRMs (ranging from 3.85B to 32.8B parameters), spanning critical domains such as GSM8K (math), TAT-QA (finance), MedQA (clinical), LEXam (legal), ARC, CommonsenseQA2, LogiQA, OpenBookQA, QuaRTz, ReClor (general), and high-stakes evaluation datasets including MATH, FinQA, MedMCQA, LegalBench, MMLU-Pro, and BBH (Khanmohammadi et al., 13 Jan 2026). Each model’s output is segmented into coherent “chunks”—units of thought commonly ending in intermediate results—using an expanded keyword list (“wait,” “alternatively,” “on second thought,” etc.). Correctness labels (1/0/null) are assigned to each chunk and the final answer via a standardized automated annotation protocol using a judge model (GPT-5-nano). This yields fine-grained supervisory signals for evaluating both step-wise and final predictions.

3. Confidence Estimation: Methods and Architectures

RMCB evaluates more than fifteen distinct estimators:

Baselines:

• YVCE (“verbalized confidence”): Model self-report on 10-class verbal scale, mapped to [0,1]
• P(IK): Probing of the hidden state after prompt to estimate “I-Know” probability
• PHSV: Per-chunk MLP classifier trained on hidden states
• TLCC: Token-level uncertainty features (e.g., top-1 probability, entropy), aggregated per chunk

Sequential/Graph Models:

• SFHS: Stacked final hidden states modeled via MLP, Conv1D, or biLSTM
• GNN-SB (Sequential-Binary): Each chunk as node, edges as step transitions; GAT, GCN, GraphSAGE backbones
• GNN-SR (Relational): Fully connected DAG with edge features (NLI entailment, proximity, cosine similarity)

Text-Encoder Methods:

• ETTIN: Full trace as raw text input, mean-pooled embeddings through MLP
• ETTIN-HGA: Hierarchical Gated Attention on chunk-specific text, multi-head attention, auxiliary chunk correctness prediction

Each estimator is tuned for AUROC and (1-ECE) composite objective, with constraints on trainable parameter count (≤3.2M) and minimum sensitivity/specificity (Khanmohammadi et al., 13 Jan 2026). Methods relying on token-level logits or single hidden state probes consistently underperform those that model structural or semantic characteristics of the reasoning chain.

4. Evaluation Metrics and Protocols

RMCB standardized four primary metrics:

Metric	Formula	Role
AUROC	Probability correct ranked above incorrect (threshold-free)	Discrimination
ECE	$$\sum_{j=1}^b \frac{|B_j|}{n}\!\left|\mathrm{conf}(B_j)-\mathrm{acc}(B_j)\right|$$	Calibration
Brier Score	$\frac1n\sum_{k=1}^n(p_k - o_k)^2$	Calibration
Accuracy	$\frac1N\sum_{i=1}^N \mathbbm{1}_{[\,y_i\equiv y_i^*\,]}$	Task performance

AUCPR, F1, and specificity are also reported for binary prediction tasks. Calibration performance is visualized with reliability diagrams (accuracy vs. confidence bin), crucial for revealing systematic overconfidence in high-probability bins despite low empirical correctness (Damani et al., 22 Jul 2025, Khanmohammadi et al., 13 Jan 2026).

5. Empirical Findings: Trade-offs, Model Complexity, Behavioral Insights

RMCB evidence indicates a persistent trade-off: text-based encoders yield best discrimination (ETTIN AUROC=0.672), while structurally-aware models (ETTIN-HGA, GNN-SB-GCN) deliver superior calibration (ETTIN-HGA ECE=0.148) (Khanmohammadi et al., 13 Jan 2026). No single architecture dominates both, and complexity does not guarantee better performance. On MedMCQA, calibration universally degrades across models, suggesting sensitivity to domain characteristics.

Behavioral analysis reveals that reasoning models equipped with extended CoT and “slow thinking” behaviors (alternatives, verification, backtracking) dynamically adjust their verbalized confidence, leading to progressive improvements in calibration as the chain unfolds. Non-reasoning models, or those lacking slow thinking, demonstrate coarse or overconfident predictions (Yoon et al., 20 May 2025). In-context learning with slow-thinking exemplars enables calibration gains even for baseline models, suggesting that dynamic, reflective behaviors—not proprietary architectures—are the primary drivers of improvement.

6. Protocols for Measuring and Calibrating Confidence

RMCB formalizes protocols for both qualitative and quantitative confidence estimation (Pawitan et al., 2024, Damani et al., 22 Jul 2025):

• Qualitative confidence: Persistence under reconsideration (the probability the answer remains unchanged after a “rethink” prompt)
• Quantitative confidence: Self-reported score (0–100%) or categorical bins, supported by token-level probability and betting odds
• Test-time ensembling: Confidence-weighted majority vote across multiple sampled answers improves both accuracy and calibration
• Multi-stage introspective UQ: Second-stage (“reader”) models review CoT traces to detect flaws and revise confidence estimates, yielding further reductions in ECE on challenging benchmarks (Mei et al., 22 Jun 2025)

Explicit reporting of conditional accuracy (correct if kept/change), session resetting to avoid spurious persistence, and prompt variation are recommended to foster protocol robustness.

7. Current Limitations and Future Directions

Despite the breadth and depth of RMCB, the following limitations and research areas are identified (Yoon et al., 20 May 2025, Yuan et al., 27 Feb 2025, Mei et al., 22 Jun 2025):

• All primary signals for calibration rely on chunk-level hidden states, with an observable performance ceiling
• Data skew toward QA and math reasoning; other domains (dialogue, code generation) are less represented
• Discretization artifacts from coarse confidence bins may obscure calibration at fine-grained scales
• Absence of statistically significant improvement with larger or more recent model architectures
• Open challenges in defining ground-truth confidence in domains lacking large-scale human forecast aggregation (e.g., FOReCAst’s reliance on Metaculus)

Future RMCB design recommendations include stratification by task difficulty, reporting multi-granularity calibration at intermediate steps, benchmarking introspective modules, standardization on proper scoring rules (Brier, log-loss), and public leaderboards with open-source evaluation pipelines. The field urgently requires new methods leveraging richer generative and structural cues to surpass current representation-based paradigms (Khanmohammadi et al., 13 Jan 2026, Damani et al., 22 Jul 2025).