Superforecasters: Metrics and Methods

• Superforecasters are individuals with robust calibration and consistently low Brier scores, outperforming both naive aggregates and traditional experts.
• They use granular Bayesian reasoning, problem decomposition, and continuous recalibration to refine probabilistic forecasts across diverse domains.
• Empirical benchmarks from prediction tournaments demonstrate a 20–40% performance improvement over standard forecasting methods.

A superforecaster is an individual whose probabilistic estimates for real-world events consistently surpass both naive crowd aggregates and traditional domain experts, as quantified by strictly proper scoring rules. The term arose from large-scale prediction tournaments such as the Good Judgment Project, where a statistically stringent process identifies the most accurate forecasters by empirical metrics—chiefly, sustained low Brier scores, persistent relative skill, and robust calibration over hundreds of questions in diverse domains (Dardaman et al., 2023, Alur et al., 10 Nov 2025). The study of superforecasters combines quantitative evaluation, cognitive science, and advanced statistical aggregation, and has catalyzed research in both human judgment and AI-assisted forecasting.

1. Formal Definition and Identification Criteria

Superforecasters are characterized by exceptional performance in probabilistic forecasting competitions. In the IARPA Good Judgment Project (2011–2015), the top 2% of roughly 25,000 contributors met strict empirical thresholds: consistently low mean Brier scores—often below 0.12 on the core tournament set, placing them above the 90th percentile of all participants (including domain-expert teams) (Dardaman et al., 2023, Alur et al., 10 Nov 2025).

The Brier score, the principal metric, is defined for $N$ binary events as

$$\text{BS} = \frac{1}{N} \sum_{i=1}^N (f_i - o_i)^2,$$

where $f_i \in [0, 1]$ is the forecast probability and $o_i \in \{0, 1\}$ is the resolved outcome. Zero is perfect; 0.25 corresponds to perpetual 50/50 guesses. Superforecasters outperform crowd averages and display skill persistence—rankings based on past Brier performance predict future accuracy (Alur et al., 10 Nov 2025).

2. Performance Benchmarks and Empirical Results

Multiple large-scale studies demonstrate the empirical advantage of superforecasters. In the original IARPA tournament, superforecasters achieved Brier scores 20–30% lower than the median forecaster and typically 40% lower than naive or random models (Dardaman et al., 2023). On public platforms such as Good Judgment Open and Metaculus, elite teams realized Brier score reductions of 0.05–0.10 (10–25% improvement) versus the unfiltered crowd. Internal prediction markets in commercial settings (e.g., Google with 175,000 forecasts from 10,000 employees) corroborate a persistent 15–20% accuracy benefit for high-reputation forecasters (Dardaman et al., 2023).

Recent benchmarks formalizing superforecaster performance include ForecastBench and MarketLiquid, which aggregate hundreds to thousands of resolved events drawn from sources such as Metaculus, Polymarket, and Manifold. On ForecastBench (FB-Market, 76 questions), the superforecaster median achieved a Brier score of 0.0740 versus the public median of 0.1035 and the market consensus of 0.0965 (Alur et al., 10 Nov 2025). These performance gaps hold across topic categories (politics, macroeconomics, technology, sports).

3. Cognitive and Methodological Distinctions

Superforecasters have been extensively studied for cognitive style and methodological rigor. Characteristic traits include:

• Intellectual humility: treating beliefs as provisional, updating beliefs promptly with evidence.
• Granular Bayesian reasoning: incrementally adjusting probabilities in light of new information (“granular Bayesianism”) (Alur et al., 10 Nov 2025).
• Active open-mindedness: incorporating dissenting views, recognizing complexity and causal heterogeneity, decomposing problems into subproblems.
• Explicit calibration: regular scorecard-based feedback and correction of systematic bias (Dardaman et al., 2023).

A formalization of their methodology includes:

• Decomposition: breaking complex questions into analytically distinct subproblems.
• Reference-class forecasting: establishing baselines via analogous historical events.
• Continuous recalibration: quantifying and minimizing calibration error and maximizing resolution (the divergence of forecast bins from base rates).
• Tetlock’s “Ten Commandments”: systematic practitioner guidelines including triage, error-balancing, scenario mapping, and collaborative peer review (Dardaman et al., 2023).

4. Aggregation Techniques and Algorithmic Selection

Superforecasters are both identified and operationalized through statistical aggregation frameworks. Standard procedures involve:

• Tournament-based performance weighting: ranking forecasters via history-adjusted means or more advanced neural models.
• In deep neural aggregation, as proposed in "Deep Neural Ranking for Crowdsourced Geopolitical Event Forecasting" (Nebbione et al., 2018):
  • Each forecast is encoded with probabilistic vectors, self-reported confidence, topical embeddings (via LDA), and historic performance.
  • A Siamese neural network produces pairwise probabilities that forecaster $i$ is more accurate than $j$, inducing a ranking via INCR-INDEG weighted tournament sorting.
  • Aggregates are built using a cutoff percentile (e.g., top 10%), yielding lower mean Brier scores than simple historical-mean cutoffs, especially when incorporating topical and confidence metadata (ablation drops up to 8% and 5%, respectively).

Post-aggregation, extremizing procedures can further enhance crowd performance by pushing probabilities away from 0.5, particularly when historical analysis shows underreaction (Dardaman et al., 2023). Recent advances also demonstrate improved aggregation via ensemble and reconciliation systems, including agentic search and supervisor coordination in LLM-based forecasters (Alur et al., 10 Nov 2025).

5. Human vs. Machine Forecasting

Recent research systematically compares superforecasters against state-of-the-art LLM and AI-based systems on held-out event sets. The AIA Forecaster system, combining agentic search, supervisor reconciliation, and post hoc statistical calibration (Platt scaling/extremization), achieves performance statistically indistinguishable from the superforecaster median on ForecastBench datasets (e.g., Brier 0.0753 vs. 0.0740 for FB-Market) (Alur et al., 10 Nov 2025). However, in larger prediction markets (MarketLiquid, 1,610 questions), market consensus remains slightly superior, though blending AI and market predictions via simplex regression yields further improvements.

In "Evaluating LLMs on Real-World Forecasting Against Human Superforecasters" (Lu, 6 Jul 2025), twelve contemporary models were benchmarked on 464 binary events. The best-performing LLM attained a Brier score of 0.1352, exceeding the average Metaculus human crowd (0.149), but lagging well behind the superforecaster panel (mean Brier 0.1222, median 0.0196 on holdout). The gap is maintained across all seven question categories. Calibration analysis reveals that leading LLMs display overconfidence on high-probability events and underconfidence on certain narrative prompts.

A summary table illustrates comparative results (from (Alur et al., 10 Nov 2025)):

Benchmark	Public Median	Market Consensus	Superforecasters	SOTA LLM	AIA Forecaster
FB-Market (76 questions)	0.1035	0.0965	0.0740	0.1070	0.0753
FB-7-21 (498 questions)	0.1451	n/a	0.1110	0.1330	0.1076
FB-8-14 (602 questions)	0.1510	n/a	0.1152	0.1450	0.1099
MarketLiquid (1610 questions)	n/a	0.1106	n/a	0.1324	0.1258

All entries are Brier scores; lower is better.

6. The Prediction Tournament Paradox and Selection Implications

A critical statistical property in large forecasting tournaments is the "Prediction Tournament Paradox" (Aldous, 2019). Simulations reveal that, when the skill variance among contestants is moderate, the winner of a single tournament is often not drawn from the genuinely top-ranked forecasters but from mid-skill participants experiencing outlier luck. The expected winner's true rank can be as low as 80–120 out of 300 contestants, despite the use of proper scoring rules and repeated head-to-head superiority of highly skilled forecasters.

The underlying mechanism is the mean–variance trade-off: higher skill yields lower expected error but also smaller variance in total score, limiting the chance to "get lucky," whereas mid-tier contestants, with higher variance, occasionally outperform through random outcome sequences. Consequently, promoting individuals solely on single-tournament wins risks overfitting to noise. Remediation includes multi-tournament averaging and the use of calibration statistics to identify robust skill, rather than relying on leaderboard ranks from finite contests (Aldous, 2019).

7. Organizational Applications and Future Directions

Organizations can leverage superforecasting insights through structured talent identification, continuous feedback, and aggregation methodology (Dardaman et al., 2023). Key elements include:

• Running regular internal forecasting tournaments or using public platforms (e.g., Good Judgment Open, Metaculus) to surface high performers.
• Training in probabilistic reasoning, decomposition, and calibration.
• Using diverse small-team aggregation (“prediction pods”), performance weighting, and extremization techniques.
• Integrating quantitative forecasts into strategic decision-making cycles, with explicit calibration diagnostics for senior stakeholders.

Advances in forecasting methodology—as exemplified by neural aggregation (Nebbione et al., 2018), ensemble LLM pipelines (Alur et al., 10 Nov 2025), and market-AI hybrids—continue to narrow the performance gap between expert humans and machines. Open research questions include embedding forecaster interaction models, adaptation in low-data regimes, and jointly learning across domains.

In high-stakes and rapidly evolving domains, superforecasting methods—anchored in explicit quantification, transparent assumptions, structured aggregation, and continual recalibration—remain central to empirical strategy and risk management.