Predicting Empirical AI Research Outcomes with Language Models

Abstract: Many promising-looking ideas in AI research fail to deliver, but their validation takes substantial human labor and compute. Predicting an idea's chance of success is thus crucial for accelerating empirical AI research, a skill that even expert researchers can only acquire through substantial experience. We build the first benchmark for this task and compare LMs with human experts. Concretely, given two research ideas (e.g., two jailbreaking methods), we aim to predict which will perform better on a set of benchmarks. We scrape ideas and experimental results from conference papers, yielding 1,585 human-verified idea pairs published after our base model's cut-off date for testing, and 6,000 pairs for training. We then develop a system that combines a fine-tuned GPT-4.1 with a paper retrieval agent, and we recruit 25 human experts to compare with. In the NLP domain, our system beats human experts by a large margin (64.4% v.s. 48.9%). On the full test set, our system achieves 77% accuracy, while off-the-shelf frontier LMs like o3 perform no better than random guessing, even with the same retrieval augmentation. We verify that our system does not exploit superficial features like idea complexity through extensive human-written and LM-designed robustness tests. Finally, we evaluate our system on unpublished novel ideas, including ideas generated by an AI ideation agent. Our system achieves 63.6% accuracy, demonstrating its potential as a reward model for improving idea generation models. Altogether, our results outline a promising new direction for LMs to accelerate empirical AI research.

• The paper formulates a pairwise prediction problem and develops a benchmark using 6,000 training pairs and 1,585 expert-verified test pairs.
• The paper introduces a retrieval-augmented GPT-4.1 model that boosts prediction accuracy from 51.9% to 77%, effectively surpassing human expert performance.
• The paper validates the system's robustness through stress tests and demonstrates its generalization on novel, unpublished AI-generated ideas.

Predicting Empirical AI Research Outcomes with LLMs

Problem Formulation and Benchmark Construction

The paper addresses the challenge of forecasting the empirical success of AI research ideas prior to their implementation, a task that traditionally requires substantial human expertise and computational resources. The authors formalize this as a pairwise prediction problem: given two research ideas and a set of benchmarks, the goal is to predict which idea will empirically outperform the other. This formulation is operationalized through a large-scale benchmark, constructed by automatically extracting idea pairs and their outcomes from recent conference papers across NLP, ML, CV, and robotics domains. The benchmark comprises 6,000 training pairs and 1,585 human-verified test pairs, with rigorous post-processing to ensure label quality and avoid contamination from the base model's knowledge cutoff.

The benchmark design emphasizes objective empirical effectiveness, in contrast to prior work focused on subjective criteria such as novelty or excitement. Each example includes detailed idea descriptions, benchmark definitions, and binary outcome labels aggregated over multiple metrics to mitigate noise and ambiguity. Human annotators with domain expertise verify the test set, correcting extraction errors and ensuring fair comparisons.

System Architecture: Retrieval-Augmented Fine-Tuned LM

The proposed system integrates a fine-tuned GPT-4.1 model with an agentic paper retrieval module. The retrieval agent iteratively generates queries to search for relevant literature, decomposes novel ideas into sub-components, and summarizes full papers with respect to the query context. This approach leverages neural search (via exa.ai) and LM-based relevance filtering, ensuring that only pre-cutoff papers are retrieved to prevent information leakage. Summarizing entire papers, rather than relying on abstracts, yields a substantial accuracy improvement (from 38.8% to 53.0%), as abstracts often lack sufficient detail for nuanced comparisons.

The fine-tuning procedure involves training GPT-4.1 on historical idea pairs with outcome labels. Chain-of-thought (CoT) augmentation, using LM-generated rationales, does not improve performance, likely due to the low quality of self-generated CoTs in this domain. The final system reasons over the research goal, idea descriptions, and retrieved evidence to output a binary prediction.

Empirical Evaluation: LM vs. Human Experts

Zero-shot performance of frontier LMs (GPT-4.1, o3, Claude 3.5 Sonnet) is near random, even with retrieval augmentation, indicating that the task is non-trivial and not solved by general LM capabilities. Fine-tuning GPT-4.1 on the benchmark data increases accuracy from 51.9% to 77% on the test set, demonstrating the necessity of targeted capability elicitation.

A direct comparison with 25 expert NLP researchers, each with substantial publication and citation records, reveals that the LM system outperforms human experts by a significant margin. On a challenging subset of 45 NLP idea pairs, the system achieves 64.4% accuracy versus 48.9% for majority-voted human predictions. Even the best-performing individual annotators per topic do not match the LM's performance. Inter-annotator agreement is comparable to peer-review settings, underscoring the inherent difficulty of the task.

Robustness and Stress Testing

To assess robustness, the authors conduct extensive stress tests targeting superficial features that could confound predictions, such as idea recency, description length, and association with famous labs. The system exhibits low sensitivity to these features, maintaining stable accuracy across perturbed subsets. Further, LM-designed stress tests automatically generate and validate hundreds of hypotheses about potential shortcuts. The system clears 88% of these tests, with unsupported subset accuracy remaining above 61% even for flagged cases, indicating resilience against spurious correlations.

Generalization to Unpublished and AI-Generated Ideas

The system's generalizability is evaluated on a set of 33 novel, unpublished ideas from an AI ideation study, including both human- and LM-generated proposals. Ground truth outcomes are established via costly human implementation and experimentation. The system achieves 63.6% accuracy, demonstrating its applicability to genuinely novel research directions and its potential as a reward model for automated ideation pipelines.

Implications and Future Directions

The results establish that specialized, retrieval-augmented LMs can surpass human experts in predicting empirical research outcomes, with strong numerical gains and robustness to common human biases. This capability has direct implications for accelerating empirical AI research by enabling more efficient resource allocation, prioritization, and iterative refinement of ideas. The system can be integrated as a reward model in automated research workflows, guiding idea generation and selection without the need for expensive implementation.

Theoretically, the work suggests that LMs, when properly fine-tuned and augmented with targeted retrieval, can internalize complex patterns of empirical success that are difficult for humans to articulate or learn from limited experience. The failure of CoT augmentation in this context highlights the need for high-quality rationales and possibly more advanced simulation-based reasoning.

Future research may explore more interpretable models, inference-time experiment simulation, and integration with end-to-end automated research agents. Scaling the benchmark to additional domains and further improving robustness to adversarial shortcuts remain open challenges.

Conclusion

This paper demonstrates that fine-tuned, retrieval-augmented LMs can reliably predict the empirical success of AI research ideas, outperforming domain experts and generalizing to novel, unpublished proposals. The approach is robust to superficial features and offers a practical pathway for accelerating empirical research workflows. The findings motivate further development of LM-based forecasting systems and their integration into automated scientific discovery pipelines.