Reliability of language agents in creating real economic value in professional settings

Determine whether contemporary language-model-based agents can reliably create economic value in real-world professional environments, beyond performance on exam-style benchmarks, by consistently delivering expert-level work products that meet domain constraints and professional standards.

Background

The paper argues that existing evaluation suites are largely exam-style or highly structured, and many are reaching saturation, leaving open the question of whether agentic systems translate to dependable value creation in real professional workflows.

To address this gap, the authors propose OneMillion-Bench, which emphasizes economically consequential tasks requiring retrieval, evidence-grounded reasoning, and adherence to domain-specific constraints. The open question motivates the benchmark’s goal of measuring whether agent performance equates to tangible, reliable professional outputs.