Direct comparison of self‑evolving agent systems on a shared benchmark

Establish a standardized, shared benchmark and evaluation protocol that enable direct comparison of self‑evolving agent design and automated multi‑agent architecture search systems on scientific computing tasks that require heterogeneous tool use, structured task decomposition, and domain‑grounded validation.

Background

Work on automated agent design has mostly been validated on coding and general reasoning tasks, whereas scientific computing introduces different constraints such as heterogeneous tools, structured workflows, and domain-grounded evaluation. Because prior systems optimize different design spaces (e.g., unconstrained agent code versus DAG-structured workflows), there is no unified basis for comparison. A shared scientific benchmark would enable rigorous, apples-to-apples evaluation across methods.