SUPER: Evaluating Agents on Setting Up and Executing Tasks from Research Repositories

Abstract: Given that LLMs have made significant progress in writing code, can they now be used to autonomously reproduce results from research repositories? Such a capability would be a boon to the research community, helping researchers validate, understand, and extend prior work. To advance towards this goal, we introduce SUPER, the first benchmark designed to evaluate the capability of LLMs in setting up and executing tasks from research repositories. SUPERaims to capture the realistic challenges faced by researchers working with Machine Learning (ML) and NLP research repositories. Our benchmark comprises three distinct problem sets: 45 end-to-end problems with annotated expert solutions, 152 sub problems derived from the expert set that focus on specific challenges (e.g., configuring a trainer), and 602 automatically generated problems for larger-scale development. We introduce various evaluation measures to assess both task success and progress, utilizing gold solutions when available or approximations otherwise. We show that state-of-the-art approaches struggle to solve these problems with the best model (GPT-4o) solving only 16.3% of the end-to-end set, and 46.1% of the scenarios. This illustrates the challenge of this task, and suggests that SUPER can serve as a valuable resource for the community to make and measure progress.

• The paper introduces SUPER, a benchmark that evaluates LLMs' ability to autonomously set up and execute research tasks from code repositories.
• It defines three task sets—Expert, Masked, and Auto—to assess various aspects of automation from curated tasks to large-scale challenges.
• Results reveal that even state-of-the-art models like GPT-4o struggle with complex setups, underscoring the need for improved execution and reproducibility.

Evaluating Agents on Setting Up and Executing Tasks from Research Repositories: An Overview of the SUPER Benchmark

The paper "SUPER: Evaluating Agents on Setting Up and Executing Tasks from Research Repositories" introduces SUPER, a benchmark specifically designed to evaluate the capabilities of LLMs in automating the setup and execution of tasks derived from research code repositories. This benchmark represents an essential step towards the autonomous reproduction of results from research repositories, crucial for the empirical Machine Learning (ML) and NLP research communities. The paper systematically addresses the significant challenges faced by LLMs in this context and proposes robust evaluation metrics.

Benchmark Construction

The SUPER benchmark is structured into three distinct problem sets: Expert, Masked, and Auto, each serving specific purposes.

1. Expert Set: This set comprises 45 tasks, manually curated and solved by experts. It prioritizes realistic but computationally manageable tasks, ensuring that they can be executed in a standard Jupyter environment without requiring GPU resources and within a reasonable timeframe (under 10 minutes per task). The tasks are based on research papers and are drawn from the Papers With Code repository. They cover a range of activities, from reproducing results to running modified experiments on new datasets or models.
2. Masked Set: Derived from the Expert set, this collection comprises 152 sub-problems obtained by masking specific parts of the gold solutions. It enables a more granular evaluation of an agent's ability to tackle individual components of the setup process, such as dependency management, data configuration, and error resolution. This fine-grained approach allows for a detailed assessment of model capabilities in handling diverse and intricate issues within a repository.
3. Auto Set: Consisting of 604 automatically generated tasks, this set is designed for large-scale development and model fine-tuning. It leverages LLMs to generate tasks based on repository readme files and targets repositories that are less curated and documented, thereby simulating real-world challenges of executing arbitrary research code. This dataset allows for more extensive testing and improvement of agents.

Evaluation Metrics

The evaluation of agents on SUPER is performed using a combination of accuracy-based and progress-based metrics:

• Accuracy Evaluation: This metric compares the agent's output to gold-standard solutions provided by experts. It is strict, requiring exact matches (within a small error margin) for reported metrics and outputs.
• Landmark-Based Evaluation: Recognizing that a strict accuracy measure might not capture incremental progress, this evaluation checks for specific landmark outputs indicating successful completion of intermediate steps, such as data loading or model training stages.
• Script-Executed Metric: Used for the Auto set, this heuristic checks if a key script runs without exceptions over a minimum duration, serving as a proxy for successful execution.

Results and Analysis

Initial experiments with LLMs, including GPT-4o and several open-source models, reveal significant challenges in the automated execution of research tasks. The state-of-the-art GPT-4o model achieves limited success, solving only 16.3% of the Expert tasks and 46.1% of the Masked sub-problems. This underlines the difficulty of the benchmark and the complexity inherent in setting up and running research code from diverse repositories.

Analysis of model performance on sub-problems from the Masked set highlights that LLMs are more effective at resolving well-specified, localized issues such as exception handling and dependency management. In contrast, tasks requiring deeper understanding and navigation of code repositories, such as configuring datasets or hyper-parameters, remain more challenging.

Implications and Future Directions

The SUPER benchmark serves multiple vital functions. Practically, it provides a structured framework for evaluating and improving LLMs and agents in automating research setups, potentially reducing the manual burden on researchers. Theoretically, it stimulates advancements in developing more robust and capable LLMs that can handle the nuanced and intricate tasks required for empirical research work.

Future developments in AI, fueled by benchmarks like SUPER, could lead to more sophisticated models capable of understanding and executing complex research code autonomously. This progress would contribute to increased reproducibility and verification of scientific experiments, facilitating more rapid and reliable advancements in ML and NLP fields.

In conclusion, the SUPER benchmark offers a rigorous, multi-faceted approach to evaluating the capabilities of LLMs in setting up and executing research tasks, addressing both practical challenges and theoretical advancements. It paves the way for further research into the development of autonomous agents capable of significantly enhancing the efficiency and reproducibility of empirical scientific research.