PACE: A Proxy for Agentic Capability Evaluation

Abstract: Evaluating LLM agents on benchmarks like SWE-Bench and GAIA can be expensive, time-consuming, and requires complex infrastructure. A single evaluation can cost thousands of dollars and take days to complete. In contrast, non-agentic LLM benchmarks that test individual capabilities (e.g., reasoning, code generation) are fast and cheap to run. In this paper, we investigate whether performance on expensive agentic benchmarks can be accurately predicted by the performance on a small, carefully selected subset of atomic evaluation instances. We introduce PACE, a framework that constructs proxy benchmarks by selecting instances from existing non-agentic evaluations whose aggregate scores most reliably predict model performances on agentic benchmarks. Given a pool of candidate instances spanning atomic capabilities, PACE fits a regression that maps a model's scores on a compact subset of source instances to its score on the target agentic benchmark. The subset itself is curated by combining two complementary instance-selection strategies, target-relevance local selection and globally informative global selection. We apply PACE to the 4 target agentic benchmarks in this paper, which yields PACE-Bench, the concrete proxy benchmark that we evaluate in the paper. Experiments across 14 models, 4 agentic benchmarks, and 19 non-agentic benchmarks show that PACE-Bench predicts agentic scores with leave-one-out cross-validation (LOOCV) mean absolute error (MAE) under 4%, Spearman correlation above 0.80, and pairwise model-ranking accuracy around 85%, all at much less than 1% of the full agentic evaluation cost. We further analyze the selected proxy instances, revealing which skills each agentic benchmark uniquely demands. PACE enables practitioners to obtain reliable estimates of agentic performance during model development, selection, and routing, without the overhead of full agent evaluation.

• The paper proposes a cost-effective proxy framework that selects a compact subset of non-agentic benchmark instances to predict agentic performance.
• It employs dual-regression models with noise-aware and bootstrapped training to achieve <4% MAE, >0.80 Spearman correlation, and ~85% ranking accuracy.
• The framework offers a ~100-fold cost reduction, enabling rapid model selection, robust performance tracking, and efficient agent evaluation.

PACE: A Proxy for Agentic Capability Evaluation—Technical Analysis

Motivation and Core Problem

The evaluation landscape for LLM-based agents is rapidly shifting from isolated, static benchmarks to complex, high-overhead agentic evaluations that probe multi-step tool usage, long-horizon planning, and interactive error recovery. Notable agentic benchmarks (e.g., SWE-Bench, GAIA, WebArena) impose substantial financial and logistical costs, often requiring days of compute and significant framework setup per model. In contrast, non-agentic, atomic capability benchmarks remain fast and inexpensive, evaluating critical subskills such as instruction following, reasoning, or code generation.

The key open question addressed in this work is whether a well-chosen, compact subset of non-agentic benchmark instances can serve as a high-fidelity proxy for downstream agentic benchmark performance. This extends beyond traditional benchmark compression or within-benchmark subset selection, seeking to exploit cross-benchmark predictive structure to offer extreme cost reductions for agentic score estimation and model ranking.

The PACE Framework

PACE (Proxy for Agentic Capability Evaluation) structurally formulates the proxy design as an instance selection and regression problem: Given a fixed budget $C$ of queries and a source pool of diverse non-agentic benchmark instances, select the subset whose aggregate scores most accurately induce both absolute agentic benchmark scores and pairwise model ranking.

PACE operationalizes this via a decoupling of proxy subset selection and regression prediction:

1. Noise-Aware Regression: PACE learns two regressors. For absolute score prediction (Goal A), a linear ridge or least-squares regressor predicts a model’s average agentic score from its scores on the selected source instances. For pairwise model ranking (Goal B), a logistic Bradley-Terry model predicts which model outperforms another agentically, based again on the proxy subset.
2. Dual-Criteria Instance Selection: Instance selection integrates two independent, complementary criteria, computed via SVD:
   • Target-Relevance (Local): The Spearman correlation between a source instance’s score vector and target agentic scores across models identifies instances highly predictive for the agentic benchmark.
   • Global Leverage (Global): The leverage score of each source instance in the latent space of the cross-model source instance matrix captures instances of high information content or diversity. Selection mixes both criteria, allocating the budget to local/global top-$k$ instances, and, if necessary, augmenting to ensure uniqueness.
3. Bootstrapped Regression Training: To account for the inherent sampling noise due to few agentic instances and limited calibration models, target means are bootstrapped over sampled subsets of agentic test cases per model, stabilizing regression weights and enhancing generalization.
4. Prediction and Ensembles: Final predicted scores are generated by a data-validated ensemble of local and global regressors.

   Figure 1: Overview of PACE: dual filter-based instance selection from the source pool, followed by noise-aware regression for agentic score estimation and pairwise preference prediction.

Experimental Protocol and Main Results

PACE is instantiated and evaluated across 14 models, 4 challenging agentic benchmarks (GAIA, SWE-Bench Multimodal, SWE-Bench Verified, SWT-Bench), and 19 non-agentic source benchmarks that together span 11 classes of LLM capability.

• Evaluation Protocol: Strict leave-one-out cross-validation (LOOCV) is used for predicting unseen model performance.
• Proxy Budget: Most experiments focus on $C=100$ proxy queries, a regime designed for maximal cost-effectiveness.

Quantitative Performance

Summary Metrics (LOOCV, $C=100$):

• Absolute agentic score MAE: < 4%
• Spearman correlation: > 0.80
• Pairwise ranking accuracy: ~85%
• Cost: $\sim$1% of full agentic benchmark evaluation; $\sim$100-fold improvement over random target subsampling for matched accuracy

  Figure 2: Cost-quality trade-off—PACE (blue) dominates random agentic instance subsampling (red) across key metrics; at fixed accuracy, PACE achieves roughly $1/100$ the evaluation cost.

Proxy Allocation Insights

An analysis of the learned allocations reveals that agentic tasks have distinct, but partially overlapping, underlying capability profiles. The PACE selection consistently includes instances probing instruction following and reasoning, but allocates different fractions to capabilities such as long-context aggregation, planning, or multimodal processing according to agentic task demands.

• For GAIA, there is heavy allocation to instruction-following and test verification instances, matching its requirement for strict answer adherence and self-validation.
• For SWE-Bench Verified and Multimodal, allocations emphasize Multimodal Understanding, Information Retrieval, and cross-instance aggregation, reflecting their repository-level, multimodal, and long-context reasoning demands.

  Figure 3: Distribution of selected source instances across capability classes for each agentic target—a clear alignment of acquired subskills to the nature of the agentic task.

  

  Figure 4: Source benchmark allocation per (target, source) pair at $C=100$, revealing both shared and distinct source-benchmark usage patterns for different agentic evaluations.

Ablation Analyses and Baseline Comparisons

• Removal of bootstrapping consistently increases MAE (by $\sim$0.8%) and reduces Spearman correlation, supporting the need for label-denoising under limited agentic instance access.
• Sparse regression approaches (Lasso, Ridge), which combine selection and regression, exhibit strong in-sample fits but sharp degradation under LOOCV due to overfitting. PACE’s decoupled, filter-based selection is more robust across model cohorts and generalizes better for proxy prediction.

Practical and Theoretical Implications

Practical Implications:

The demonstrated performance and efficiency of PACE suggest that agentic benchmark proxies can be robustly constructed without access to agent harnesses or substantial API budgets, providing affordable, interpretable gauges of agentic readiness. This is immediately useful for:

• Dense, low-cost model selection and ablation during agentic LLM development cycles
• Model routing and prediction in settings where full agentic evaluation is prohibitive
• Capability tracking as LLMs progress and as new agentic tasks are introduced

Theoretical Implications:

The success of PACE validates the existence of strong cross-benchmark predictive structures, bolstering the case for meta-benchmarking and learned instance selection across disparate tasks. It further motivates future research in instance selection criteria, robust transfer of evaluation signals, and proxy gaming resistance.

Limitations and Future Directions:

Proxy gaming, calibration set coverage, and limited diversity in source/target benchmarks warrant caution. Periodic refreshing of the calibration cohort and proxy pool is necessary as LLM families evolve. There is scope for adaptive, dynamic proxy generation and further analysis of proxy reliability under wider distributional shifts, model architectures, or novel agentic scaffoldings.

Conclusion

PACE offers a principled, rigorous framework for dramatically reducing the cost of agentic capability evaluation via supervised instance selection from broad pools of fast, atomic benchmarks. Its dual-criterion selection, label-noise-aware regression, and robust generalization properties position it as a practical tool for the next generation of rigorous LLM agent evaluation pipelines, with implications for both the theoretical understanding and day-to-day operations of LLM development. Its design offers a template for future proxy-based evaluation strategies in increasingly complex, resource-constrained AI benchmarking ecosystems.