Do Agent Optimizers Compound? A Continual-Learning Evaluation on Terminal-Bench 2.0

Abstract: Most reported gains from agent-optimization methods are one-shot: an agent is optimized against a fixed benchmark and the resulting improvement is reported as if it were a stable property of the method. This does not test the setting that matters for deployed agents, where optimization is applied recursively as new failures and new tasks appear over time. The central question this raises is whether optimizer-driven gains compound: after an agent has been optimized once, can it be optimized again on newly arrived tasks without eroding the gains the first round produced? We study this question with a two-phase continual-learning evaluation built from hard tasks in Terminal-Bench 2.0, comparing three approaches to agent-harness optimization (GEPA, Meta Harness, and RELAI's Verifiable Continual Learning, RELAI-VCL) under identical optimization budgets. All three methods improve over the baseline agent in the conventional, static, single-phase setting. However, once new tasks are introduced, the methods diverge sharply: GEPA's optimized agent transfers below the unoptimized baseline, Meta Harness transfers well but fails to improve further once given a second optimization budget, and RELAI-VCL is the only method that both transfers positively to unseen tasks and continues improving after those tasks are folded into the optimization objective, reaching the highest pass rate at every evaluated stage and the highest lifelong average pass rate overall (76.4% vs. 66.0% for GEPA, 64.6% for Meta Harness, and 58.7% for the baseline). Our key observation was that optimization gains compounded only when regression control was built into the optimization loop, providing an inductive bias against shortcut solutions that fail to generalize.

• The paper demonstrates that regression-aware optimization (RELAI-VCL) consistently compounds improvements while mitigating catastrophic forgetting.
• It employs a two-phase evaluation on Terminal-Bench 2.0, comparing GEPA, Meta Harness, and RELAI-VCL across static, transfer, and re-optimization scenarios.
• The study outlines practical implications for robust agent deployment and underscores the need for benchmarks that capture realistic task correlations.

Expert Summary: Do Agent Optimizers Compound? A Continual-Learning Evaluation on Terminal-Bench 2.0

Introduction and Motivation

The paper "Do Agent Optimizers Compound? A Continual-Learning Evaluation on Terminal-Bench 2.0" (2607.14004) interrogates the behavior of agent optimizers under a continual-learning regime, specifically whether gains from agent harness optimization persist and can be recursively compounded as new tasks are introduced. Conventional evaluations report improvements post one-shot optimization on static benchmarks, but the operational reality is staged or continual adaptation, often as new failure modes or task types emerge. The principal question addressed is whether agent harness optimizers—methods for automated prompt, tool, memory, and control code refinement—can compound improvements without regressions on previously solved tasks.

Protocol Design and Methodology

The paper introduces a two-phase evaluation protocol utilizing Terminal-Bench 2.0, a cohort of verifiable, hard terminal-based tasks. Phase 1 involves optimizing a baseline agent (GPT-5.5 based, Harbor/Ternimus2 framework) on an initial task set ($\mathcal{T}_1$) with a fixed rollout budget. Agents are then evaluated both on $\mathcal{T}_1$ (static) and an expanded set $\mathcal{T}_1 \cup \mathcal{T}_2$ (transfer) without further optimization. Phase 2 grants an additional budget for harness optimization on the union, assessing final pass rates. Four metrics are reported: Phase 1, Transfer, Final, and Lifelong Average pass rate (unweighted mean across the three preceding metrics). This protocol allows deconvolution of static optimization strength, transfer/generalization, and continued improvement upon task expansion.

Agent Harness Optimization Methods Compared

Three agent harness optimizers are evaluated:

• GEPA: Reflective prompt evolutionary search, mutating agent prompts based on language feedback while retaining improvements. Tends to encode per-task specifics within prompts.
• Meta Harness: Outer-loop harness code search via an agentic proposer (Codex/GPT-5.5), accepts candidates based on task-set evaluation feedback. Fixes are generic, targeting robustness.
• RELAI-VCL: Regression-aware continual-learning optimizer, enforcing a no-regression constraint inside the search loop. Candidates improving new tasks but regressing on prior tasks are rejected. Harness edits are intended to be broadly generalizable.

All optimizers operate from an identical baseline agent and receive equal rollout budgets in each phase. GEPA's code-mutation variant is omitted from Phase 2 due to failure to produce valid candidates.

Results: Static, Transfer, and Continued Improvement

Static Optimization (Phase 1)

All methods improve the baseline in the static regime. GEPA achieves 70.8%, Meta Harness 66.6%, and RELAI-VCL achieves 79.2% pass rate on $\mathcal{T}_1$ versus 62.5% for baseline, with RELAI-VCL showing the most substantial gains and the most generic harness modifications.

Figure 1: RELAI-VCL achieves the highest static optimization pass rate at 79.2% on Phase 1 tasks, outperforming GEPA and Meta Harness.

Transfer to Unseen Tasks

Evaluating Phase-1-optimized agents on the expanded set reveals sharp divergence:

• GEPA shows negative transfer (54.5%), below baseline, indicative of overfitting via prompt leakage of task-specific information.
• Meta Harness improves markedly (68.2%) due to robust, generic harness fixes.
• RELAI-VCL leads at 72.7%, with candidates transferring positively (65.0% on unseen $\mathcal{T}_2$), supporting the claim that regression-aware search acts as an implicit generalization filter.

  Figure 2: RELAI-VCL maintains strong transfer performance to previously unseen tasks, outperforming GEPA and Meta Harness.

Phase 2 Re-optimization

Upon re-optimization, GEPA recovers to 72.7% by direct optimization against expanded tasks, Meta Harness stalls and regresses to 59.1%, while RELAI-VCL continues to improve, reaching 77.3%. Only RELAI-VCL demonstrates both positive transfer and further gains after task set expansion, supporting compounding optimization.

Figure 3: RELAI-VCL maintains continued improvement after phase 2 re-optimization, avoiding the stagnation or regression observed in alternative methods.

Lifelong Average Performance

RELAI-VCL achieves the highest lifelong average pass rate (76.4%), substantially ahead of GEPA (66.0%), Meta Harness (64.6%), and baseline (58.7%).

Figure 4: Summary of lifelong average pass rates highlights RELAI-VCL's superior compounding and generalization properties.

Interpretation and Practical Implications

The evaluation illustrates that regression-aware search, as enforced by RELAI-VCL, is critical for compounding harness optimization gains. Enforcing this constraint within the search loop curbs catastrophic forgetting and biases the optimization toward genuinely generalizable interventions over narrow, task-specific shortcuts. Static benchmark performance cannot adequately diagnose overfitting or stagnation; compounding requires both robust transfer and sustained improvement under task arrival.

From a practical perspective, optimizers intended for deployment must internalize regression constraints to achieve durable agent improvements, especially as real-world task distributions shift continuously. Regression-avoidance as a search-time constraint—rather than post-hoc check—enables reliable agent escalation in production settings, relevant for continuous integration pipelines, auto-diagnosis loops, and broader agentic workflows.

Limitations and Future Directions

Terminal-Bench 2.0 tasks are loosely correlated, limiting the realism of cross-task dependency exploitation and shortcut generalization failures. Practical settings often feature highly correlated, domain-specific tasks, where exploiting shared structure or reciprocal failure modes is common. Repeatability and verifier availability are assumed, but are rarely guaranteed in operational deployments. More realistic benchmarks should address task correlation, limited observability, imperfect verifiers, and long-horizon iterative optimization beyond two phases.

Anticipated future developments include:

• Benchmarks and protocols capturing correlated failure modes and domain drift
• Agent harness optimizers integrating explicit memory, skill accumulation, and trace-driven harness adaptation with in-loop regression constraints
• Adaptive optimization strategies for partial, noisy, or single-instance feedback
• Scaling the compounding evaluation to meta-optimization of the optimizer itself (hyperagents)

Conclusion

The paper establishes, through controlled continual-learning evaluation on Terminal-Bench 2.0, that regression-aware agent harness optimization (RELAI-VCL) is uniquely capable of compounding improvements across task arrivals, retaining prior gains and facilitating further progress. Optimization gains compound only when regression control is built directly into the search loop. These findings suggest that agentic continual-learning protocols and regression-aware harness optimizers are prerequisites for robust, production-scale agent deployment, with future benchmarks needing to reflect the true complexity and correlation structures of deployed agent workloads.

Figure 5: RELAI-VCL consistently leads across all stages and achieves the highest lifelong average pass rate, demonstrating compounding optimization capabilities under continual learning.