Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

Do Agent Optimizers Compound? A Continual-Learning Evaluation on Terminal-Bench 2.0

Source

arXiv

arxiv.org/abs/2607.14004v1

Read the source

Source

Plain-Language Brief

The research evaluates if improvements from agent-optimization methods can stack up over time in a continual learning setting. Three optimizer-driven methods (GEPA, Meta Harness, and RELAI's Verifiable Continual Learning) were tested on hard tasks from Terminal-Bench 2.0. In the initial phase, all methods improved over the baseline agent. However, when new tasks were introduced, the performance of GEPA and Meta Harness dropped while RELAI-VCL continued improving and outperformed other methods in both transfer to unseen tasks and overall lifelong average pass rate.

What the Document Says

  • Three agent-optimization methods (GEPA, Meta Harness, and RELAI's Verifiable Continual Learning) were compared under identical optimization budgets.
  • The study used a two-phase continual learning evaluation built from hard tasks in Terminal-Bench 2.0.
  • In the initial phase, all methods improved over the baseline agent.
  • When new tasks were introduced, GEPA's optimized agent performed worse than the unoptimized baseline.
  • Meta Harness transferred well but did not improve further when given a second optimization budget.
  • RELAI-VCL continued improving after new tasks were folded into the optimization objective and outperformed other methods in pass rate at every stage and lifelong average pass rate.
  • The key observation was that optimization gains compounded only when regression control was built into the optimization loop.

Key Concepts Mentioned

  • agent-optimization methods
  • continual learning
  • Terminal-Bench 2.0
  • GEPA
  • Meta Harness
  • RELAI's Verifiable Continual Learning
  • pass rate

Claims Made by the Document

  • Optimizer-driven gains compound when regression control is built into the optimization loop.
  • RELAI-VCL outperforms other methods in pass rate and lifelong average pass rate.

Evidence Used

  • The performance of GEPA, Meta Harness, and RELAI-VCL was compared under identical optimization budgets.
  • The results showed that RELAI-VCL continued improving while the other methods dropped in performance when new tasks were introduced.
  • The key observation from the study was that optimization gains compounded only when regression control was built into the optimization loop.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Most reported gains from agent-optimization methods are one-shot: an agent is optimized against a fixed benchmark and the resulting improvement is reported as if it were a stable property of the method. This does not test the setting that matters for deployed agents, where optimization is applied recursively as new failures and new tasks appear over time. The central question this raises is whether optimizer-driven gains compound: after an agent has been optimized once, can it be optimized again on newly arrived tasks without eroding the gains the first round produced? We study this question with a two-phase continual-learning evaluation built from hard tasks in Terminal-Bench 2.0, comparing three approaches to agent-harness optimization (GEPA, Meta Harness, and RELAI's Verifiable Continual Learning, RELAI-VCL) under identical optimization budgets. All three methods improve over the baseline agent in the conventional, static, single-phase setting. However, once new tasks are introduced, the methods diverge sharply: GEPA's optimized agent transfers below the unoptimized baseline, Meta Harness transfers well but fails to improve further once given a second optimization budget, and RELAI-VCL is the only method that both transfers positively to unseen tasks and continues improving after those tasks are folded into the optimization objective, reaching the highest pass rate at every evaluated stage and the highest lifelong average pass rate overall (76.4% vs. 66.0% for GEPA, 64.6% for Meta Harness, and 58.7% for the baseline). Our key observation was that optimization gains compounded only when regression control was built into the optimization loop, providing an inductive bias against shortcut solutions that fail to generalize.