Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

Do Agent Optimizers Compound? A Continual-Learning Evaluation on Terminal-Bench 2.0

Source

arXiv

arxiv.org/abs/2607.14004v1

Read the source

Source

Plain-Language Brief

Researchers investigated if improvements made by agent-optimization methods persist when agents are optimized again on new tasks. They found that one method called RELAI-VCL was the only one that both transferred positively to unseen tasks and continued improving after those tasks were incorporated into the optimization objective, outperforming other methods in terms of pass rate.

What the Document Says

  • Most reported gains from agent-optimization methods are not tested in deployed agents, where optimization is applied recursively over time.
  • The study evaluates three methods of agent-harness optimization: GEPA, Meta Harness, and RELAI's Verifiable Continual Learning (RELAI-VCL) under identical optimization budgets.
  • All three methods improve over the baseline agent in the conventional setting.
  • When new tasks are introduced, GEPA's optimized agent performs worse than the unoptimized baseline, Meta Harness transfers well but fails to improve further with a second optimization budget, and RELAI-VCL both transfers positively to unseen tasks and continues improving.
  • RELAI-VCL has the highest pass rate at every evaluated stage and the highest lifelong average pass rate (76.4%) compared to GEPA (66.0%), Meta Harness (64.6%), and the baseline.
  • The observation is that optimization gains compound only when regression control is built into the optimization loop.

Key Concepts Mentioned

  • agent-optimization methods
  • continual learning
  • Terminal-Bench 2.0
  • GEPA
  • Meta Harness
  • RELAI's Verifiable Continual Learning (RELAI-VCL)
  • optimization budget
  • pass rate

Claims Made by the Document

  • Optimizer-driven gains compound only when regression control is built into the optimization loop.

Evidence Used

  • The performance of the RELAI-VCL method was compared to others, showing better performance when regression control was integrated.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Most reported gains from agent-optimization methods are one-shot: an agent is optimized against a fixed benchmark and the resulting improvement is reported as if it were a stable property of the method. This does not test the setting that matters for deployed agents, where optimization is applied recursively as new failures and new tasks appear over time. The central question this raises is whether optimizer-driven gains compound: after an agent has been optimized once, can it be optimized again on newly arrived tasks without eroding the gains the first round produced? We study this question with a two-phase continual-learning evaluation built from hard tasks in Terminal-Bench 2.0, comparing three approaches to agent-harness optimization (GEPA, Meta Harness, and RELAI's Verifiable Continual Learning, RELAI-VCL) under identical optimization budgets. All three methods improve over the baseline agent in the conventional, static, single-phase setting. However, once new tasks are introduced, the methods diverge sharply: GEPA's optimized agent transfers below the unoptimized baseline, Meta Harness transfers well but fails to improve further once given a second optimization budget, and RELAI-VCL is the only method that both transfers positively to unseen tasks and continues improving after those tasks are folded into the optimization objective, reaching the highest pass rate at every evaluated stage and the highest lifelong average pass rate overall (76.4% vs. 66.0% for GEPA, 64.6% for Meta Harness, and 58.7% for the baseline). Our key observation was that optimization gains compounded only when regression control was built into the optimization loop, providing an inductive bias against shortcut solutions that fail to generalize.