Task-CoEvolve: Efficient Harness Optimization via Adaptive Validation Task Selection

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Atsuyuki Miyai, Kiyoharu Aizawa, Toshihiko Yamasaki
• Venue: arXiv: cs.CL
• Published: 2026-08-20
• Collected: 2026-08-21
• URL: http://arxiv.org/abs/2608.20169v1
• PDF: https://arxiv.org/pdf/2608.20169v1
• DOI: UNKNOWN
• Registry group: academic_apis_and_preprints
• Evidence role: canonical_academic_index
• Language: en
• Country / region: UNKNOWN

Plain-Language Brief

The paper presents Task-CoEvolve, a new approach to optimize language model agent harnesses efficiently by adaptively selecting validation tasks. It addresses the challenge of using fixed validation sets by co-evolving the validation tasks with the harness, focusing on tasks where candidate harnesses disagree for more informed comparisons, and estimating full-set scores from sampled tasks.

What the Document Says

• The paper introduces an approach to optimize language model agent (LLM) harnesses through adaptive validation task selection called Task-CoEvolve.
• Existing methods for harness optimization evaluate a fixed validation set in full at every iteration, which is costly even for tasks that become less discriminative as the harness evolves.
• Task-CoEvolve aims to solve this problem by co-evolving the validation tasks with the harness.
• It focuses on tasks where candidate harnesses disagree as they are more informative for distinguishing among them.
• Task-CoEvolve uses variance-weighted sampling based on past outcomes to focus evaluation on tasks near the agent's capability frontier.
• The sampling distribution adapts as the harness evolves.
• It estimates full-set scores from the sampled tasks by accounting for their sampling probabilities, enabling consistent comparisons across iterations.
• Experiments on online text classification and Terminal-Bench 2.1 show that Task-CoEvolve outperforms fixed-subset baselines and matches the final performance of full-set search while reducing evaluation costs during optimization by 80%.

Key Concepts Mentioned

• language model agent harness optimization
• adaptive validation task selection
• variance-weighted sampling
• capability frontier
• full-set score estimation

Claims Made by the Document

• Task-CoEvolve co-evolves the validation tasks with the harness.
• Tasks on which candidate harnesses disagree are more informative for distinguishing among them.
• Task-CoEvolve focuses evaluation on tasks near the agent's capability frontier.
• Task-CoEvolve estimates full-set scores from sampled tasks.
• Experiments show that Task-CoEvolve outperforms fixed-subset baselines.

Evidence Used

• The text mentions 'a novel approach to efficient LLM agent harness optimization through adaptive validation task selection' and describes the methods used for this approach.
• It states 'We propose Task-CoEvolve, which co-evolves the validation tasks with the harness...' and explains the challenges it addresses.
• The text says 'It focuses on tasks where candidate harnesses disagree as they are more informative for distinguishing among them.'
• It mentions 'Task-CoEvolve uses variance-weighted sampling based on past outcomes to focus evaluation...', explaining how it focuses on tasks near the agent's capability frontier.
• The text states 'It then estimates full-set scores from the sampled tasks by accounting for their sampling probabilities...'
• Experimental results are presented, showing that Task-CoEvolve outperforms fixed-subset baselines and matches the final performance of full-set search while reducing evaluation costs.

Methods / Approach

• variance-weighted sampling
• full-set score estimation

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

We present a novel approach to efficient LLM agent harness optimization through adaptive validation task selection. Harness optimization iteratively rewrites the harness code based on validation performance, enabling substantial performance gains without updating the underlying model weights. Existing approaches, however, evaluate a fixed validation set in full at every iteration, incurring substantial evaluation costs even on tasks that become less discriminative as the harness evolves. We propose $\textbf{Task-CoEvolve}$, which co-evolves the validation tasks with the harness by addressing two challenges: selecting informative tasks and estimating full-set performance from partial evaluations. Task-CoEvolve builds on the observation that tasks on which candidate harnesses disagree are more informative for distinguishing among them than tasks that are consistently solved or failed. It uses variance-weighted sampling based on past outcomes to focus evaluation on tasks near the agent's capability frontier, with the sampling distribution adapting as the harness evolves. It then estimates full-set scores from the sampled tasks by accounting for their sampling probabilities, enabling consistent comparisons across iterations despite evaluating different subsets. Experiments on online text classification and Terminal-Bench 2.1 show that Task-CoEvolve consistently outperforms fixed-subset baselines and matches the final performance of full-set search while reducing the number of evaluations during optimization by 80%. Code will be released at https://github.com/Agent4Science-UTokyo/Task-CoEvolve.