Task-CoEvolve: Efficient Harness Optimization via Adaptive Validation Task Selection

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Atsuyuki Miyai, Kiyoharu Aizawa, Toshihiko Yamasaki
• Venue: arXiv: cs.CL
• Published: 2026-08-20
• Collected: 2026-08-21
• URL: http://arxiv.org/abs/2608.20169v1
• PDF: https://arxiv.org/pdf/2608.20169v1
• DOI: UNKNOWN
• Registry group: academic_apis_and_preprints
• Evidence role: canonical_academic_index
• Language: en
• Country / region: UNKNOWN

Plain-Language Brief

Researchers have developed a method called Task-CoEvolve to improve the efficiency of harness optimization for large language models (LLMs). This method co-evolves validation tasks with the harness by focusing on tasks that are challenging for the current state of the harness. By doing so, it reduces the number of evaluations during optimization while maintaining or even surpassing the final performance of full-set search.

What the Document Says

• The paper introduces a novel approach for efficient LLM agent harness optimization called Task-CoEvolve.
• Task-CoEvolve co-evolves the validation tasks with the harness to focus on informative tasks and estimate full-set performance from partial evaluations.
• It selects tasks that candidate harnesses disagree on as they are more informative for distinguishing among them.
• Task-CoEvolve uses variance-weighted sampling based on past outcomes to focus evaluation on tasks near the agent's capability frontier.
• The sampling distribution adapts as the harness evolves.
• It estimates full-set scores from sampled tasks by accounting for their sampling probabilities.
• Experiments on online text classification and Terminal-Bench 2.1 show that Task-CoEvolve outperforms fixed-subset baselines and matches the final performance of full-set search while reducing the number of evaluations during optimization by 80%.
• The code for Task-CoEvolve will be released at https://github.com/Agent4Science-UTokyo/Task-CoEvolve.

Key Concepts Mentioned

• Harness Optimization
• Large Language Models (LLMs)
• Adaptive Validation Task Selection
• Informative Tasks
• Capability Frontier
• Variance-weighted Sampling

Claims Made by the Document

• Task-CoEvolve co-evolves the validation tasks with the harness to improve efficiency.
• It focuses on tasks near the agent's capability frontier for better performance.
• Experiments show that Task-CoEvolve outperforms fixed-subset baselines.

Evidence Used

• Description of Task-CoEvolve and its approach.
• Results from experiments on online text classification and Terminal-Bench 2.1.

Methods / Approach

• Variance-weighted sampling
• Full-set score estimation

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

The paper does not mention any limitations by the authors, but as an assistant, I would note that the effectiveness of Task-CoEvolve may depend on the quality and diversity of the tasks used for validation.

Open Questions

• What is the impact of different task qualities and diversities on the performance of Task-CoEvolve?

Notes for Future Q&A

May include interpretation. Labeled where it does.

The paper does not discuss the impact of different task qualities and diversities on the performance of Task-CoEvolve.

Raw Abstract / Excerpt

We present a novel approach to efficient LLM agent harness optimization through adaptive validation task selection. Harness optimization iteratively rewrites the harness code based on validation performance, enabling substantial performance gains without updating the underlying model weights. Existing approaches, however, evaluate a fixed validation set in full at every iteration, incurring substantial evaluation costs even on tasks that become less discriminative as the harness evolves. We propose $\textbf{Task-CoEvolve}$, which co-evolves the validation tasks with the harness by addressing two challenges: selecting informative tasks and estimating full-set performance from partial evaluations. Task-CoEvolve builds on the observation that tasks on which candidate harnesses disagree are more informative for distinguishing among them than tasks that are consistently solved or failed. It uses variance-weighted sampling based on past outcomes to focus evaluation on tasks near the agent's capability frontier, with the sampling distribution adapting as the harness evolves. It then estimates full-set scores from the sampled tasks by accounting for their sampling probabilities, enabling consistent comparisons across iterations despite evaluating different subsets. Experiments on online text classification and Terminal-Bench 2.1 show that Task-CoEvolve consistently outperforms fixed-subset baselines and matches the final performance of full-set search while reducing the number of evaluations during optimization by 80%. Code will be released at https://github.com/Agent4Science-UTokyo/Task-CoEvolve.