Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

Automated Discovery Has No Universally Superior Harness

Source

arXiv

arxiv.org/abs/2607.18235v1

Read the source

Source

Plain-Language Brief

A study investigates the effectiveness of different automated discovery systems (OpenEvolve and TTT-Discover) in various machine learning models and problems. The research finds that no single system consistently outperforms others, indicating that the choice of a discovery harness should be tailored to specific problems and models rather than being considered a universal recipe. The study also suggests that early discovery progress predicts final performance, implying the potential for adaptive allocation of resources to improve results.

What the Document Says

  • Autonomous discovery systems like OpenEvolve and TTT-Discover are used as general-purpose harnesses but consist of several design choices.
  • Existing harnesses are compared with too few independent trials, making it hard to distinguish methodological improvements from run-to-run variance.
  • The study decomposes these harnesses into components and evaluates 30 budget-matched harnesses across 12 model-problem pairs using more than 3.1 million LLM rollouts and repeated-trial statistical analysis.
  • The results show that no fixed harness is superior across the evaluated model-problem pairs, with variants of OpenEvolve generally underperforming simpler alternatives.
  • Harness choice should be tailored to specific problems and models instead of being considered a universal recipe.
  • Early discovery progress predicts final performance, and adaptive allocation of resources based on early performance can outperform both fixed harnesses and non-adaptive harness ensembles.
  • The study suggests shifting from fixed harness selection to online adaptation guided by early performance.

Key Concepts Mentioned

  • autonomous discovery systems
  • OpenEvolve
  • TTT-Discover
  • harnesses
  • archives
  • parent selection
  • exploration
  • budget allocation
  • machine learning models
  • problems
  • run-to-run variance
  • repeated-trial statistical analysis
  • adaptive allocation

Claims Made by the Document

  • Existing harnesses are often compared with too few independent trials.
  • No fixed harness is reliably superior across the evaluated model-problem pairs.
  • Variants of OpenEvolve generally underperform simpler alternatives.
  • Harness choice should be tailored to specific problems and models.
  • Early discovery progress predicts final performance.

Evidence Used

  • More than 3.1 million LLM rollouts and repeated-trial statistical analysis were used in the study.
  • The results from the evaluation of 30 budget-matched harnesses across 12 model-problem pairs are presented.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Autonomous discovery systems such as OpenEvolve and TTT-Discover are often used as general-purpose harnesses. However, in practice these are composite systems combining several design choices about archives, parent selection, exploration, and budget allocation into a single recipe. Because discovery runs are expensive and inherently stochastic, existing harnesses are often compared using too few independent trials to distinguish key methodological improvements from run-to-run variance. We systematically decompose OpenEvolve-style evolutionary search and the TTT-Discover search harness into its constituent components and systematically evaluate 30 budget-matched harnesses across 12 model-problem pairs using more than 3.1 million LLM rollouts and repeated-trial statistical analysis. Our results show that discovery harnesses have a generalization problem: No fixed harness is reliably superior across the evaluated model-problem pairs, and variants of OpenEvolve generally underperform simpler alternatives. Thus, harness choice is better viewed as a hyperparameter rather than as a universal recipe, and should be tailored to the specific problem and underlying model. We also find that early discovery progress predicts final performance, and use this property to present a budget-matched adaptive-allocation experiment that starts multiple harnesses, prunes weak partial runs, and reallocates compute to stronger survivors, outperforming both commitment to a randomly sampled fixed harness and a non-adaptive harness ensemble. Together, these results motivate shifting from fixed harness selection to online adaptation guided by early performance. We release all run pools including baseline null distributions for every model-problem pair as reusable statistical infrastructure against for future harness proposals.