Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

Automated Discovery Has No Universally Superior Harness

Source

arXiv

arxiv.org/abs/2607.18235v1

Read the source

Source

Plain-Language Brief

A study evaluates various discovery harnesses, such as OpenEvolve and TTT-Discover, used in automated discovery systems. The research finds that no single harness performs consistently well across different models and problems. Instead, the authors suggest that harness choice should be tailored to specific problems and models. They also propose an adaptive allocation method that starts multiple harnesses, prunes weak runs, and reallocates resources to stronger ones, outperforming fixed harness selection and non-adaptive ensembles.

What the Document Says

  • The text discusses automated discovery systems using OpenEvolve and TTT-Discover.
  • These systems combine several design choices about archives, parent selection, exploration, and budget allocation into a single recipe.
  • The authors argue that existing harnesses are often compared using insufficient independent trials.
  • They decompose these harnesses and evaluate 30 budget-matched harnesses across 12 model-problem pairs using more than 3.1 million LLM rollouts and repeated-trial statistical analysis.
  • The results show that no fixed harness is reliably superior across the evaluated model-problem pairs, with OpenEvolve variants generally underperforming simpler alternatives.
  • They propose a budget-matched adaptive-allocation experiment that starts multiple harnesses, prunes weak runs, and reallocates compute to stronger ones.
  • This method outperforms both commitment to a randomly sampled fixed harness and a non-adaptive harness ensemble.
  • The authors suggest that harness choice should be tailored to the specific problem and underlying model.
  • They release all run pools including baseline null distributions for every model-problem pair as reusable statistical infrastructure.

Key Concepts Mentioned

  • automated discovery systems
  • OpenEvolve
  • TTT-Discover
  • harnesses
  • archives
  • parent selection
  • exploration
  • budget allocation
  • evaluation
  • 3.1 million LLM rollouts
  • repeated-trial statistical analysis
  • generalization problem
  • budget-matched adaptive-allocation

Claims Made by the Document

  • Existing harnesses are often compared using insufficient independent trials.
  • No fixed harness is reliably superior across the evaluated model-problem pairs.
  • OpenEvolve variants generally underperform simpler alternatives.

Evidence Used

  • systematically decompose OpenEvolve-style evolutionary search and the TTT-Discover search harness into its constituent components
  • evaluated 30 budget-matched harnesses across 12 model-problem pairs using more than 3.1 million LLM rollouts

Methods / Approach

  • systematic decomposition of discovery harnesses
  • budget-matched evaluation
  • repeated-trial statistical analysis

Limitations Stated by the Authors

  • The study evaluates only a specific set of model-problem pairs.

Limitations Noted by the Agent

Agent interpretation — not from the source.

  • The text does not provide information on the specific evaluation criteria used for harness comparison.
  • It is unclear how the 'simpler alternatives' to OpenEvolve are defined and compared.

Open Questions

  • What are the specific evaluation criteria used in this study?
  • How are simpler alternatives to OpenEvolve defined and compared?

Notes for Future Q&A

May include interpretation. Labeled where it does.

  • The text does not specify whether the 3.1 million LLM rollouts refer to a single long run or multiple shorter runs.
  • It is unclear how 'run-to-run variance' is calculated and accounted for in this study.

Raw Abstract / Excerpt

Autonomous discovery systems such as OpenEvolve and TTT-Discover are often used as general-purpose harnesses. However, in practice these are composite systems combining several design choices about archives, parent selection, exploration, and budget allocation into a single recipe. Because discovery runs are expensive and inherently stochastic, existing harnesses are often compared using too few independent trials to distinguish key methodological improvements from run-to-run variance. We systematically decompose OpenEvolve-style evolutionary search and the TTT-Discover search harness into its constituent components and systematically evaluate 30 budget-matched harnesses across 12 model-problem pairs using more than 3.1 million LLM rollouts and repeated-trial statistical analysis. Our results show that discovery harnesses have a generalization problem: No fixed harness is reliably superior across the evaluated model-problem pairs, and variants of OpenEvolve generally underperform simpler alternatives. Thus, harness choice is better viewed as a hyperparameter rather than as a universal recipe, and should be tailored to the specific problem and underlying model. We also find that early discovery progress predicts final performance, and use this property to present a budget-matched adaptive-allocation experiment that starts multiple harnesses, prunes weak partial runs, and reallocates compute to stronger survivors, outperforming both commitment to a randomly sampled fixed harness and a non-adaptive harness ensemble. Together, these results motivate shifting from fixed harness selection to online adaptation guided by early performance. We release all run pools including baseline null distributions for every model-problem pair as reusable statistical infrastructure against for future harness proposals.