Agent in a Bottle: Can LLM Agents Turn Their Capabilities Into Cheap, Scalable Artifacts?

Source facts

The preprint introduces BOTTLED, a benchmark in which agents receive an unlabelled workload and must complete it under fixed time, compute, and LLM API budgets. Agents may train a small model or write a reusable program to turn general capabilities into task-specific solutions. Across ten models and three tasks, the authors report that strong zero-shot performance does not reliably predict bottling ability. They report substantial savings on query-product relevance classification, including about 82% of Opus 5's zero-shot macro-F1 at roughly 657 times lower reported cost.

Analysis

The paper treats agentic problem solving as an investment decision: spend resources once to create a reusable artifact, then amortize that cost over repetitive workloads. The benchmark is relevant to practical deployment economics, but the reported savings depend on the selected tasks, baselines, and accounting of projected full-workload cost.

Uncertainty/Doubt

• This is an arXiv preprint. The benchmark results, model configurations, and cost accounting were not independently reproduced here.