An Empirical Study of Harness Design for Coding Agents

Source facts

The paper varies planning, action space, and context management in a fixed coding harness and evaluates 176 matched settings across four models on SWE-Bench Verified and Terminal-Bench 2.1. It reports that context management is especially valuable under tight context budgets, rule-based elision before LLM summarization is efficient, planning helps weaker models more for accuracy and stronger models more for cost, and predefined tools help models with weaker bash proficiency.

Analysis

This is a component-level approach to agent evaluation: the harness, not only the base model, becomes an experimental variable. The results support model- and budget-aware orchestration rather than assuming one universal agent scaffold.

Uncertainty/Doubt

• This is an arXiv preprint; the matched settings, benchmark results, and cost conclusions were not independently reproduced here.