Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note4 min read

Reasoning effort, not tool access, buys first-try reliability in agentic code generation: an observational study

Source

arXiv

arxiv.org/abs/2607.02436v1

Read the source

Source

Plain-Language Brief

A study found that the level of reasoning effort in agentic code generation, rather than the availability of additional capabilities like testing tools and design prompts, significantly affects the reliability of software. The study reveals that high-level models produce better results compared to low-cost local models, and increasing reasoning effort from 'High' to 'xHigh' can greatly improve first-try reliability.

What the Document Says

  • Agentic coding assistants were tested for their ability to build a real-time retrospective board from one detailed specification.
  • The study used a fixed 14 criterion functional rubric and visual quality review for scoring.
  • Ninety independent agent runs were conducted, spanning several model generations, two harnesses, a testing tool, two design-oriented prompts, and two reasoning effort levels.
  • Frontier models performed better, with scores near the ceiling, while a low-cost local model scored between 24 to 37 points.
  • Container deployment was the most common defect, failing in 44% of runs.
  • The testing tool increased cost by 42% to 68%, but did not improve functional score or reliability.
  • Increasing reasoning effort from 'High' to 'xHigh' lifted first-try perfect runs from 28% to 89% and reduced corrective prompts significantly.
  • Design-oriented prompts improved visual quality without lifting function.
  • The study concludes that most first run failures come from weak reasoning, which can be prevented by a stronger model or more effort.

Key Concepts Mentioned

  • agentic code generation
  • testing tools
  • design-oriented prompts
  • reasoning effort
  • first-try reliability
  • functional rubric

Claims Made by the Document

  • More capability does not necessarily yield better software.
  • High-level models produce better results compared to low-cost local models.
  • Container deployment is a common defect in agentic code generation.
  • The testing tool did not improve functional score or reliability.
  • Increasing reasoning effort can greatly improve first-try reliability.
  • Design-oriented prompts can improve visual quality without lifting function.

Evidence Used

  • Ninety independent agent runs built the same application and were scored on a fixed rubric.
  • The scores of frontier models and a low-cost local model were compared.
  • The failure rate of container deployment was analyzed across model generations.
  • The cost, functional score, and reliability of the testing tool were compared.
  • The effect of reasoning effort levels on first-try perfect runs and corrective prompts was studied.
  • The visual quality improvement by design-oriented prompts was observed.

Methods / Approach

  • observational study
  • scoring system
  • comparison of scores

Limitations Stated by the Authors

  • The study only examined one specific application and did not consider other potential factors influencing the performance of agentic coding assistants.
  • The study did not investigate the long-term effects of reasoning effort on software reliability.

Limitations Noted by the Agent

Agent interpretation — not from the source.

  • The study does not provide information about the specific models, harnesses, and testing tools used in the experiment.
  • The study does not clarify what is meant by 'xHigh' reasoning effort level.

Open Questions

  • What are the long-term effects of reasoning effort on software reliability?
  • How do different models, harnesses, and testing tools affect the performance of agentic coding assistants?

Notes for Future Q&A

May include interpretation. Labeled where it does.

  • It is important to note that the study does not explicitly state how the 'High' and 'xHigh' reasoning effort levels were defined or implemented.
  • The study also does not explain how the testing tool was integrated with the agentic coding assistant.

Raw Abstract / Excerpt

Agentic coding assistants are increasingly given extra capabilities, such as browser based testing tools and design oriented system prompts, on the assumption that more capability yields better software. This study tested that assumption directly. Ninety independent agent runs built the same application, a real time retrospective board, from one detailed specification, each scored on a fixed 14 criterion functional rubric (42 point maximum) and a visual quality review. The runs spanned several model generations, two agent harnesses, two reasoning effort levels, a testing tool, and two design oriented prompts. Capability tier dominated: frontier models clustered near the ceiling while a low cost local model fell to 24 to 37 points. A criterion level analysis revealed what run totals conceal. Container deployment was the dominant defect, failing first try in 44 percent of runs, with its failure rate shifting sharply across model generations while mean totals moved less than a point. The testing tool raised cost by 42 to 68 percent without improving functional score or reliability, even on interface visible criteria. Raising reasoning effort from High to xHigh lifted first try perfect runs from 28 percent to 89 percent and cut corrective prompts about five fold, for 9 to 29 percent more cost. A design oriented prompt raised visual quality, 4.5 versus 3.0 on a 5 point scale, without lifting function, and a one paragraph paraphrase of its directive reproduced the entire lift. The practical lesson is to match the fix to the failure: most first run failures came from weak reasoning, which a stronger model or more effort prevents, not from visible flaws a checking tool would catch.