Reasoning effort, not tool access, buys first-try reliability in agentic code generation: an observational study

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Achint Mehta
• Venue: arXiv: cs.SE
• Published: 2026-07-02
• Collected: 2026-07-03
• URL: http://arxiv.org/abs/2607.02436v1
• PDF: https://arxiv.org/pdf/2607.02436v1
• DOI: UNKNOWN

Plain-Language Brief

A study found that more reasoning effort in agentic code generation (AI programs assisting human coders) leads to higher reliability and fewer errors, rather than the use of additional tools or capabilities.

What the Document Says

• Agentic coding assistants were tested for building a real-time retrospective board application.
• The test involved 90 independent runs using a detailed specification and scored on a 14 criterion functional rubric.
• Frontier models performed well, while a low-cost local model performed poorly.
• Container deployment was the most common defect, failing in 44% of runs.
• A testing tool increased cost by 42-68% but did not improve functionality or reliability.
• Increasing reasoning effort from High to xHigh increased perfect first try runs from 28% to 89%, and reduced corrective prompts by about five times.
• A design oriented prompt improved visual quality without lifting function.

Key Concepts Mentioned

• agentic coding assistants
• real-time retrospective board
• detailed specification
• functional rubric
• container deployment
• testing tool
• reasoning effort
• design oriented prompt
• visual quality

Claims Made by the Document

• More capability in agentic coding assistants does not necessarily lead to better software.
• Weak reasoning is a common cause of first-run failures that can be addressed with stronger models or more effort.
• Checking tools may not improve functionality or reliability.

Evidence Used

• results from the study showing performance differences between models and tools

Methods / Approach

• testing 90 independent runs of agentic coding assistants on a real-time retrospective board application
• scoring based on a fixed 14 criterion functional rubric

Limitations Stated by the Authors

• The study only tested one specific application and may not generalize to other types of software.

Limitations Noted by the Agent

Agent interpretation — not from the source.

• The data provided does not specify the exact methodology used for testing or scoring, which could impact the results.
• The study does not discuss potential bias in the selection of models or tools tested.

Open Questions

• What other factors might influence the reliability and functionality of agentic coding assistants beyond reasoning effort and tool use?
• How can the findings of this study be applied to different types of software development projects?

Notes for Future Q&A

May include interpretation. Labeled where it does.

• The study does not specify the exact definitions or criteria for the functional rubric used.
• The cost increase associated with using the testing tool is not quantified.

Raw Abstract / Excerpt

Agentic coding assistants are increasingly given extra capabilities, such as browser based testing tools and design oriented system prompts, on the assumption that more capability yields better software. This study tested that assumption directly. Ninety independent agent runs built the same application, a real time retrospective board, from one detailed specification, each scored on a fixed 14 criterion functional rubric (42 point maximum) and a visual quality review. The runs spanned several model generations, two agent harnesses, two reasoning effort levels, a testing tool, and two design oriented prompts. Capability tier dominated: frontier models clustered near the ceiling while a low cost local model fell to 24 to 37 points. A criterion level analysis revealed what run totals conceal. Container deployment was the dominant defect, failing first try in 44 percent of runs, with its failure rate shifting sharply across model generations while mean totals moved less than a point. The testing tool raised cost by 42 to 68 percent without improving functional score or reliability, even on interface visible criteria. Raising reasoning effort from High to xHigh lifted first try perfect runs from 28 percent to 89 percent and cut corrective prompts about five fold, for 9 to 29 percent more cost. A design oriented prompt raised visual quality, 4.5 versus 3.0 on a 5 point scale, without lifting function, and a one paragraph paraphrase of its directive reproduced the entire lift. The practical lesson is to match the fix to the failure: most first run failures came from weak reasoning, which a stronger model or more effort prevents, not from visible flaws a checking tool would catch.