Reasoning effort, not tool access, buys first-try reliability in agentic code generation: an observational study

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Achint Mehta
• Venue: arXiv: cs.SE
• Published: 2026-07-02
• Collected: 2026-07-03
• URL: http://arxiv.org/abs/2607.02436v1
• PDF: https://arxiv.org/pdf/2607.02436v1
• DOI: UNKNOWN

Plain-Language Brief

A study examined the impact of additional capabilities on agentic coding assistants in terms of software quality and reliability. The results showed that while high-tier models performed well, a low-cost local model underperformed. Increasing reasoning effort led to a significant improvement in first-try reliability, reducing corrective prompts by almost fivefold, with only minimal additional cost.

What the Document Says

• Agentic coding assistants were tested for their performance.
• The test involved building a real-time retrospective board using one detailed specification.
• Each run was scored on a 14-criterion functional rubric and visual quality review.
• The runs used various model generations, agent harnesses, reasoning effort levels, testing tools, and design-oriented prompts.
• Frontier models generally performed well, but a low-cost local model underperformed (24-37 points out of 42).
• Container deployment was the most common defect, failing in 44% of runs.
• The testing tool increased cost by 42-68% without improving functional score or reliability.
• Increasing reasoning effort from High to xHigh led to more first-try perfect runs (from 28% to 89%) and fewer corrective prompts (9-29% more cost).
• A design-oriented prompt improved visual quality without lifting function.
• The study suggests focusing on improving reasoning rather than relying on checking tools for visible flaws.

Key Concepts Mentioned

• agentic coding assistants
• reality retrospective board
• functional rubric
• model generations
• reasoning effort levels
• testing tool
• design-oriented prompts

Claims Made by the Document

• High-tier models perform well.
• A low-cost local model underperforms.
• Container deployment is a common defect.
• The testing tool does not improve functional score or reliability.
• Increasing reasoning effort improves first-try reliability and reduces corrective prompts.
• A design-oriented prompt improves visual quality without affecting function.

Evidence Used

• Ninety independent agent runs were conducted
• Each run was scored on a fixed rubric
• The results were analyzed by criterion level

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Agentic coding assistants are increasingly given extra capabilities, such as browser based testing tools and design oriented system prompts, on the assumption that more capability yields better software. This study tested that assumption directly. Ninety independent agent runs built the same application, a real time retrospective board, from one detailed specification, each scored on a fixed 14 criterion functional rubric (42 point maximum) and a visual quality review. The runs spanned several model generations, two agent harnesses, two reasoning effort levels, a testing tool, and two design oriented prompts. Capability tier dominated: frontier models clustered near the ceiling while a low cost local model fell to 24 to 37 points. A criterion level analysis revealed what run totals conceal. Container deployment was the dominant defect, failing first try in 44 percent of runs, with its failure rate shifting sharply across model generations while mean totals moved less than a point. The testing tool raised cost by 42 to 68 percent without improving functional score or reliability, even on interface visible criteria. Raising reasoning effort from High to xHigh lifted first try perfect runs from 28 percent to 89 percent and cut corrective prompts about five fold, for 9 to 29 percent more cost. A design oriented prompt raised visual quality, 4.5 versus 3.0 on a 5 point scale, without lifting function, and a one paragraph paraphrase of its directive reproduced the entire lift. The practical lesson is to match the fix to the failure: most first run failures came from weak reasoning, which a stronger model or more effort prevents, not from visible flaws a checking tool would catch.