Reasoning effort, not tool access, buys first-try reliability in agentic code generation: an observational study

Abstract: Agentic coding assistants are increasingly given extra capabilities, such as browser based testing tools and design oriented system prompts, on the assumption that more capability yields better software. This study tested that assumption directly. Ninety independent agent runs built the same application, a real time retrospective board, from one detailed specification, each scored on a fixed 14 criterion functional rubric (42 point maximum) and a visual quality review. The runs spanned several model generations, two agent harnesses, two reasoning effort levels, a testing tool, and two design oriented prompts. Capability tier dominated: frontier models clustered near the ceiling while a low cost local model fell to 24 to 37 points. A criterion level analysis revealed what run totals conceal. Container deployment was the dominant defect, failing first try in 44 percent of runs, with its failure rate shifting sharply across model generations while mean totals moved less than a point. The testing tool raised cost by 42 to 68 percent without improving functional score or reliability, even on interface visible criteria. Raising reasoning effort from High to xHigh lifted first try perfect runs from 28 percent to 89 percent and cut corrective prompts about five fold, for 9 to 29 percent more cost. A design oriented prompt raised visual quality, 4.5 versus 3.0 on a 5 point scale, without lifting function, and a one paragraph paraphrase of its directive reproduced the entire lift. The practical lesson is to match the fix to the failure: most first run failures came from weak reasoning, which a stronger model or more effort prevents, not from visible flaws a checking tool would catch.

• The paper demonstrates that enhanced agent reasoning effort significantly improves first-try functional reliability, with success rates rising from 28% to 89% in high-tier models.
• It employs a controlled ablation study to isolate model capability, tool augmentation, and design prompts, revealing that common UI testing tools add cost without commensurate reliability gains.
• The study highlights that visually appealing UIs require concise design prompts rather than increased model size or reasoning effort, guiding future agentic benchmarking methods.

Reasoning Effort and the Limits of Tool Augmentation in Agentic Code Generation

Overview

The paper "Reasoning effort, not tool access, buys first-try reliability in agentic code generation: an observational study" (2607.02436) presents an experimental and systematic evaluation of agentic code generation for a complex software task, dissecting the impact of model tier, tool augmentation, system prompts, and explicit reasoning effort on first-try functional reliability and visual quality. Through ninety identically specified, independently executed runs building a real-time retrospective board application, the study offers strong empirical evidence that factors such as model architecture and enforced agent reasoning effort have a pronounced, non-substitutable effect on unaided functional correctness, while popular augmentations (notably browser-driven UI testing tools) consistently fail to deliver reliability improvements commensurate with their cost.

Experimental Structure and Variable Isolation

The study utilizes a rigorously controlled multitiered ablation design. Agents of various capability tiers (Claude Opus 4.7/4.6, Sonnet 4.6, Gemini 3.1, Qwen) are tasked, via two agent harnesses, to implement a standardized OpenSpec for a retrospective board application. Key manipulated variables include: (1) access to a Playwright-driven browser testing framework, (2) augmentation with a design-oriented system prompt (both full and a deliberately concise paraphrase), and (3) adjustable agent "reasoning effort" with replicates at both High and xHigh settings, isolating deliberative capacity from generative power or augmentation. Each run undergoes granular rubric-based functional scoring and independently rated UI visual analysis.

Key Findings

Model Capability Tier as Primary Determinant

The dominant source of functional performance variance is model capability, with frontier models (Opus 4.7/4.6, Sonnet 4.6) routinely achieving ceiling-skirting functional scores (mean ≈ 41/42); in contrast, a low-cost local LLM (Qwen) incurs a substantial performance deficit (down to 24/42) and demonstrates radically higher token utilization and cost estimations, reflecting heavy, often futile, self-debugging iterations. Notably, token and API cost within the top capability tier displays considerable range, but does not correlate with higher functional yield within that band, indicating further diminishing returns above a capability threshold.

Reasoning Effort as the Controlling Factor for First-Try Reliability

A pivotal result is the direct measurement of unaided functional reliability. At standard (High) reasoning effort, first-try perfect runs are limited (28% for Opus 4.7 High), but when increased to xHigh, first-try perfection surges to 89%, and the necessity for corrective, user-issued prompts drops by a factor of five. Crucially, this leap in reliability accrues at a modest session cost increase (median 9–29%). The effect is consistent across base, tool-augmented, and design-augmented cells, save for a slight attenuation under design prompts due to their increased interaction complexity.

Inutility of UI Testing Tools for Reliability

Contrary to prevalent design assumptions, addition of the Playwright-based interactive browser testing tool has no measurable effect on functional score or first-try reliability, neither for global functional totals nor interface-visible defect classes. It consistently and substantially increases cost (42–68%) due to session context re-reading, with no commensurate gain. The critical insight is failure mode specificity: nearly all first-attempt failures cluster in build and deployment environment criteria (Docker, development environment setup), outside the reach of interface-driven testing tools.

Design-Prompt-Driven UI Quality Decoupled from Reliability

The study establishes that high-fidelity, visually modern UIs can be elicited by general design-oriented prompting, and that this qualitative lift is prompt-content specific, not dependent on model tier, system prompt length, or emphatic tone. Neither raising model capability nor agent reasoning effort increases visual polish in the absence of these instructions. However, the increased visual ambition introduces reliability risks: design-prompted runs exhibit a higher rate of failures on complex UI interaction criteria (notably drag-and-drop), even at maximum reasoning effort.

Minimal Prompt Sufficiency and Prompt Injection Sensitivity

A one-paragraph paraphrase of the design directive reproduces the entirety of the visual lift seen from the lengthy third-party prompt, demonstrating that prompt content—rather than form, emphasis, or verbosity—is the critical variable. This signals vulnerability and strong effect size for prompt injection in agentic frameworks, a phenomenon with direct implications for both research comparison methodologies and security considerations.

Structural Variability and Sampling Non-Determinism

Replicated runs even under identical agent and prompt configurations generate significant variance in implementation strategy and codebase structure, e.g., total file count, source lines, and hand-authored CSS. Such variance shrinks with increased model generation quality and higher reasoning effort but remains irreducible by token sampling controls due to platform-imposed restrictions, especially on leading cloud APIs.

Implications for Benchmarking and Engineering Practice

Methodological Recommendations

• Post-repair (after human intervention) functional scores significantly underestimate differences in first-try performance between model generations and configurations. The study recommends supplementing aggregate functional scores with explicit reporting of first-try reliability and repair burden.
• Augmentation strategies (e.g., UI testing tools) require quantitative, not anecdotal, validation for practical benefit, particularly considering the non-obvious cost/reliability trade-offs and dominant non-interface-related failure modes in real software tasks.

Guidance for Agentic Software Engineering

• Critical engineering effort should be allocated to increasing agent reasoning (or deliberative) effort and, where possible, leveraging the highest model tier available, as these sum factors dominate unaided pass rates and reduce downstream intervention cost.
• Visual quality enhancements should be explicitly solicited via concise design directives. Increasing reasoning effort or model size without such instructions does not elicit visually polished UI output.
• Heavy tool augmentation should only be applied in a failure-mode-targeted manner. Misaligned tool deployment increases cost without contributing to net reliability.
• Prompt injection, whether malicious or accidental (e.g., by agent frameworks), poses substantial risks given its capability to reshape agent output (functionally or visually) without user visibility or control.

Speculative Directions

• Deliberative depth (reasoning effort) appears to meaningfully compress run-to-run stochastic variance and eliminate certain classes of environment failures, implying that future advances should focus as much on meta-cognitive orchestration within agents as on raw generative power.
• The observed independence between visual capability and functional reliability, as well as between code structure and functional score, suggests further research is warranted on developing corpus-aligned, domain-adaptive, and interpretable control levers for agentic code generation.

Conclusion

This study robustly delineates the boundaries of value-add between model architecture, explicit agent effort, automated tool augmentation, and system prompt engineering in agentic code generation. The empirical findings support a direct operational maxim: to reliably automate complex software tasks, invest first in model quality and agentic reasoning effort, invoke specificity in UI desiderata, and apply verification tools judiciously and only where empirically shown to address salient failure modes. High-level claims regarding benchmarking and prompt injection sensitivity also raise critical questions for the design and interpretation of future agentic evaluation suites, and carry concrete implications for practical LLM-powered software co-engineering.