Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

Three-Phase Evaluation of AI-Assisted Software Development Life Cycle

Source

arXiv

arxiv.org/abs/2607.05125v1

Read the source

Source

Plain-Language Brief

The paper evaluates the impact of increasing levels of AI autonomy on software development productivity, adherence to requirements, and developer cognitive workload using a web application case study. The study involved three phases: partial AI-assisted development with GitHub Copilot, an AI-exclusive workflow with GitHub Copilot, and an AI-exclusive workflow with AWS Kiro. Metrics included development effort, requirement adherence, AI-interaction efficiency, and mental workload measures. The results showed reduced development effort, improved requirement adherence, lower self-reported mental workload, and slightly increased developer frustration as AI autonomy increased. The AWS Kiro phase had the best overall performance.

What the Document Says

  • The paper evaluates an exploratory study on the impact of increasing levels of AI autonomy on software development productivity, requirement adherence, and developer cognitive workload.
  • The study involved four developers reimplementing a full-stack web application across three sequential phases: partial AI-assisted with GitHub Copilot, an AI-exclusive workflow with GitHub Copilot, and an AI-exclusive workflow with AWS Kiro.
  • Metrics used in the evaluation included development effort (hours), requirement adherence (RITM score), AI-interaction efficiency, and NASA-TLX workload measures.
  • Results showed that higher levels of AI autonomy were associated with reduced development effort, improved requirement adherence, lower self-reported mental workload, and slightly increased developer frustration.
  • The AWS Kiro phase achieved the strongest overall performance on most measured dimensions.

Key Concepts Mentioned

  • AI autonomy
  • software development
  • productivity
  • requirement adherence
  • developer cognitive workload
  • GitHub Copilot
  • AWS Kiro

Claims Made by the Document

  • Higher levels of AI autonomy reduce development effort.
  • Higher levels of AI autonomy improve requirement adherence.
  • Higher levels of AI autonomy lower self-reported mental workload.
  • Developer frustration increases modestly with higher levels of AI autonomy.

Evidence Used

  • Results from a case study on reimplementing a full-stack web application.

Methods / Approach

  • Case study
  • Reimplementation of a full-stack web application in three sequential phases
  • Evaluation metrics: development effort, requirement adherence, AI-interaction efficiency, and NASA-TLX workload measures

Limitations Stated by the Authors

  • The study is exploratory and involves a small team of four developers working on one case study.

Limitations Noted by the Agent

Agent interpretation — not from the source.

  • The study does not address long-term effects, scalability, or robustness of AI-assisted software development.
  • The study does not compare the results with traditional software development methods.

Open Questions

  • What are the long-term effects, scalability, and robustness of AI-assisted software development?
  • How do AI-assisted software development methods compare to traditional methods in terms of productivity, requirement adherence, and developer cognitive workload?

Notes for Future Q&A

May include interpretation. Labeled where it does.

  • The study uses GitHub Copilot and AWS Kiro as AI tools for software development.
  • RITM stands for Requirements Traceability Matrix.

Raw Abstract / Excerpt

This paper presents an exploratory evaluation of how increasing levels of AI autonomy affect software development productivity, requirement adherence, and developer cognitive workload. A team of four developers reimplemented the same full-stack web application across three sequential phases: partial AI-assisted development using GitHub Copilot, an AI-exclusive workflow using GitHub Copilot, and an AI-exclusive workflow using AWS Kiro. Evaluation metrics included development effort (hours), requirement adherence (RITM score), AI-interaction efficiency, and NASA-TLX workload measures. Across phases, higher levels of AI autonomy were associated with reduced development effort, improved requirement adherence, and lower self-reported mental workload, while developer frustration increased modestly. The AWS Kiro phase achieved the strongest overall performance on most measured dimensions, suggesting that tooling architecture may influence outcomes independently of AI autonomy level.