Coding agents and software engineering
61 notes, newest first. Includes the vault tags coding agents, code generation, software engineering, programming.
ParanoiaEval: Benchmarking Unnecessary Defensive Work in Agentic Coding
The preprint introduces a benchmark for unified evaluation of risk-treatment capabilities in coding agents. It maps Avoidance, Transfer, Mitigation, and Acceptance from software risk management to 200…
Shockingly Simple Self-retrospection Improves Agentic Models Without RL
The preprint introduces Retrospection-Only Fine-Tuning (ROFT), an online procedure in which an agent attempts a task, observes feedback, generates a retrospective explanation, and fine-tunes only on the explanation…
SciWalker: Synthesizing Scientific Coding Problems with Operator Graphs and Execution Feedback
The preprint describes a framework for synthesizing scientific coding problems by sampling operator chains from graphs built from scientific library interfaces and operation modes. An LLM generates problem statements…
CliffCompaction: Cost-Efficient Compaction for Long-Horizon Coding Agents
CliffCompaction is an autocompaction technique for long-horizon coding agents that reduces context cost while preserving or improving benchmark performance. It only truncates or drops content rather than rephrasing it…
Grow the Harness, Not the Context: From Strategy-Free Scaffolds to Reusable Specialist Agents
Growing Harness learns the agent harness itself from a strategy-free scaffold. Function-level traces localize failures, an optimizer repairs bounded code surfaces, and a held-out success-first gate rolls back repairs…
SWE-Serve: Benchmarking Agentic Engineering For Production Inference Serving
SWE-Serve contains 53 repository-grounded tasks derived from production changes to SGLang, spanning six inference-engineering families. Tasks use hidden functional and regression tests, end-to-end serving tests…
CodeMidas: Scaling Agentic Coding RL Environments from Code Itself
CodeMidas converts implemented functionality in open-source codebases into executable reinforcement-learning environments using source code as the only task-specific input. Agents formulate behavioral specifications…
An Empirical Study of Harness Design for Coding Agents
The paper varies planning, action space, and context management in a fixed coding harness and evaluates 176 matched settings across four models on SWE-Bench Verified and Terminal-Bench 2.1. It reports that context…
From Agent Behaviour to Agent-Friendly Documentation: An Empirical Study of How Coding Agents Discover, Read, and Write Technical Documentation
A study investigates how autonomous coding agents interact with technical documentation, revealing that they mostly work with internal documents and notes, and less often with classical technical documentation or API…
Repo0: Design-Driven Zero-to-All Code Generation
The paper introduces Repo0, a framework for zero-to-all code generation that maintains an explicit architectural state using a Dual-Directed Acyclic Graph (Dual-DAG). It iteratively evolves component boundaries guided…
TDD-Agent: Test-Driven Reasoning for Code Generation
Researchers introduce TDD-Agent, a system that uses the test-driven development (TDD) paradigm for code generation. The system first generates tests to clarify expected behaviors before implementation and then refines…
When Agents Coordinate: Measuring Coordination in Multi-Agent AI Coding
The authors studied how AI coding agents coordinate while working on programming tasks. They developed a method to measure this coordination by representing each run as a network with nodes (agents and files) connected…
VICBench: A Multi-Language Benchmark for Code Vulnerability Detection
The paper introduces VICBench, a new benchmark dataset for evaluating code vulnerability detection tools. VICBench contains 100 verified vulnerability-inducing commits (VICs) across 88 projects in Python, Java, and…
VICBench: A Multi-Language Benchmark for Code Vulnerability Detection
Researchers have created a benchmark dataset called VICBench for evaluating code vulnerability detection tools. The dataset contains 100 verified vulnerabilities across 88 projects in Python, Java, and C++, covering 48…
ChemWorld: Programmable Chemical Worlds for Controlled and Replayable Agent Experimentation
The paper introduces ChemWorld, a programmable chemical environment designed for repeatable and controllable agent experimentation in autonomous chemistry. The environment allows researchers to compile reusable process…
OctoLong: Mid-Training On Cross-Repository Code Contexts Enhances Long-Context Modeling
In this research, a new pipeline called OctoLong is introduced to enhance the training of language models (LM) by providing them with long code contexts. This is done through the use of an AST parser, a language server…
RepoProbe: Benchmarking Architecture-Aware Repository Comprehension with Checklists
The text discusses a new benchmark called RepoProbe for evaluating the ability of Large Language Models (LLMs) to understand repository-scale code. The benchmark uses open-ended questions from GitHub Discussions and…
Agentic Method for Deterministic Validation of Legacy Code Migration
The paper proposes a new method called 'Locksmith Loop' for testing the migration of COBOL programs to Java. This method is designed to improve coverage beyond input-search plateaus, reaching near complete coverage in…
Looping Is Not Reliability: State-Bound Evidence and Typed Revision Contracts for Agentic Code Repair
The paper discusses the unreliability of looping in coding agents and the challenge of ensuring that correct patches are retained, verified, and submitted. The authors conducted several studies to understand this gap…
Looping Is Not Reliability: State-Bound Evidence and Typed Revision Contracts for Agentic Code Repair
The paper explores the lack of reliability guarantees in coding agents that use generate-test-revise loops. The authors conduct studies to understand the gap between finding a correct patch and retaining, verifying…
From Resource Flow to Executable Tests: Petri-Net-Guided LLM Test Generation for Concurrent Stateful Rust APIs
The authors present a methodology using Petri nets to generate tests for concurrent stateful Rust APIs, aiming to bridge the gap between formal scenario design and low-cost test concretization.
From Resource Flow to Executable Tests: Petri-Net-Guided LLM Test Generation for Concurrent Stateful Rust APIs
The authors propose a methodology for generating executable tests for concurrent stateful Rust APIs using Petri nets. This approach aims to bridge the gap between formal scenario design and low-cost test…
From Resource Flow to Executable Tests: Petri-Net-Guided LLM Test Generation for Concurrent Stateful Rust APIs
The paper presents a methodology using Petri nets for test generation in concurrent stateful Rust APIs, aiming to bridge the gap between formal scenario design and low-cost test concretization. It allows large language…
ICAE-Bench: Evaluating Coding Agents as Interactive Project Builders
The paper introduces ICAE-Bench, a benchmark for evaluating coding agents in interactive project building settings. Instead of being limited to completing code under fully specified instructions, these agents are…
Multi-stage Dynamic Selection for Cross-Project Defect Prediction
The paper proposes a new framework for Cross-Project Defect Prediction (CPDP) that addresses distribution shift issues between training and target projects by using a two-stage multiple classifier system selection…
Autoresearch with Coding Agents: Generalizers and Metric-Maximizers on Quran Recitation Data
Two coding agents were tested on a production task involving Quranic verse identification and transcription splitting. Both agents independently developed similar algorithms, but one (OpenAI Codex) performed poorly due…
TRIM: Reducing AI-Generated CodeSlop via Agent Trajectory Minimization
The research presented in 'TRIM: Reducing AI-Generated CodeSlop via Agent Trajectory Minimization' aims to address the issue of verbose and redundant code generated by coding agents, which are increasingly used for…
Early Adoption of Agentic Coding Tools by GitHub Projects
The paper investigates the early adoption of automated coding tools by GitHub projects. It found that while intensive adoption is concentrated in a few projects, smaller projects tend to have higher participation…
Early Adoption of Agentic Coding Tools by GitHub Projects
This research investigates the adoption and management of agentic coding tools in software development on GitHub. The study analyzes 25,264 agentic pull requests from 2,361 popular repositories to understand (1) how…
Generative Compilation: On-the-Fly Compiler Feedback as AI Generates Code
Researchers introduce 'generative compilation', an approach to get feedback from compilers on partially generated code during the code generation process. The method aims to reduce errors and improve the functionality…
Multi-Perspective Agentic Program Repair via Code Property Graphs and Temporal Execution Graphs
A new automated program repair framework called CT-Repair is presented, which addresses two limitations of existing large language models in this area. The first issue is the large and repetitive nature of raw…
Heuristic Learning for Active Flow Control Using Coding Agents
A new method for active flow control is proposed in this research paper. Instead of optimizing neural network parameters, the authors use coding agents to search directly for explicit executable feedback laws. The…
ProjAgent: Procedural Similarity Retrieval for Repository-Level Code Generation
ProjAgent is a new system for generating code at the repository level. It introduces a method called 'procedural similarity' to retrieve relevant functions from the repository, even if those functions have different…
ProjAgent: Procedural Similarity Retrieval for Repository-Level Code Generation
The paper proposes ProjAgent, a new system for generating code at the repository level by introducing 'procedural similarity' as a retrieval signal. This method decomposes the target function into steps and retrieves…
ProjAgent: Procedural Similarity Retrieval for Repository-Level Code Generation
The paper introduces ProjAgent, a system for generating code at the repository level that considers procedural similarity as a new method for retrieving appropriate functions. ProjAgent deconstructs target functions…
ProjAgent: Procedural Similarity Retrieval for Repository-Level Code Generation
The paper proposes ProjAgent, a system for generating code at the repository level. It introduces 'procedural similarity' as a new way to retrieve relevant functions, addressing a limitation of existing methods that…
ProjAgent: Procedural Similarity Retrieval for Repository-Level Code Generation
The paper presents ProjAgent, a system for generating code at the repository level that uses 'procedural similarity' as a new signal in the retrieval process. This system decomposes the target function into…
Harnessing Code Agents for Automatic Software Verification
The authors propose a new method for automatic software verification using large language models (LLMs) in the form of 'code agents'. Instead of wireting a fixed proof strategy, they allow the code agent to choose its…
RuBench: A Repository-Level Agentic Coding Benchmark with Natively Authored Russian Task Specifications
The author introduces RuBench 1.0, a benchmark for evaluating coding agents that use native language tasks instead of English. The benchmark includes 25 tasks mined from five open-source repositories and is judged by…
RuBench: A Repository-Level Agentic Coding Benchmark with Natively Authored Russian Task Specifications
The paper introduces RuBench 1.0, a new benchmark for coding agents that uses tasks specified in Russian rather than English. The benchmark is designed to measure the performance of coding agents when they are given…
Distributed Attacks in Persistent-State AI Control
Researchers introduce Iterative VibeCoding, a setting for AI control that studies the deployment of autonomous AI coding agents and their potential for distributed attacks. The study finds that gradual attacks, which…
Reasoning effort, not tool access, buys first-try reliability in agentic code generation: an observational study
A study examined the impact of additional capabilities on agentic coding assistants in terms of software quality and reliability. The results showed that while high-tier models performed well, a low-cost local model…
Reasoning effort, not tool access, buys first-try reliability in agentic code generation: an observational study
A study found that the level of reasoning effort in agentic code generation, rather than the availability of additional capabilities like testing tools and design prompts, significantly affects the reliability of…
Reasoning effort, not tool access, buys first-try reliability in agentic code generation: an observational study
A study compared different agentic coding assistants and their abilities in building a real-time retrospective board. The results showed that the level of reasoning effort, rather than access to additional tools or…
Reasoning effort, not tool access, buys first-try reliability in agentic code generation: an observational study
A study found that more reasoning effort in agentic code generation (AI programs assisting human coders) leads to higher reliability and fewer errors, rather than the use of additional tools or capabilities.
TestEvo-Bench: An Executable and Live Benchmark for Test and Code Co-Evolution
The paper introduces TestEvo-Bench, a new benchmark for testing software that evolves with its code. This benchmark is designed to evaluate how well a test automation agent understands the impact of a code change on…
Understanding Agent-Based Patching of Compiler Missed Optimizations
The paper presents a study on the effectiveness of AI agents in addressing missed optimizations in compilers. The authors found that while coding agents can often optimize given examples, the generated patches…
Are Performance-Optimization Benchmarks Reliably Measuring Coding Agents?
The paper evaluates three repository-level performance optimization benchmarks (GSO, SWE-Perf, and SWE-efficiency) used to measure coding agents. The authors find issues with the stability of runtime for these…
Are Performance-Optimization Benchmarks Reliably Measuring Coding Agents?
The paper evaluates three performance-optimization benchmarks (GSO, SWE-Perf, and SWE-efficiency) used to assess coding agents. The authors find that these benchmarks have issues with stability, scoring rules, and the…
LLVM-Bench: Benchmarking and Advancing Large Language Models for LLVM Compiler Issue Resolution
The authors present LLVM-Bench, a large-scale benchmark for testing the effectiveness of large language models (LLMs) in resolving issues related to the LLVM compiler. They found that current LLM-based techniques are…
SWE-Doctor: Guiding Software Engineering Agents with Runtime Diagnosis from Multi-Faceted Bug Reproduction Tests
Researchers propose a new software agent called SWE-Doctor that uses multiple bug reproduction tests (BRTs) to generate more effective patches for resolving software issues. The agent first generates multi-faceted…
AxDafny: Agentic Verified Code Generation in Dafny
Researchers present AxDafny, a framework for generating executable code and proof artifacts in Dafny using a verifier-guided repair method. The study uses a benchmark called LiveCodeBench-Pro-Dafny with 250 programming…
ManimAgent: Self-Evolving Multimodal Agents for Visual Education
Researchers present ManimAgent, a self-evolving multimodal agent that uses a dual-channel Episodic Memory Bank to carry reflection experience across tasks in generating Python code using the Manim library for…
ManimAgent: Self-Evolving Multimodal Agents for Visual Education
The research presents ManimAgent, a self-evolving multimodal agent that learns from its failures and successes in generating Python code using the Manim library to render mathematical animations. The agent stores these…
TraceLab: Characterizing Coding Agent Workloads for LLM Serving
Researchers collect and release a dataset of approximately 4,300 coding-agent sessions from their own use of Claude Code and Codex. The data shows that coding-agent workloads feature long autonomous loops, long…
TraceLab: Characterizing Coding Agent Workloads for LLM Serving
Researchers have collected and released a dataset of 4300 coding agent sessions, containing around 350,000 LLM steps and 430,000 tool calls from their own use of Claude Code and Codex. The analysis shows that…
TraceLab: Characterizing Coding Agent Workloads for LLM Serving
Researchers present TraceLab, a dataset of 4,300 coding-agent sessions collected from their own usage of Claude Code and Codex. The data reveals characteristics of these workloads, such as long autonomous loops, long…
A Deterministic Control Plane for LLM Coding Agents
The author proposes a new control plane for coding agents, which are tools that have broad access to files and shell commands. The current configuration layer of these agents is poorly managed, with configurations…
The Spec Growth Engine: Spec-Anchored, Code-Coupled, Drift-Enforced Architecture for AI-Assisted Software Development
The paper presents the Spec Growth Engine, a framework to address the structural failure modes introduced by AI coding agents in software development. These failure modes include context explosion and silent spec-code…
The Spec Growth Engine: Spec-Anchored, Code-Coupled, Drift-Enforced Architecture for AI-Assisted Software Development
The paper proposes a new framework called the Spec Growth Engine to improve AI coding agents during software development. The framework addresses two main issues: context explosion and silent spec-code drift. It uses a…
Bayesian control for coding agents
The paper proposes a Bayesian controller for coding agents that make decisions based on a belief over candidate correctness, rather than fixed rules. This method is found to be most useful when verification is costly…
