ParanoiaEval: Benchmarking Unnecessary Defensive Work in Agentic Coding

Source facts

The preprint introduces a benchmark for unified evaluation of risk-treatment capabilities in coding agents. It maps Avoidance, Transfer, Mitigation, and Acceptance from software risk management to 200 evidence-controlled repository-level task pairs. The authors report unnecessary risk treatment in 11.2%–58.7% of runs across eight representative models and argue that task capability and appropriate risk treatment are distinct dimensions.

Analysis

ParanoiaEval frames over-defensive behavior as a measurable failure mode: an agent can be capable at the task while still responding poorly to evidence about risk. This connects coding-agent evaluation to established software-engineering practice rather than treating every defensive action as intrinsically beneficial.

Uncertainty/Doubt

• This is an arXiv preprint. The human calibration, task-pair design, and reported rates were not independently reproduced here.