Agent Error Dataset: Scaling 50,000 Error--Diagnosis Pairs for Failure Analysis and Error-Aware Post-Training

Source facts

The preprint introduces 50,228 error-diagnosis pairs from 9,961 tasks across 33 environments, 19 harness families, and 23 policy models. Its Agentic Error-to-Training pipeline preserves traces and execution metadata, generates diagnoses and corrections, and checks them against recorded evidence. The abstract reports higher verifier pass rates for first-proposal corrections and improved exact-step agreement after diagnosis fine-tuning.

Analysis

The dataset operationalizes failure traces as training material rather than treating an unsuccessful rollout as only a negative reward. Its replay and diagnosis structure could support more systematic error taxonomies, recovery training, and cross-environment evaluation.

Uncertainty/Doubt

• This is an arXiv preprint. Dataset coverage, replay availability, diagnosis quality, and the reported training gains were not independently audited here.