FDE-Gym: The Last Mile of AI Delivery

Source facts

FDE-Gym presents 95 end-to-end delivery tasks across 20 customer scenarios and 18 sectors. Agents must clarify ambiguous requirements, interact with scripted stakeholders, operate in production-like systems, and improve a customer success metric. Across 13 configurations from seven providers, the best configuration achieved a mean score of 0.56 and resolved 14 of 95 tasks at the paper's high-success threshold. The authors identify discovery of unstated constraints as a primary bottleneck.

Analysis

The benchmark targets the gap between solving explicit tickets and delivering effective systems in live customer contexts. Its scoring design makes stakeholder interaction costs and business uplift part of the evaluation surface, which may expose deployment failures hidden by well-specified task benchmarks.

Uncertainty/Doubt

• This is an arXiv preprint. The benchmark construction, provider configurations, and reported scores were not independently reproduced here.