Screen Before You Serve: Simulation for Production Customer Experience AI Agents at 140M Scale

Source facts

The paper presents a hypothesis-driven simulation workflow for screening customer-experience agents before deployment. Synthetic customers and simulated tool outputs exercise multi-step workflows without invoking production backends. The authors report using Snowglobe on Nubank's Card Delivery and Card Management agents, finding high correlation between simulated and production version-level binary evaluator scores across four deployed versions. The abstract reports simulation-guided improvement in a live A/B test and screening of open-weight configurations across more than 16,000 simulated conversations.

Analysis

This is a concrete enterprise-evaluation pattern: use simulated interaction and tool environments to widen model, prompt, and reasoning-configuration exploration before exposing customers. The claimed production correlation is potentially useful, but simulator validity and evaluator stability determine whether the workflow catches rare operational failures.

Uncertainty/Doubt

• This is an arXiv preprint describing a company deployment case. The Nubank deployment context, simulator-to-production correlation, A/B-test effects, and scale claims were not independently reproduced here.