Accelerating AI Agent Deployment via Simulated Evals

The Breakthrough

Nubank achieved a 20x increase in agent shipping velocity by replacing manual data curation and production-based A/B testing with a synthetic simulation framework that generates grounded, multi-turn conversation trajectories for offline evaluation.

What Actually Worked

• Grounded Persona Generation: The team uses the Snowglobe SDK to generate synthetic personas (e.g., a 34-year-old designer ordering a credit card) that include consistent, mocked account context, address data, and specific conversational tones.
• Tool Mocking: The simulation framework wraps the agent and intercepts tool calls, providing valid, mocked outputs that allow the agent to progress through complex, stateful workflows without needing live backend connections.
• Tightened Feedback Loop: By integrating simulation data into an automated evaluation pipeline, the team runs offline evals in minutes rather than waiting days for human-curated data or weeks for sparse, noisy production feedback.
• Sim-to-Real Validation: The team verified the simulation quality by comparing eval results against real production traces and human review, finding that simulated data is comparable to real-world interactions 80% of the time.

Before / After

• Shipping Velocity: Reduced from weeks of manual curation and production A/B testing to a few hours of simulation-based validation.
• Agent Performance: One specific agent saw a 2x increase in TNPS (customer satisfaction) and a 4% improvement in self-service rate (SSR) after adopting the simulation-driven optimization loop.

Context

Building AI agents for a bank with 135 million customers requires high reliability, but evaluating multi-turn, stateful agents is traditionally a bottleneck. Manual authoring is time-consuming, and relying on production traces is risky and slow. Nubank addressed this by building a simulation layer that allows them to test agent harnesses, prompt changes, and open-source model swaps in a sandbox environment before ever exposing them to live users.

Content References

{"type": "tool", "title": "Snowglobe", "url": "https://snowglobe.ai/", "context": "recommended"}, {"type": "paper", "title": "KDD Paper", "context": "mentioned"}