Shipping Regulated Healthcare AI via Simulation and Automated Evals

The Simulation-First Safety Framework

Ufonia manages the clinical voice agent Dora by replacing traditional reactive deployment loops with a simulation-based safety stack called Matrix. Because clinical errors can cause patient harm, the team cannot rely on live A/B testing or rollbacks. Instead, they simulate thousands of clinical interactions using two distinct LLM roles: a patient simulator (Patbot) and an automated judge (Bev Judge). This framework allows the team to test for specific hazards, such as hallucinated medical advice or ignored red-flag symptoms, before any code reaches a real patient.

Validating the Judge and Optimizing Prompts

The team validates their automated judge against a ground-truth corpus of 240 cases labeled by 10 clinicians across 10 specialties. The judge achieved an F1 score of 0.96 and near-perfect sensitivity, ensuring that potential hazards are flagged even at the risk of occasional false alarms. Once the system identifies a failure, the team uses genetic prompt optimization (JPE) to automatically refine system prompts. Rather than hand-tuning, they define a cost matrix that penalizes missed red flags more heavily than false alarms, allowing the optimizer to iterate toward a prompt that satisfies specific clinical safety constraints.

Shipping Evidence, Not Models

The final deployment strategy treats simulation as a necessary but insufficient condition for release. The team ships the evidence of safety rather than the model itself. Every deployment follows a staged rollout: simulation, supervised clinical evaluation, and finally, monitored autonomy. This process creates a flywheel where every real-world call generates new data, which is then fed back into the simulation framework to identify new hazards and refine the system for future iterations.