Scaling Voice Agents: Lessons from Self-Driving Infrastructure
Y Combinatorgo watch the original →
the gist
Brooke Hopkins, CEO of Coval, explains how applying self-driving car simulation and observability techniques to enterprise voice agents creates a necessary, scalable infrastructure for production AI.
The Parallels Between Autonomous Driving and Voice AI
Brooke Hopkins argues that the challenges of deploying voice agents in production are fundamentally similar to the problems solved in autonomous vehicle development at Waymo. Both systems require a robust 'perception-reasoning-control' loop. In voice, this translates to speech-to-text (perception), LLM reasoning (planning), and text-to-speech (control). Because these systems are autonomous and act on behalf of users, they require rigorous simulation and observability to prevent failures that are far more egregious than those of human agents, such as vocal hallucinations or unexpected behavioral shifts.
Moving Beyond Simple Evals
Many early voice agent developers over-index on word error rate (WER) or basic transcription accuracy. Hopkins suggests that these metrics are often vanity metrics; the real challenge lies in ensuring the agent understands intent and successfully completes multi-step workflows. Coval focuses on providing the infrastructure for 'scalable evaluation,' allowing enterprises to test millions of conversations without relying on manual QA or live production testing, which is both expensive and risky.
The Enterprise Wedge
Coval’s success stems from focusing on the enterprise sector early. While startups move faster, enterprises possess long-term roadmaps and complex, existing call-flow infrastructure (IVR trees) that make them ideal candidates for voice agent integration. Hopkins emphasizes that the 'aha' moment for the company came when a potential customer offered to pay for a solution before a single line of code was written—a clear signal of market pull and acute pain. By focusing on the needs of Fortune 500 companies, Coval built tools that support hundreds of engineers collaborating on a single AI system, a direct application of Hopkins' experience at Waymo.
Building for Scale
As voice agents move from simple customer support tasks to complex concierge and logistics roles, the infrastructure must evolve to handle high-volume, unstructured data. Hopkins notes that the future of voice AI lies in 'controllable real-time models' that can share embeddings and context across the perception, reasoning, and control layers, rather than relying on brittle, disconnected cascading architectures.