5 Voice Agent Failure Modes: From Demo to Production

The Latency-Intelligence Tradeoff

Most voice agents fail because they rely on frontier LLMs that are too slow for real-time interaction. While LLM research focuses on "thinking" models, voice agents require "thinking" to be disabled to maintain a time-to-first-audio (TTFA) under 300ms. Venky B argues that teams should move away from general-purpose frontier models toward smaller, self-hosted open-source models like Qwen 3.5 or Gemma 4. The key is balancing token fertility (tokens per word) with instruction-following capability. For multilingual use cases, Gemma 4 is currently superior, offering better token efficiency. For production, Mixture of Experts (MoE) models often work out-of-the-box, while fine-tuning should be reserved for 8B+ parameter models to avoid degrading the model's base intelligence.

Treating Transcription as Brittle

Transcription engines are inherently unreliable, especially with proper nouns, code-switching, and noisy audio. Instead of relying on raw transcription, developers must implement a post-processing layer. This involves using an LLM to "clean" the transcript based on domain context—for example, correcting a phone number where a digit was misidentified as a letter. Furthermore, developers should employ dynamic keyword boosting, injecting context-specific vocabulary only when the agent is in a specific state, rather than polluting the transcription engine with a massive, static keyword list.

Data Collection as a Typed Field Problem

Perhaps the most significant failure mode is treating voice input as a free-form conversation. By reframing data collection as a structured form-filling exercise (using concepts like Pydantic or TypeScript interfaces), accuracy can jump from 30% to 95%. Each field should be treated as a unit test. If a user provides a phone number, the system should validate length and format immediately. If the input is ambiguous, the agent should not guess; it should trigger a specific confirmation flow or request a repeat. This "shape-first" approach removes the need for constant, brittle prompt engineering.

Normalization and Turn Detection

Never pipe raw LLM output directly into a Text-to-Speech (TTS) engine. An intermediary normalization layer is mandatory to strip emojis, markdown, and formatting artifacts that confuse synthesis engines. Additionally, robust turn detection (barge-in) is critical for a natural experience. If the agent cannot detect when a user is interrupting, the conversation feels robotic and frustrating, leading to high abandonment rates.