Engineering Low-Latency Voice Agents

The Pipeline Architecture

Voice agents currently rely on a cascading pipeline: audio input flows through a speech-to-text (STT) model, an LLM for reasoning and tool calling, and finally a text-to-speech (TTS) model. To maintain a human-like conversational cadence, the total system latency must remain under 500ms, as users typically disconnect if response times exceed one second. The LLM is the primary latency bottleneck, requiring a target time-to-first-token (TTFT) of 200 to 300ms.

Optimizing for Speed and Intelligence

To balance model intelligence with strict latency budgets, engineers should focus on the following strategies:

• Colocation: Moving inference models into the same data center as the agent orchestrator reduces network latency from ~75ms to ~5ms, providing a 30% overhead reduction in an already optimized pipeline.
• Thinker-Talker Pattern: Deploy a small, fast LLM to handle conversation flow and simple responses. When a complex request or tool call is required, the small model triggers a single call to a larger, more capable model, keeping the primary interaction path fast.
• Model Selection: Use LLMs in the 8B to 30B parameter range. Models larger than this typically exceed latency budgets, while smaller models often fail to execute complex tool-calling instructions reliably.
• Streaming STT: Transition from batch-based models (like standard Whisper) to streaming-native architectures that utilize cached activations and limited look-ahead windows to minimize time-to-complete-transcript.

Performance Benchmarks

• STT: Target a P90 latency of under 100ms with a word error rate (WER) of approximately 6%.
• TTS: Aim for a real-time factor (RTF) of less than 1.0 to prevent audio buffering during playback.
• Network: Reducing inter-datacenter hops to local network communication can save ~70ms per request.