Optimizing Realtime Voice Agents with Cascaded Stacks

The Breakthrough

EliseAI achieves low-latency voice interaction with high-intelligence models by decoupling the voice stack into perception, planning, and control layers, using speculative execution and audio prefix caching to hide the latency of slow, accurate LLMs.

What Actually Worked

• Speculative Transcription: The system runs a fast, streaming transcriber (Flux) alongside a slower, high-context transcriber (Scribe v2). If the slower engine provides a correction, the system discards the previous output and replaces it, provided new audio has not yet arrived.
• Background Tool Calling: Instead of forcing the main LLM to wait for tool results, a background agent executes tool calls independently. The results are injected into the main model's context, allowing the model to generate responses as if it had performed the tool call itself.
• Prefix Audio Caching: The system maintains a cache of previously generated audio for common conversational openers. When a user speaks, the agent begins playing cached audio immediately while the LLM generates the remainder of the response.
• Seamless Audio Stitching: The system streams the full response to a text-to-speech engine (Cartesia) while simultaneously playing the cached prefix. Once the cached audio finishes, the system suppresses the redundant portion of the new audio stream and seamlessly transitions to the live output to avoid audible seams.

Context

Bohan Li of EliseAI maps the voice agent architecture to a self-driving car stack, where transcription acts as perception, the LLM acts as planning, and speech synthesis acts as control. By optimizing each layer independently, the team minimizes the round-trip latency inherent in using frontier-level models for real-time voice applications.