Why Current Voice Agents Are Just Walkie-Talkies
AI Engineergo watch the original →
the gist
Most voice agents are half-duplex, forcing artificial turn-taking that breaks natural flow. Achieving full-duplex interaction requires moving from single-stream sequences to multi-stream models or decoupling the natural interface from the reasoning engine.
The Limitation of Half-Duplex Interaction
Most real-time voice models currently operate in a half-duplex mode, meaning the system is either listening or speaking, but never both. This architecture forces an artificial turn-taking structure that prevents human-like back-channeling, such as saying "mm-hmm" or "yeah" while the other party is talking. In human conversation, overlap occurs up to 20% of the time, and current models that force a strict sequence break the conversational flow, making them feel like walkie-talkies rather than conversational partners.
Solving the Audio Tokenization Problem
Raw audio is too dense for standard LLMs, as 24 kHz audio generates 72,000 timesteps for every three seconds of speech. Because transformer attention costs grow quadratically with sequence length, processing raw audio is computationally prohibitive. The solution involves using neural codecs to compress audio into dense, token-like representations. To achieve full-duplex capability, the model must move beyond a single sequence of tokens to a multi-stream architecture where two token streams are modeled simultaneously, allowing both the user and the agent to be active, silent, or overlapping at any given time.
The Tension Between Naturalness and Intelligence
There is a fundamental trade-off between naturalness and reasoning capability. Every unit of model capacity dedicated to processing audio modalities is capacity taken away from reasoning. Consequently, speech-to-speech models often exhibit lower intelligence than cascaded systems that route audio through text-based LLMs. To resolve this, two paths are emerging:
- Scaling: Training larger, frontier-level models that are instruction-tuned to handle audio natively, though this is computationally expensive and slow to iterate.
- Decoupling: Using a small, full-duplex, on-device model as a natural interface that delegates complex reasoning and tool-calling tasks to a background text-based LLM. This approach is more economically viable and allows developers to swap the reasoning back-end without being locked into a specific speech-to-speech model provider.