Voice AI's 'Her' gaps: duplex, paralinguistics, cost

Latency and Architecture Limits

Cascaded systems process speech-to-text, then LLM, then text-to-speech, with streaming variants still exceeding 200ms for TTS alone while humans complete full understanding-to-response in 200ms. Tool calls add 500ms to 4s latency, dominating delays; one mitigation inserts fillers from LLM to sustain conversation during waits, as in a demo travel agent saying "Tokyo is such an incredible choice" while querying hotels. Speech-to-speech models cut pipeline latency but remain half-duplex, blocking overlap like backchanneling (up to 20% of conversation time in Japanese). Full-duplex models like Moshi handle interruptions, coughs, and overlaps robustly, as shown in a demo where it responds mid-sentence to "How long is it going to take us to get there" with "It's approximately 5 months" before finishing the query.

Paralinguistic Understanding

Voice conveys tone, hesitation, and discomfort lost in transcription; Her's AI infers "I take it from your tone that you're challenging me" from prosody. Speech-to-speech retains this data, but models trained on text-audio QA pairs ignore it. Moshi demonstrates flow but remains "very stupid" without tool calls, observability, or paralinguistic exploitation for relevant responses.

Scalability and Cost Barriers

Voice APIs run at a loss for hyperscalers; TTS dominates bills, burning startup funds before user growth. Privacy demands local processing. Gradium Phoneon delivers on-device TTS on smartphone CPU (<1B parameters), outperforming Coconet (lacks cloning), enabling consumer apps without API fees, as in a demo: "Gradium Phoneon runs locally on the CPU which means high fidelity neural speech right here without those intergalactic cloud government hacks."