Engineering a Voice-First AI Companion
AI Engineergo watch the original →
the gist
Tolan maintains a voice-first AI companion by prioritizing sub-two-second latency, routing turns to models based on emotional stakes, and rebuilding context per turn to handle conversational volatility.
Optimizing for Conversational Volatility
Voice interaction requires fundamentally different assumptions than text-based LLM apps. While text chat assumes stable context and slow turns, voice users frequently interrupt, change subjects mid-sentence, and talk while multitasking. The Tolan team discovered that a latency drift of just 500 milliseconds significantly degraded user retention. To mitigate this, they implemented smart turn-taking that analyzes speech patterns to distinguish between genuine interruptions and casual filler, reducing early cutoffs by over 50% at the cost of 60 milliseconds of latency.
Pipeline and Context Management
To maintain performance, the team treats latency as a product feature rather than a background metric. They measure every stage of the pipeline independently: end-of-utterance detection, transcription, model generation, and text-to-speech playback. They employ a tiered model fleet, using a 'tone router' classifier to assign turns to either a frontier model or a smaller, faster model based on emotional stakes. High-stakes interactions, such as onboarding or crisis support, always receive the frontier model, while casual chatter is offloaded to smaller models to optimize unit economics.
Memory is handled as a retrieval system rather than a static transcript. The system stores embedded facts and preferences in a vector database, performing nightly compression to merge duplicates and resolve contradictions. Crucially, the team avoids reusing context across turns to prevent 'drift' where the model relies on stale information. Instead, they reassemble the context window for every turn using a combination of recent message summaries, persona cards, retrieved memories, and real-time app state.
Agent-Driven Development
Internal development relies heavily on a fleet of specialized agents. These agents handle implementation, code review, and bug triage by integrating with tools like Sentry, DataDog, and Linear via MCP. The team found that treating the codebase as the primary documentation source allowed agents to perform effectively. By running multiple agents in parallel—where one generates code and another enforces standards—they improved their crash-free rate from 99.6% to 99.9% and reduced runtime errors by over 50%.