A Linguistic Framework for Diagnosing Voice AI Failures

The Linguistic Failure Map

Voice AI failures are rarely isolated bugs but rather structural breakdowns in communication. Midam Kim proposes a grid-based framework to diagnose these issues, mapping both the listening and speaking channels across four interdependent levels: sounds, words, interaction, and mental models. When an agent fails, it is usually because these layers are not aligned or because the agent fails to track the user's mental model over the timeline of the call.

The Problem of Vanishing Context

Unlike chat interfaces where history persists as text, voice communication is ephemeral. Sounds, words, and interaction cues vanish the moment they are spoken. The only element that accumulates throughout a call is the user's mental model. If an agent does not dynamically track this model, it will inevitably fail to provide relevant information, leading to user frustration, task abandonment, and unnecessary escalations to human agents.

Diagnosing and Orchestrating Agents

To improve voice agent performance, developers must move beyond simple ASR and TTS tuning. Effective orchestration requires:

• Dynamic Context Retention: Maintaining state across the entire call timeline rather than treating each turn as an independent event.
• Interactive Clarification: Implementing strategies to handle ambiguity, such as asking for clarification rather than simply repeating the same failed prompt.
• Turn-Taking Optimization: Improving turn detection and latency management to prevent the agent from cutting off the user or talking over them.
• Emotional Awareness: Incorporating emotion detection to adjust the agent's tone and strategy when a user becomes irritated.

ServiceNow developed a benchmark tool called EVA Bench to help developers diagnose these specific linguistic failures within their own systems.