Voice Agents Should Use UI Actions Instead of Just Speech

Rethinking Voice Interaction Modalities

Developers often default to a speech-to-speech loop, but voice agents can and should utilize three distinct modalities: speech-to-speech, speech-to-action, and event-to-speech. The most neglected of these is speech-to-action, where the model acts as a controller for software rather than a conversational partner. By mapping existing API endpoints and application handlers to tool calls, developers can allow users to drive complex software—such as creative tools or government forms—entirely through voice commands.

Moving Beyond Verbal Responses

Software has spent decades refining non-verbal communication, including notifications, field highlighting, button state changes, and cursor movement. Voice agents should leverage these established UI patterns to provide feedback. When a model performs a task, it does not need to describe the action in audio; it can simply update the interface to reflect the new state. This approach reduces cognitive load and avoids the latency and awkwardness of long-winded verbal confirmations.

Leveraging Native Audio Models

Traditional voice systems rely on a chained approach—transcribing audio to text, processing via an LLM, and synthesizing speech—which introduces significant latency and loses critical information like tone, cadence, and emotional intent. Modern models like GPT-4o use native audio tokens, allowing for continuous streaming and better reasoning. Developers can further improve these interactions by using preambles to inform users when the model is performing background tasks, such as parallel tool calls or complex reasoning, effectively managing user expectations during high-latency operations.