Integrating OpenAI Realtime Voice into Compound Engineering

The Core Concept: Live Voice-Driven Polish

The author is evolving the 'Compound Engineering' plugin by introducing a 'live mode' to the existing polish command. The goal is to bridge the gap between human design intuition and machine execution. By leveraging OpenAI's Realtime Voice API, the developer can interact with a live website, provide verbal feedback (e.g., 'move this element,' 'change this color'), and have an agent implement those changes in real-time. This shifts the workflow from a batch-process (record, annotate, send, wait) to a continuous, conversational loop.

Architecture and Implementation

The implementation relies on a combination of existing tools and new abstractions:

• Riffre Library: Originally designed for capturing screen and DOM interactions into a zip file, it is being refactored to support a streaming contract. This allows the agent to receive live data rather than waiting for a completed recording.
• Compound Engineering Plugin: This serves as the primary host. The author is integrating the live mode directly into the polish command, allowing users to toggle between 'traditional' and 'live' modes.
• Cursor Projects: The author heavily utilizes Cursor's 'projects' feature, which provides an 'always-on' cloud/local hybrid environment. This allows the agent to maintain context across multiple repositories (Compound Engineering and Riffre) and access various runners/workers seamlessly.
• Visual Feedback: To ensure the agent has sufficient context, the system captures screenshots and DOM metadata periodically during the voice session. The author is also exploring using perfect-freehand (by the author of TLDraw) to implement a lightweight drawing overlay for visual annotations.

The Decision-Making Process

The author emphasizes a 'zoom-out, then zoom-in' approach. By framing the project as a potential year-long endeavor, they identify the necessary abstractions—specifically, creating a library (Riffre) that is framework-agnostic and decoupled from the main engineering plugin. This separation of concerns ensures that the streaming capability can eventually be used for non-coding knowledge work ('Compound Work') as well as UI polish.

Managing Agent Autonomy

A significant tension exists regarding how much the agent should do autonomously. The author proposes three modes: 'Instant' (agent acts immediately), 'Smart' (agent batches changes), and 'Never' (manual control). The preference is for a hybrid approach: the agent handles small, low-risk UI tweaks instantly, while larger refactors or complex changes are batched for review to prevent conflicting state changes.