Building Real-Time Photorealistic Avatars with World Models

Real-Time Video Generation Strategy

LemonSlice approaches avatar generation by treating video models as world models focused on human interaction. To enable real-time performance, the team enforces a causal attention mask during training, which prevents the model from looking into the future. This constraint is critical because, in a live interaction, future frames do not exist. To meet latency requirements, they collapse standard multi-step denoising processes (typically 30 steps) into a single-step generation pass.

Solving Error Accumulation and Hardware Orchestration

Generating video for hours without reset introduces significant error accumulation, where each generated frame inherits and compounds artifacts from previous ones. LemonSlice utilizes a proprietary method to mitigate this drift, allowing for continuous operation over 8 to 16 hours. Beyond the model architecture, the team emphasizes the importance of the "model harness"—the orchestration layer managing GPU and CPU threads. This system ensures that data queues and buffers remain synchronized to prevent stuttering, which the team identifies as a primary source of durable product value.

The Role of Audio and Future EQ Layers

Expressive facial movement is driven by audio embeddings. Because standard audio encoders are trained on monotone audiobooks, LemonSlice trains custom encoders to capture the nuance and emotion required for photorealistic avatars. The company is currently developing an "emotion engine" that maps audio and text inputs to specific emotional states and physical actions. Their long-term roadmap involves moving toward a single, end-to-end EQ (Emotional Quotient) model that processes user video and audio directly, while delegating high-level reasoning and tool calling to a separate IQ (Intelligence Quotient) model.