Text Diffusion: Low-Latency Generative AI

The Shift from Autoregressive to Diffusion

Standard LLMs (GPT-4, Gemini) operate autoregressively, generating one token at a time. This creates a bottleneck where the model cannot see future tokens, preventing self-correction during the generation process. Text diffusion models, by contrast, initialize a full sequence of tokens as noise and iteratively refine that canvas. This allows the model to attend to the entire sequence bidirectionally, enabling it to detect and fix reasoning errors mid-generation.

Hardware Efficiency and Latency

Autoregressive models are memory-bound; for every single token generated, the entire model architecture and KV cache must be streamed from HBM (High Bandwidth Memory) to the tensor cores. This is inefficient. Diffusion models are compute-bound rather than memory-bound. By generating blocks of tokens over a fixed number of denoising steps (e.g., 24 steps for 256 tokens), the model performs significantly fewer memory transfers. This architectural change results in raw generation speeds reaching upwards of 2,000 tokens per second, which is sufficient to power real-time, interactive applications.

Adaptive Computation and Self-Correction

Because diffusion models are iterative, they exhibit "adaptive computation." The model can be trained to determine when it has reached a sufficient level of confidence, allowing it to spend more compute on complex tasks (like quantum physics explanations) and less on trivial ones (like reciting digits of pi). Furthermore, the bidirectional nature allows for "in-place editing." Unlike autoregressive models that must regenerate text from the point of error, a diffusion model can see the entire context and modify specific tokens within a block to fix bugs or update content without a full rewrite.

Tradeoffs and Challenges

Despite the latency advantages, text diffusion faces a major hurdle: throughput. Autoregressive models excel at batching, allowing them to serve many users simultaneously on a single GPU. Diffusion models, requiring multiple forward passes for a single request, hit compute thresholds much faster, making them more expensive to serve at scale. Consequently, they are currently relegated to research previews rather than production-grade, high-traffic APIs.

Notable Quotes

• "Bidirectional attention means it can see future tokens and go back to fix mistakes. Autoregressive models cannot do that."
• "It's not just the same thing faster. It can really unlock some new applications... like an operating system where every click generates the next screen."
• "The model naturally gets to decide... when it is finished and return the response, and typically we see that harder evals take more time."