TTS Adopts LLM-Style Autoregressive Frame Generation

The Breakthrough

Samuel Humeau explains that dominant TTS architectures now use autoregressive transformers to generate sequences of compressed audio frames, one 80ms frame at a time, via neural audio codecs that reduce 200kbps raw audio to about 500 tokens per second.

What Actually Worked

• Neural audio codecs process audio in 80ms frames (12 frames/sec) and encode each into 37 tokens, compressing from 200kbps to a few thousand bps; training uses reconstruction losses, adversarial losses, and guidance to retain acoustic features and text information while dropping irrelevancies.
• Backbone employs a 4B-parameter autoregressive transformer with one step per frame; Mistral's model generates the full 37 tokens per frame at once using a diffusion/flow matching decoder, differing from autoregressive recomputation of all tokens per step.
• Conditioning prepends a few seconds of reference audio for voice cloning, followed by the full text prompt; model supports multilingual inference, preserving accents (e.g., French voice speaking English retains strong French accent).
• Streaming outputs first audio packet immediately for playback, reducing perceived latency in agents; single-GPU inference yields 17ms time-to-first-audio (excluding network).
• Voice agent demo integrates STT, fast LLM for text, and TTS; handles conference queries with cloned voice (e.g., 'Paul'), streaming audio packets before full generation completes.

Context

Audio's high information density (200kbps for MP3-quality) prevents direct transformer input, unlike text (~15 bits/sec even for fast speech). Labs converged on codecs to tokenize frames densely, enabling LLM-style autoregression for sequential generation. This supports responsive voice agents where LLM streams text to TTS, but streaming text input remains unsolved—options include interleaving text/audio tokens or dual-stream blending to start voicing after the first LLM token, avoiding wait for full text.

Notable Quotes

• "one token of a vocabulary of a thousand it's 10 bits of information and the audio requires much much more uh like a much larger bit rates for example a standard quality mp3 that's 200 kilobits per second"
• "we cut the audio as uh with pieces of 80 milliseconds so 12 frame per second and we transform each frame into several tokens like 37 in our case so we reduce the problem to about 500 tokens per second"
• "if you remove the network and with a single gpu you have 17 milliseconds between the moment where you input your text and the moment where you have the first audio you can play"

Content References

Mistral's TTS technical report is referenced for full details on their flow matching decoder and training.