NVIDIA Nemotron 3.5 ASR Streaming Model Overview

Cache-Aware Streaming Architecture

NVIDIA Nemotron 3.5 ASR utilizes a cache-aware streaming mechanism to eliminate the redundant compute costs associated with traditional overlapping-window transcription. Instead of re-encoding overlapping audio chunks, the model caches the encoder's self-attention activations and reuses these states for subsequent frames. This approach functions similarly to KV-caching in LLM decoding, allowing for significant latency reduction. Users can configure the attention context size at runtime to trade off between latency and accuracy, with supported chunk sizes ranging from 80 milliseconds to over 1 second.

Word Boosting for Domain Accuracy

To address transcription errors on specialized terms like product names or surnames, the model supports word boosting at decode time. This technique does not require fine-tuning or weight adjustments. Instead, it uses a boosting tree to inject specific phrases and their associated weights into the decoding process. When the model generates tokens, it applies a positive bias to the scores of the boosted phrases, increasing the likelihood of their selection. This allows developers to maintain high accuracy for custom vocabulary without the overhead of retraining the model.

Multilingual Support and Diarization

The model supports 40 languages from a single checkpoint, with 19 languages categorized as out-of-the-box ready and 13 considered production-level. For the remaining 8 languages, the model provides a base that can be adapted via fine-tuning. Additionally, the model integrates with the NeMo framework to support speaker diarization. While real-time diarization remains challenging, the system can capture speaker embeddings to perform speaker-level attribution in batch or podcast-style processing workflows.