MiniMax M3 Architecture and Inference Optimizations

Architectural Shift to Sparse Attention

MiniMax M3 moves away from the full attention mechanisms used in previous generations, adopting a sparse attention approach to address the compute and memory bottlenecks inherent in long-context inference. By limiting the attention mechanism to focus only on top-K relevant tokens rather than every token in the sequence, the model reduces the computational load required for attention scores. This is paired with Grouped Query Attention (GQA) to minimize the KV cache footprint, allowing multiple queries to share cache data and reducing the volume of data transferred between HBM and SRAM.

Tiled IO and Memory Efficiency

To mitigate the bandwidth bottleneck between HBM and SRAM, MiniMax implemented a tiled processing strategy. Instead of reading scattered tokens, the system groups tokens into tiles (e.g., 100 tokens per tile) and processes them as an outer loop. This allows the hardware to read a continuous chunk of the KV cache once, process all relevant queries against that tile, and significantly reduce redundant memory access. These optimizations collectively address the physical limitations of GPU memory hierarchies, where compute capacity often outstrips the speed at which data can be moved to the processing units.

Performance Gains

According to the MiniMax M3 report, these architectural changes result in a per-token compute requirement that is 1/20th of the previous M2 generation when handling 1 million tokens of context. The model achieves a 9.7x speedup in the pre-fill stage and a 15.6x speedup in the decoding stage, while reportedly maintaining performance parity with full attention models across most benchmarks.