Engineering Infrastructure for Open-Weight Multimodal Models

Multimodal Training and Architecture

MiniMax trained the M3 model from scratch by simultaneously training on text and image data. This joint training prevents the common issue of modality collapse, where models lose performance in one domain after being trained on another. By training both modalities together, the model develops attention maps where text and visual tokens naturally align, allowing for better performance in tasks like website generation and visual reasoning. The model utilizes sparse attention mechanisms, which require specialized kernel development to maintain performance at scale.

Inference and Agentic Workloads

Serving models with 1 million token context lengths and agentic capabilities requires a shift in the inference stack. Unlike traditional chat workloads, agentic tasks involve uploading entire codebases and performing hundreds of multi-turn tool calls. This necessitates aggressive optimization of the KV cache, which the team treats similarly to a distributed file system. Inference providers must continuously tune attention kernels and quantization strategies post-launch to reduce latency, often iterating on performance daily.

Reinforcement Learning for Long-Horizon Tasks

To enable models to perform complex, long-horizon tasks like replicating 12-hour research runs or optimizing GPU kernels, the team focuses on environment design and reward formulation. They use a self-evolution approach where the model builds its own training harnesses. Evaluation for these tasks involves intermediate submission checks and validation-test splits to ensure the model is genuinely solving the problem rather than hacking the environment. The team uses these models to accelerate their own internal development, creating a feedback loop that improves both the model and the infrastructure.