Running 744B Parameter Models on Consumer Hardware via Colibri

Memory-Mapped Expert Loading

Colibri facilitates running 744B parameter models on consumer hardware by treating the NVMe SSD as an extension of system RAM. Because the Mixture-of-Experts architecture in GLM 5.2 only activates a small subset of parameters per token, the engine avoids loading the entire 370 GB model into memory. Instead, it maintains a small "hot" set of dense layers (attention, embeddings, shared experts) in 9.9 GB of RAM and fetches the required routed experts from disk on demand.

Performance Optimization Techniques

• 4-bit Quantization: The model weights are compressed to 4-bit to reduce the footprint to 370 GB, while the multi-token prediction head is kept in 8-bit precision to prevent performance collapse.
• Expert Caching: The engine tracks frequently accessed experts and pins them to available RAM to minimize redundant disk reads over time.
• Multi-Token Prediction: By utilizing the model's native multi-token prediction head, the system generates 2.2 to 2.8 tokens per forward pass, partially offsetting the latency of disk-based expert fetching.
• KV Cache Compression: The model uses Multi-Latent Attention to compress the KV cache to 576 numbers per token, enabling million-token context windows to fit within 25 GB of RAM.

Implementation Constraints

Colibri is a zero-dependency C implementation that operates without a Python runtime or CUDA requirements. While it allows execution of frontier-level models on consumer hardware, it is currently a proof-of-concept for asynchronous tasks rather than real-time chat, as fetching 11 GB of expert data from disk per token remains a significant latency bottleneck. Users require at least 25 GB of RAM and a high-speed NVMe SSD to achieve functional performance.