Running 744B Parameter Models on Consumer Hardware with Colibrì

The Breakthrough

Colibrì, a pure-C inference engine, enables the execution of massive Mixture of Experts (MoE) models like the 744B parameter GLM-5.2 on consumer hardware by treating local storage as a cache layer and only loading active expert sub-networks into memory.

What Actually Worked

• Expert Streaming: The engine keeps dense model components (roughly 17B parameters) resident in RAM while streaming specific expert sub-networks from the SSD only when required for a given token.
• Learning Cache: Colibrì records expert routing patterns during inference to pin frequently used experts into spare RAM, reducing disk I/O latency over time.
• Speculative Decoding: The engine utilizes a multi-token prediction head to generate multiple tokens per pass, improving throughput compared to standard autoregressive generation.
• Memory Optimization: It employs Multi-Head Latent Attention (MLA) to compress the KV cache by approximately 57 times, significantly lowering the per-token memory footprint.

Before / After

• MacBook M2 Max (External SSD): First-token latency of 158 seconds, with over 80% of time spent waiting on disk I/O.
• Workstation (Internal NVMe, 64GB RAM): First-token latency of 17 seconds, with disk wait time reduced to 48% of the total execution time.

Context

Running frontier-class models typically requires massive VRAM capacity to hold all weights simultaneously. Colibrì exploits the sparse nature of MoE architectures, where only about 5% of parameters (40B out of 744B) are active per token. The experiment demonstrates that while the model can run on consumer hardware, performance is strictly gated by the speed of the storage medium and the amount of available system RAM, rather than the raw compute power of the GPU.