Optimizing MoE Inference with FreeToken

The Breakthrough

FreeToken improves inference speed for Mixture-of-Experts (MoE) models that exceed GPU VRAM by dynamically caching frequently used experts on the GPU and streaming missing experts over PCIe, rather than permanently offloading specific layers to the CPU.

What Actually Worked

• Dynamic Expert Caching: The engine treats GPU memory as a cache for the most frequently accessed experts, keeping the full model in system RAM as the source of truth.
• Double Buffering: During the prefill phase, the engine streams the next layer's experts over PCIe while the GPU computes the current layer, effectively hiding transfer latency.
• Q Policy*: When a cache miss occurs, the engine profiles the machine's specific PCIe and RAM bandwidths to decide whether to fetch the expert to the GPU or compute it directly on the CPU, choosing the faster path for that specific hardware.
• FTW Weight Format: The engine uses a custom weight format that allows models to load directly from disk into the required memory layout, bypassing the repackaging overhead common in other inference tools.

Before / After

• Task: Running Qwen 3.6 (35B parameter, 8-bit quantization) on an RTX 5090 (32GB VRAM).
• Ollama: 14 minutes 20 seconds total time; 58 tokens per second.
• FreeToken: 4 minutes 40 seconds total time; 132 tokens per second.

Context

Standard inference tools like Ollama often handle VRAM overflow by pinning specific layers to the CPU. Because MoE models activate different experts per token, this static approach forces every token to traverse the slow CPU, creating a bottleneck. FreeToken addresses this by treating the GPU as a cache for the most active experts. However, the tool is only advantageous when the model exceeds available VRAM. If a model fits entirely within the GPU, Ollama remains faster due to the lack of streaming overhead.

Content References

• tool: FreeToken, https://github.com/FlashML-org/FreeToken, mentioned