Running 26B Parameter Models on 2GB of RAM

Architecture and Memory Efficiency

Turbo Fieldfare optimizes the execution of Gemma 4, a 26 billion parameter Mixture of Experts (MoE) model, by keeping only the essential components in RAM. While the full model requires approximately 14GB, this approach maintains a resident memory footprint of only 1.35GB. The system splits the model into two parts: a static set of weights (attention, router, embeddings, and one shared expert) that remains in memory, and the remaining 128 experts per layer (totaling 12.9GB) which are stored on the SSD and fetched on demand.

Leveraging Unified Memory

Performance relies on Apple Silicon's unified memory architecture, which eliminates the need to copy data across a PCI bus between system RAM and discrete VRAM. By using a custom file format that matches the memory layout required by the Metal GPU kernel, the system avoids costly data conversion steps. The CPU reads weights directly from the SSD into a buffer accessible by the GPU. To hide disk latency, the system executes the shared expert computation concurrently with the disk fetch for the required experts. Additionally, an LFU (Least Frequently Used) cache keeps 16 experts per layer in memory to improve hit rates for frequently accessed weights, relying on the predictable nature of the MoE router.