Running 35B Mixture-of-Experts Models on iPhone via Flash-MoE

Memory-Efficient Inference via Expert Streaming

To run a 35B parameter Mixture-of-Experts (MoE) model on hardware with limited RAM, the Flash-MoE engine avoids loading the entire model into memory. Instead, it keeps only the core components—embeddings, attention mechanisms, routers, and shared experts—resident in RAM, totaling approximately 1.4GB. The remaining 40 layers, each containing 256 small experts, reside on the device's SSD. During inference, the router identifies the eight required experts per token, which are then read from storage on demand. The system relies on the iOS page cache to manage these reads, as the raw flash storage speed (1.6GB/s) is insufficient to support the required 5GB/s throughput if every read were a cold disk access.

Tiered Quantization and Performance

To further reduce the memory footprint and improve cache hit rates, the model employs tiered quantization. Approximately 25% of the experts, identified as the most frequently used, are quantized to 4-bit, while the remaining cold experts are compressed to 2-bit. This reduces the total model size from roughly 19GB to 13GB, allowing a larger portion of the model to reside in the OS-managed page cache. This configuration achieves 11 tokens per second on an iPhone 17, though the heavy compute load causes significant thermal throttling.

Addressing Inference Loops

A critical bug in the initial iOS port caused the model to enter an infinite token repetition loop. This occurred because the async_pre_read_weight function performed a size check that failed for the smaller 2-bit cold experts, causing them to be silently skipped. By updating the function to validate reads against the specific size of each expert, the model correctly loads all required parameters, resolving the repetition issue.