oMLX Persists KV Cache to SSD for 30x Faster Local LLMs on Mac

The Breakthrough

Jun Kim built oMLX, an Apple Silicon LLM server that persists the full KV cache to SSD as safetensors. The cache survives server restarts. When the conversation prefix returns, oMLX restores it from disk instead of recomputing.

What Actually Worked

• Hot tier uses block-based KV cache in RAM with prefix sharing across concurrent requests via copy-on-write, the same technique vLLM uses on CUDA.
• Cold tier serializes blocks to SSD as safetensors when the hot cache fills; restoration takes 1-2 seconds.
• v0.3.8 adds async SSD cache writes, dropping per-request overhead from 3.5 seconds to 400ms.
• FastAPI server with EnginePool routes requests to LLM, VLM, embedding, or reranker engines; scheduler runs FCFS across mlx-lm's BatchGenerator for continuous batching.
• Drop-in compatibility for OpenAI chat completions and Anthropic /v1/messages endpoints, including streaming, tool calls, and reasoning blocks.
• Install via brew tap jundot/omlx && brew install omlx for daemon or DMG for native PyObjC menu bar app; points to localhost:8000.

Before / After

Prefill time drops from 30-90 seconds per turn to 1-3 seconds on cache hit. Per-request SSD write overhead falls from 3.5 seconds to 400ms in v0.3.8.

Context

Coding agents like Claude Code, Cursor, and Cline shift conversation prefixes every turn, forcing other MLX servers to discard and recompute the KV cache from scratch. oMLX solves this for local agent loops on Mac by tiering cache across RAM (hot) and SSD (cold). The project forked from vllm-mlx 0.1, added multimodel serving, paged cache for VLMs, and memory guardrails like LRU eviction and process-wide limits (default: system RAM minus 8GB). It reached 13.2k GitHub stars in three months under Apache 2.0.

Notable Quotes

• "When the same prefix comes back the restore takes one or two seconds instead of a minute and a half."
• "The cache survives a full server restart which sounds small but it's the difference between coming back to a conversation tomorrow morning and waiting 90 seconds for prefill or coming back and getting your first token in roughly one."
• "Per request fixed overhead dropped from 3 and 1/2 seconds to 400 milliseconds."