Dwarfstar: Running Near-Frontier Models via SSD Streaming

The Breakthrough

Salvatore Sanfilippo created Dwarfstar (ds4), a specialized inference engine written in C that enables running near-frontier models like DeepSeek-V4 on consumer hardware by treating the SSD as a primary storage layer for model weights, effectively decoupling model size from available RAM.

Technical Implementation

• SSD-Backed Inference: The engine streams routed expert weights from the SSD on demand, allowing the system to run models that exceed the host machine's physical memory capacity.
• Asymmetric Quantization: The engine uses an aggressive 2-bit quantization specifically for expert weights while maintaining higher precision for attention and shared layers to preserve model quality.
• Distributed Execution: The system supports splitting a single model across multiple machines (e.g., two MacBooks connected via Thunderbolt) by partitioning layers, which parallelizes the prefill phase across devices.
• API Compatibility: Dwarfstar includes a built-in server that implements OpenAI and Anthropic API specifications, allowing developers to use existing tools like Claude Code or Codex against a local, private instance.
• Native Performance: The core is implemented in pure C with handwritten Metal and CUDA kernels, avoiding dependencies on external libraries like GGML while maintaining compatibility with standard GGUF formats.

Context

Existing local inference tools like llama.cpp prioritize broad model support and ease of use. Dwarfstar takes the opposite approach by targeting a single model architecture and optimizing it heavily for specific hardware. This project demonstrates that high-performance, private, local inference is viable on current-generation hardware if the software stack is tuned to treat storage and memory as a unified hierarchy rather than relying solely on RAM.