Running 284B Parameter Models on Consumer Hardware

Selective Quantization and Architecture

Dwarf Star (DS-4) optimizes the DeepSeek V4 Flash model for local execution by exploiting its Mixture of Experts (MoE) architecture. Instead of applying uniform quantization, DS-4 preserves load-bearing components—attention layers, routers, and shared experts—at 4-bit precision to prevent error propagation. The routed experts, which constitute the majority of the model's parameters but are only sparsely activated, are quantized to 2 bits. This selective approach reduces the model footprint from 568 GB to approximately 81 GB, allowing it to fit within 128 GB of unified memory.

SSD Streaming and Memory Management

To accommodate hardware with less than 81 GB of available RAM, DS-4 implements SSD streaming. The engine keeps load-bearing weights and a subset of frequently accessed experts in a pinned RAM cache. Remaining experts reside on the SSD and are swapped into memory on demand. This transforms RAM from a hard capacity limit into a performance dial, where lower RAM results in a lower cache hit rate and slower inference speeds rather than a complete failure to run. The system also supports distributed inference by splitting model layers across multiple machines connected via Thunderbolt 5, enabling faster pre-fill performance.

Calibration and Validation

The quantization process is calibrated using a dataset of 4,700 prompts, including code reviews and agentic tool calls, to identify which weight columns are critical for performance. By measuring the negative log likelihood drift against the official DeepSeek model, the developers confirmed that the 2-bit quantized version maintains high alignment with the original model's outputs. The system also treats the KV cache as a first-class citizen, allowing long-context sessions to be saved to disk and resumed instantly without reprocessing.