Modern Inference Engineering: KV Compaction and Speculative Drafting

Training-Driven Inference Optimizations

Modern inference engineering is increasingly defined by dedicated training processes that blur the line between model training and deployment. While local inference focuses on fitting models into constrained memory via aggressive quantization, data center inference prioritizes throughput and latency by optimizing the KV cache and speculative decoding pipelines.

KV Cache Compaction

Rather than relying on raw quantization like TurboQuant, which offers memory savings but incurs a heavy decode-time compute penalty, production systems are adopting learned KV cache compaction. The "Still" method, developed by the Baseten research team, uses a learned bottleneck to cross-attend fixed query vectors against the full KV cache. This process emits compact keys and values in a single forward pass, providing a differentiable, compressed memory representation that the LLM attends to as if it were standard context.

Advancements in Speculative Decoding

Speculative decoding has evolved from using small draft models to diffusion-based architectures. The "DFlash" method treats drafting as a diffusion process, allowing the model to generate 8 to 16 tokens simultaneously rather than one at a time. This approach yields a 3x improvement in token acceptance rates compared to previous methods like Eagle 3. Furthermore, continuous retraining of these speculators on live production prompts can lift acceptance rates by 20% to 100%, provided the system can handle the associated storage and compute overhead. Emerging research like "DSpark" aims to further improve these rates by pairing diffusion models with sequential drafters.