Deep Dive on LLM Inference at Scale

The Core Constraints of Inference

LLM inference is fundamentally limited by three pain points: memory consumption that scales linearly with context length, high Time to First Token (TTFT) during the prefill phase, and collapsing throughput when serving multiple concurrent users. The primary culprit is the KV cache, which grows as the model processes more tokens. For a model like Mistral 7B, each token requires approximately 131 KB of KV cache. At 16,000 tokens of context across 80 concurrent users, the cache alone consumes 42 GB of VRAM, exceeding the capacity of standard 24 GB consumer GPUs.

The Inference Pipeline: Prefill vs. Decode

Inference operates in two distinct phases. The 'prefill' phase processes input tokens to generate initial KV vectors; this is compute-bound and determines the TTFT. The 'decode' phase generates tokens one by one, which is memory-bound because the GPU must repeatedly fetch the entire KV cache from high-bandwidth memory (HBM) to the shared memory for every new token. Because the decode phase is limited by memory bandwidth rather than compute, increasing batch sizes or context lengths directly degrades inter-token latency.

Optimization Strategies

Optimizations are split between model-level and serving-level improvements. Model-level techniques include quantization (reducing precision), Grouped Query Attention (GQA) to reduce KV cache size, and FlashAttention for efficient tiling. Serving-level optimizations are critical for production: Paged Attention (borrowed from OS memory paging) eliminates memory fragmentation by storing KV caches in non-contiguous blocks. Continuous batching allows the server to inject new requests into the batch as soon as others finish, rather than waiting for the entire batch to complete, significantly increasing hardware utilization.

Engine Selection and Trade-offs

Choosing an inference engine involves navigating a 'trade-off triangle' between quality (context length), latency (TTFT/inter-token), and throughput (concurrent users). While vLLM and SGLang perform similarly on standard chat workloads, SGLang often outperforms vLLM in agentic workflows due to its superior handling of branching logic and complex request patterns. Engineers should benchmark their specific workload rather than relying on generic throughput numbers.