Orchestrating Distributed AI Inference at Scale

The Shift to Inference Orchestration

Inference workloads at Meta have outpaced traditional microservices, necessitating a transition from simple model serving to a complex orchestration layer. While early AI infrastructure focused on model kernels and serving frameworks like vLLM, the current bottleneck is the coupling of routing, KV cache management, and GPU scheduling. A request is now a distributed transaction where every hop—from gateway to GPU—introduces potential failure points that cannot be resolved by simple retries, as partial state (like streamed tokens) makes standard RPC recovery impossible.

The Seven-Axis Scheduler

Effective inference scheduling requires visibility across seven distinct dimensions: GPU generation (heterogeneous hardware), HBM headroom, KV cache state, model weight locality (warm vs. cold), tenant priority, latency budgets, and workflow context. Because agentic workflows involve multi-step reasoning, failing at step three of five wastes the compute spent on previous steps. Consequently, scheduling must be workflow-aware, prioritizing the completion of the entire task rather than individual request throughput.

Optimization Framework

All inference optimizations fall into one of four quadrants:

• Avoid: Reduce work via prefix, response, or semantic caching.
• Share: Increase throughput via continuous batching, chunked prefill, or speculative decoding.
• Move: Route requests to smaller models or cheaper regions based on latency and quality requirements.
• Delay: Manage load via admission control and deadline-aware queuing.

Reliability and Control Loops

Reliability must be a property of the control plane rather than the edge. Cascading failures are common in inference due to the KV cache; when a GPU host fails, the system cannot simply reroute to a cold pool without a warmup period, which can cause the remaining healthy nodes to saturate. Effective systems use observability as a control loop input to manage the trade-off triangle of latency, cost, and throughput. The primary KPI for these systems should be cost per successful task, rather than cost per token or cost per request.