Optimizing LLM Inference Routing at OpenAI

From Feedback Loops to Global Optimization

OpenAI transitioned from a proportional controller that adjusted routing weights based on local engine signals to a centralized control plane architecture. The previous system relied on a feedback loop where engines reported performance metrics, which were smoothed and compared against fleet averages. This caused oscillations, as shifting traffic away from a hot engine cooled it down, only for the controller to immediately route traffic back, which disrupted KV cache locality and made routing decisions difficult to reason about.

The Control Plane and Optimizer

The new architecture decouples routing into a global control plane and a local data plane. The control plane maintains a global view of all CPU clusters and GPU engine clusters, running an optimizer that calculates routing weights to minimize total expected end-to-end latency. This latency calculation accounts for network distance, engine-side queueing, and historical performance profiles (TTFT and TBOT). The data plane, residing in the CPU clusters, pulls these pre-computed weight snapshots asynchronously, allowing it to make routing decisions synchronously without waiting for a central controller.

Handling Capacity and Failures

To prevent overloading, the system enforces hard constraints that ensure all traffic is routed while keeping engines within their effective capacity. The system handles production instability through three primary mechanisms:

• Outlier penalties: The system detects degraded engines and proactively reduces their routing weight to allow for recovery or manual intervention.
• Dynamic retry budgets: To prevent retry storms, the system caps retries based on current utilization, tightening the budget as load increases.
• Load shedding: As a last resort, the system proactively drops a portion of traffic to prevent total system failure and ensure graceful degradation.