Optimizing Inference Clouds for Agentic Workflows

The Shift from Chat to Task-Based Inference

Agentic workflows differ fundamentally from traditional chat applications. While chat focuses on the latency of a single request-response cycle, agents operate in loops of planning, acting, and observing, where the context window grows continuously as new observations are appended. Because consecutive agent steps share massive prefixes, recomputing these prefixes for every call wastes significant compute. The primary metric for success is end-to-end task completion time, not the latency of an individual LLM call.

Engineering Pillars for Agentic Clouds

To optimize for task-level latency, inference stacks must move beyond naive load balancing and request-based scheduling. The following four pillars are essential for scaling agentic inference:

• Prefix Caching: By computing the key-value (KV) cache for shared prefixes once and reusing it across subsequent steps, the system avoids redundant prefill compute and significantly improves time-to-first-token.
• Hierarchical KV Cache Management: To handle growing contexts, memory management must be frugal, utilizing KV-cache quantization to reduce footprints and implementing hierarchical caching that spills data across GPU memory, host memory, and disk.
• Cache-Aware Routing: Rather than spreading requests evenly across a cluster, a cache-aware router directs requests to the specific replica already holding the relevant prefix in its cache, turning expensive prefill operations into simple cache hits.
• Agent-Aware Scheduling: The scheduler must recognize that individual LLM calls are part of a larger, ongoing program. This context allows for smarter speculative prefilling, preemptive work management, and informed cache eviction decisions.

Performance Impact

In production testing, these optimizations yield substantial gains. A split test conducted by the coding agent provider Cursor (Kilo) demonstrated that using FriendliAI to serve open-weight models was seven times faster than standard third-party inference providers, while simultaneously maintaining a lower error rate. This approach allows developers to leverage frontier-quality open-weight models at a fraction of the cost of closed-source alternatives.