Reducing LLM Agent Token Costs via Caching and Routing

Optimizing Context and Model Usage

To reduce token consumption, developers should implement prompt caching for system prompts and tool definitions. By setting cache_prompt = default in the agent configuration, the full system prompt is sent only on the initial call, significantly reducing the payload for subsequent iterations. Additionally, managing conversation history via a sliding window approach prevents the entire history from being sent on every turn. When using a sliding window, developers should summarize older messages and inject that summary into the context to maintain continuity without the overhead of full message logs.

Model routing is another effective lever for cost control. Rather than defaulting to frontier models for every task, developers should route requests based on complexity. Simple tasks can be handled by lower-cost models like Claude Haiku, while complex reasoning tasks are routed to more capable models like Claude Sonnet. A lightweight model can even be used as a router to determine the appropriate model for a given input.

Managing Tool Execution and Efficiency

Uncontrolled tool loops are a primary source of wasted tokens. Developers must explicitly set a maximum iteration limit for agent loops to prevent infinite execution cycles. Before deployment, observability tools should be used to audit tool call frequency and execution duration to identify inefficiencies.

Large tool outputs should be offloaded rather than passed directly back into the agent context. Instead of including raw data in every loop iteration, developers should store the results locally or in the cloud and provide a summary to the LLM. This keeps the context window lean and prevents the agent from re-processing massive datasets during every step of the reasoning chain.