Optimizing LLM Costs via Prompt Caching

Prompt Engineering2026-05-27go watch the original →

the gist

Prompt caching reduces AI costs and latency by reusing KV cache states. Achieving these savings requires architectural support on the provider side and strict cache-preserving discipline on the developer side.

The Mechanism of Prompt Caching

LLM requests consist of two phases: a compute-bound prefill phase and a memory-bound decode phase. During prefill, the model generates query, key, and value (KV) vectors for every token. Because the KV vectors for a given token are deterministic and independent of future tokens, they can be stored in a KV cache. Reusing these vectors for subsequent requests avoids redundant computation, which the "Don't Break the Cache" paper reports can yield 41% to 80% cost savings. DeepSeek achieves low-cost caching by utilizing Multi-Head Latent Attention (MLA), which reduces the KV cache size by 93%, allowing the state to be stored on distributed disk arrays rather than expensive high-bandwidth memory (HBM).

Cache-Preserving Development Patterns

To maintain cache hits, developers must treat the prompt as a hierarchical structure where stable information resides at the top. Any change to the system prompt or tool definitions invalidates the entire cache. Effective strategies include:

Use messages, not prompt updates: Instead of modifying the system prompt to reflect dynamic data like timestamps or file changes, append these as system reminder tags in the user message or tool result.
Stable tool definitions: Connect all necessary tools and MCP servers at the start of the session. Adding, removing, or updating tool parameters mid-session will bust the cache.
Cache-safe compaction: Rather than summarizing history via a separate API call with a new system prompt, perform compaction using the same system prompt and tools as the parent conversation, appending the summary as a final user message.
Strategic model selection: Switching models mid-session forces a cache rebuild. Select the appropriate model at the start of the task or use sub-agents for specific subtasks to preserve the parent session's cache.
Rewind over reset: Use /rewind to truncate conversation history back to a previous state, which allows the next request to hit the existing cache entry from that point.