Understanding LLM Prompt Caching Economics & KV Cache Architecture
In traditional Large Language Model (LLM) serving, every user query triggers a resource-intensive "prefill phase". The serving engine (such as vLLM, TensorRT-LLM, or proprietary provider runtimes) must perform matrix multiplications across every attention layer to compute the Key and Value (KV) activation tensors for every token in the prompt. For long-context applications—such as Retrieval-Augmented Generation (RAG), autonomous coding agents, and complex JSON schema definitions—the prefill phase dominates both GPU utilization and API bills.
How Prompt Caching Works Under the Hood
Prompt caching allows the inference cluster to store the computed KV cache tensors in high-speed GPU High Bandwidth Memory (HBM) or host RAM across consecutive requests. When an incoming request shares an identical prefix with a cached prompt, the serving engine skips the compute-heavy prefill phase entirely and begins autoregressive token generation immediately.
Frontier Provider Pricing Models Explained
1. Anthropic Claude (Explicit Cache TTL)
Anthropic implements a deliberate caching model with a 5-minute Time-to-Live (TTL). Writing to the cache costs 1.25x the base input price ($3.75/MTok on Sonnet). Subsequent requests hitting the cache receive a massive 90% discount ($0.30/MTok). The TTL resets to 5 minutes on every cache hit.
2. OpenAI GPT-4o (Automatic Prefix Caching)
OpenAI applies automatic prompt caching at no extra write charge. Prompts exceeding 1,024 tokens that match an existing prefix automatically receive a flat 50% discount ($1.25/MTok on GPT-4o).
3. DeepSeek V3 & R1 (Native Multi-Head Latent Attention)
Thanks to its Multi-head Latent Attention (MLA) architecture, DeepSeek compresses the KV cache footprint by up to 93%, enabling industry-leading cache pricing. DeepSeek-V3 charges only $0.014/MTok on cache hits (a 90% discount with zero write surcharge) with a minimum prefix threshold of just 64 tokens.
The Break-Even Hit Rate Formula
When evaluating providers with cache write surcharges (such as Anthropic's 1.25x write multiplier), engineers must verify that their application's cache hit rate exceeds the break-even threshold:
For Claude 3.5 Sonnet, \( (3.75 - 3.00) / (3.75 - 0.30) = 0.75 / 3.45 pprox 21.7\% \). Any agentic workflow with a cache hit rate above 22% yields compounding cost savings.