Prompt Caching & Cost

Prompt caching reuses the model's already-computed state for a repeated prompt prefix, cutting the price of those input tokens by up to about 90% and time-to-first-token substantially - if the prompt is structured so the prefix actually repeats.

What Gets Cached

Processing the input (prefill) computes key and value tensors for every token at every layer - the KV cache. Because attention is causal, the KV entries for a token depend only on the tokens before it. So if a new request starts with exactly the same tokens as an earlier one, the server can reuse the stored KV entries for that prefix and only compute the new suffix.

flowchart LR
    subgraph R1["Request 1"]
        A1["⚙️ System prompt"] --> B1["🔧 Tools"] --> C1["📚 Document"] --> D1["❓ Question A"]
    end
    subgraph R2["Request 2"]
        A2["⚙️ System prompt"] --> B2["🔧 Tools"] --> C2["📚 Document"] --> D2["❓ Question B"]
    end
    C1 -.->|"identical prefix:<br/>KV reused (cache hit)"| C2
    D2 --> N(["only Question B<br/>is computed"])

    style A2 fill:#dde4dc,stroke:#b0c4b0
    style B2 fill:#dde4dc,stroke:#b0c4b0
    style C2 fill:#dde4dc,stroke:#b0c4b0
    style D2 fill:#e8e0d4,stroke:#c8b89a

Consequences:

• Only prefixes match. One changed token invalidates everything after it. A timestamp at the top of the system prompt defeats the cache for the whole request.
• Order matters. Put stable content first (instructions, tool definitions, long documents, examples), then conversation history, then the new request.
• Serialisation must be byte-stable. Tool definitions in a different order, JSON with unsorted keys, or whitespace differences all change the tokens.
• Outputs are not cached - this is not a response cache. The model still generates fresh output every time.

Self-hosted engines do the same thing automatically: vLLM's automatic prefix caching hashes KV-cache blocks, and SGLang's RadixAttention keeps prefixes in a radix tree (see vLLM & PagedAttention).

How the Major APIs Do It

Details change often - check each provider's caching page before relying on a number. The design rules below don't change.

Break-even for explicit writes

With Anthropic's 5-minute cache, a write costs 1.25× and each read 0.1× the base input price. For a prefix reused n times within the TTL:

uncached cost = n × 1.0
cached cost   = 1.25 + (n − 1) × 0.1
break-even: n = 2  →  2.0 vs 1.35   (caching already wins at the second request)
1-hour TTL (2× write): n = 3 → 3.0 vs 2.2

A read refreshes the TTL, so steady traffic keeps a 5-minute cache warm indefinitely; the 1-hour TTL is for gaps between 5 and 60 minutes.

Designing for Cache Hits

import anthropic

client = anthropic.Anthropic()
SYSTEM = [
    {"type": "text", "text": LONG_STABLE_INSTRUCTIONS},
    {"type": "text", "text": PRODUCT_HANDBOOK,          # large, shared by every request
     "cache_control": {"type": "ephemeral"}},           # breakpoint: cache everything up to here
]

def answer(question: str, history: list[dict]):
    return client.messages.create(
        model=MODEL, max_tokens=2048,
        system=SYSTEM,                                   # identical bytes on every call
        messages=history + [{"role": "user", "content": question}],
    )
# resp.usage.cache_read_input_tokens shows the tokens served from cache

Checklist:

1. Stable first, variable last: instructions → tools → documents → examples → history → current input.
2. No per-request values in the prefix: move dates, user names and request IDs into the final user message.
3. Freeze tool definitions and their order; loading tools dynamically at the top of the prompt breaks the cache on every change.
4. Append, don't rewrite, history in agent loops - editing or summarising early turns invalidates everything after the edit. Compact rarely and deliberately (see Context Engineering).
5. Measure the hit rate from the usage fields; a sudden drop usually means something changed in the prefix.

The Other Cost Levers

A simple cost model per request:

cost = uncached_input × p_in  +  cached_input × p_cache  +  output (incl. reasoning) × p_out

Log the three token counts for every call. Most cost surprises come from output (reasoning) tokens and from cache hit rates that quietly dropped after a prompt change.

Check Yourself

Exercises

Study Notes

Must-know:

• Caching reuses prefix KV state; only exact token prefixes hit; outputs are never cached
• Anthropic: explicit breakpoints, 5 min / 1 h TTL, write 1.25× / 2×, read ~0.1×; OpenAI: automatic, ≥1,024 tokens, up to 90% off; Gemini: implicit on 2.5+, plus explicit caches
• Stable content first; no timestamps or IDs in the prefix; frozen tool order; append-only history
• Break-even for a 5-minute write is two requests
• Other levers: batch (~50%), routing, effort, output length, smaller contexts - track uncached, cached and output tokens separately

References

• Anthropic, Prompt caching
• OpenAI, Prompt caching guide
• Google, Gemini context caching
• Gim et al., Prompt Cache: Modular Attention Reuse for Low-Latency Inference (MLSys 2024)
• Zheng et al., SGLang: Efficient Execution of Structured Language Model Programs (NeurIPS 2024) - RadixAttention
• vLLM, Automatic Prefix Caching

Last reviewed: 2026-09