Long Context vs RAG vs CAG

With context windows of hundreds of thousands to millions of tokens, you can often skip retrieval and put the whole corpus in the prompt - either on every call (long-context prompting) or once, with its KV cache reused across queries (cache-augmented generation, CAG). This chapter is the decision guide.

Three Ways to Put Knowledge in Context

flowchart TD
    subgraph RAG["📡 RAG"]
        R1["Retrieve top-k chunks<br/>per query"] --> R2["Small context<br/>(~2-20K tokens)"]
    end
    subgraph LC["📜 Long-context prompting"]
        L1["Send the whole corpus<br/>with every query"] --> L2["Huge context<br/>(100K-1M+ tokens)"]
    end
    subgraph CAG["⚡ Cache-augmented generation"]
        C1["Prefill the corpus once;<br/>keep its KV cache"] --> C2["Each query reuses the cache;<br/>only the question is new"]
    end

    style RAG fill:#dde4dc,stroke:#b0c4b0
    style LC fill:#e8e0d4,stroke:#c8b89a
    style CAG fill:#d8dfe8,stroke:#b0bac8

• RAG pays for retrieval infrastructure and risks missing evidence, but each call is small and cheap and the corpus can be any size.
• Long-context prompting gives the model everything - no retrieval misses, and whole-document reasoning - but every call pays to process the full corpus, and quality degrades as context grows (see Context Engineering).
• CAG (Chan et al., 2024) keeps long context's completeness while avoiding repeated prefill: the corpus's KV cache is computed once and reused. With hosted APIs, CAG is simply prompt caching of a corpus-sized prefix; self-hosted, it means keeping (or offloading and reloading) a large KV cache.

What the Evidence Says

The consistent picture: long context wins on completeness when the corpus fits and the task needs broad reading; RAG wins on cost and scale; hybrids capture most of both.

The Cost Arithmetic

Example: a 400K-token corpus, 20,000 questions per day. Prices: $3/M input, $0.30/M cached input, $15/M output; 500 output tokens per answer.

Caching cuts long-context cost by about 10×, and RAG is still about 4× cheaper here - plus faster (prefill for 8K tokens vs reading a 400K cache) and not limited by corpus size. But if the corpus were 40K tokens, cached long context would cost about $0.02 per query - the same as RAG, with no retrieval to build or maintain.

Self-hosted CAG: KV memory

KV-cache size per token = 2 (K and V) × layers × KV heads × head dimension × bytes per value. For a model with 64 layers, 8 KV heads (GQA), head dimension 128, in bf16:

2 × 64 × 8 × 128 × 2 bytes = 262,144 bytes ≈ 256 KB per token
400K-token corpus → ~105 GB of KV cache

That is more than one H100's memory for a single cached corpus - before the model weights. Self-hosted CAG needs FP8 KV caches, KV offloading to CPU or SSD (e.g. LMCache), or a smaller corpus (see The Modern Serving Stack).

Decision Guide

flowchart TD
    A{"Does the corpus fit well<br/>within the window?"} -->|"no"| RAG1["📡 RAG / agentic RAG"]
    A -->|"yes"| B{"Same corpus for all users<br/>and changes rarely?"}
    B -->|"no"| RAG2["📡 RAG with filters"]
    B -->|"yes"| C{"Many queries per<br/>cache lifetime?"}
    C -->|"yes"| CAGN["⚡ Cached long context (CAG)"]
    C -->|"no"| D{"Needs broad reading<br/>or whole-document reasoning?"}
    D -->|"yes"| LCN["📜 Long context"]
    D -->|"no"| RAG3["📡 RAG or Self-Route"]

    style CAGN fill:#d8dfe8,stroke:#b0bac8
    style LCN fill:#e8e0d4,stroke:#c8b89a
    style RAG1 fill:#dde4dc,stroke:#b0c4b0
    style RAG2 fill:#dde4dc,stroke:#b0c4b0
    style RAG3 fill:#dde4dc,stroke:#b0c4b0

Whatever you choose, measure on your questions: build the eval once (RAG Evaluation) and run both approaches at your real corpus size.

Check Yourself

Exercises

Study Notes

Must-know:

• RAG: small cheap calls, any corpus size, risk of retrieval misses
• Long context: completeness and whole-document reasoning; cost scales with corpus; quality degrades with length
• CAG: precompute and reuse the corpus's KV cache = prompt caching for hosted APIs
• Evidence: long context often wins on quality when it fits; RAG wins on cost; Self-Route hybrids get most of both
• Cost: corpus × cached price vs RAG context × input price; break-even corpus ≈ 10× the RAG context at a 0.1× cache price
• KV memory = 2 × layers × KV heads × head dim × bytes per token - large corpora need FP8 KV or offloading
• Per-tenant data, frequent changes and huge corpora point to RAG

References

• Chan et al., Don't Do RAG: When Cache-Augmented Generation is All You Need for Knowledge Tasks (2024)
• Li et al., Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach (EMNLP 2024 Industry)
• Xu et al., Retrieval meets Long Context Large Language Models (ICLR 2024)
• Hsieh et al., RULER (COLM 2024)
• LMCache - KV cache storage and reuse for vLLM

Last reviewed: 2026-09