RAG System Design

Designing a production RAG system means turning requirements - corpus size, query volume, latency, freshness, access control and quality targets - into concrete choices for ingestion, indexing, retrieval, generation, caching and failure handling, with the numbers to justify them.

A Framework for the Design Discussion

Step	Questions to settle
1. Requirements	Corpus size and growth? Document types? QPS (average, peak)? Latency SLO (p95 TTFT and total)? Freshness (minutes, hours, daily)? Who may see what? Quality bar and cost per query?
2. Two pipelines	Ingest (offline, event-driven or batch) and query (online) - draw both
3. Components	Parsing and chunking; embedding model; index and store; retrieval (hybrid?), reranker; generator; citations
4. Numbers	Index size, latency budget, cost per query, throughput per component
5. Trade-offs	Latency vs quality (reranker, query rewriting), cost vs quality (model size, context size), freshness vs complexity
6. Operations	Evaluation gates, monitoring, failure modes, security (RAG in Production)

---

Reference Architecture

flowchart TD
    subgraph ING["📥 Ingestion (async)"]
        SRC["🗄️ Sources<br/>object storage, wikis, tickets, DBs"] --> EV["📬 Change events / scheduler"]
        EV --> Q1["🧾 Work queue"]
        Q1 --> W["⚙️ Workers<br/>parse → chunk → enrich → embed"]
        W --> IDX[("🗄️ Hybrid index<br/>vectors + BM25 + metadata/ACL")]
        W --> DOCS[("📦 Document store<br/>full text, versions")]
    end
    subgraph QRY["🔍 Query (sync)"]
        U["🧑 User + identity"] --> API["🚪 API / gateway<br/>auth, rate limits"]
        API --> QP["✏️ Query processing<br/>rewrite, filters, route"]
        QP --> RET["📡 Hybrid retrieval<br/>ACL-filtered, top 50"]
        RET --> RR["🎯 Reranker → top 5-8"]
        RR --> GEN["🤖 LLM, streaming<br/>grounded + cited"]
        GEN --> OUT(["💬 Answer + citations"])
    end
    IDX -.-> RET
    DOCS -.-> RR
    OUT -.-> FB["📊 Logs, feedback,<br/>sampled evals"]

    style ING fill:#e8e2d9,stroke:#ccc4b8
    style QRY fill:#d8dfe8,stroke:#b0bac8
    style FB fill:#dde4dc,stroke:#b0c4b0

---

Latency Budget

A p95 target of about 3 seconds end to end, with streaming so the user sees text in about 1 second:

Stage	Typical p95	Levers
Auth, routing, filters	10-30 ms	-
Query rewrite (LLM)	200-800 ms	Skip for standalone queries; use a small model
Query embedding	10-50 ms	Self-host or cache
Hybrid retrieval (ANN + BM25)	10-80 ms	Index tuning, replicas, filter selectivity
Reranking 50 candidates	30-300 ms	Smaller or GPU-hosted reranker; fewer candidates
LLM time to first token	300-1,500 ms	Smaller model, shorter context, prompt caching, lower reasoning effort
LLM generation	1-5 s	Stream it; concise answers

Rules of thumb: the LLM dominates, so context size, model choice and caching matter more than shaving milliseconds from the vector search; run retrieval branches in parallel; and every optional LLM step (rewrite, grading, reranking with an LLM) needs to earn its latency on the eval.

---

Scaling the Index

Sizing. 100M documents × 1.2 chunks × 768 dimensions × 4 bytes ≈ 369 GB of float32 vectors - more than a single machine's RAM for an in-memory HNSW index. Options, from Embeddings & Vector Search: int8 (≈92 GB) or binary with rescoring (≈12 GB in RAM), IVF-PQ or DiskANN, or a distributed managed service.

Concern	Approach
Throughput	Replicate the index; queries are read-only and scale horizontally
Size	Shard; each shard searched in parallel, results merged (top-k per shard, then global top-k)
Shard key	Hash sharding spreads load evenly but every query fans out to all shards; sharding by tenant or domain lets filtered queries hit one shard
Freshness	Event-driven upserts: on change, delete the document's chunks by doc_id, then insert new ones; idempotent workers; a dead-letter queue for failures
Big rebuilds (new embedding model, new chunking)	Build a new index alongside, evaluate, switch traffic with an alias (blue/green), keep the old one for rollback
Ingestion throughput	Batch embedding calls, parallel workers behind a queue, back-pressure on API rate limits

---

Multi-Tenancy and Access Control

Retrieval must never return a chunk the user isn't allowed to see - once a chunk is in the prompt, the model may repeat it.

Isolation model	How	Use when
Index per tenant	Separate collections/indexes	Strict contractual or regulatory isolation; few, large tenants
Namespace / partition per tenant	Engine-level partitions in a shared cluster	Many tenants of varied size - the common SaaS default
Shared index + mandatory filter	tenant_id and ACL fields on every chunk	Internal tools; many tiny tenants

Principles:

• Enforce at retrieval, server-side, from the authenticated identity - never from a parameter the client or the model can set.
• Fail closed: if the user's permissions can't be resolved, return nothing.
• Mirror source permissions (document ACLs, groups) into chunk metadata at ingest, and re-sync when permissions change - stale ACLs are a leak. How connectors, nested groups, ACL lag and late binding work is covered in Enterprise Data Integration.
• Test it: include cross-tenant probes in the eval set and alert on any hit.
• Cache keys must include the tenant and the permission set (see below).

---

Caching

Layer	Key → value	Hit rate	Risks
Query embedding cache	normalised query text → vector	Moderate	Negligible
Retrieval cache	(query, filters, index version) → candidate ids	Moderate	Stale after index updates - include index version in key
Exact answer cache	(normalised query, tenant, permissions, index version, prompt version) → answer	Low-moderate; high for FAQ-like traffic	Staleness - invalidate on document change
Semantic answer cache	nearest cached query above a similarity threshold → answer	Higher	False hits: "refund policy for Enterprise" and "for Starter" may be 0.95 similar but need different answers; cross-tenant leaks if the key omits the tenant
Prompt cache (provider)	Stable prompt prefix → KV state	Automatic	See Prompt Caching & Cost

Implement a semantic cache with a real vector index (not a scan over cache keys), partition it by tenant and permissions, use a conservative threshold tuned on labelled near-duplicate pairs, and keep TTLs short. For many products, an exact cache plus prompt caching delivers most of the savings with none of the false-hit risk.

---

Graceful Degradation

flowchart TD
    VF["🗄️ Vector search down"] --> VF1["↪️ BM25-only retrieval"]
    RF["🎯 Reranker down or slow"] --> RF1["↪️ Serve first-stage order<br/>(tighter k)"]
    LF["🤖 LLM rate-limited / down"] --> LF1["↪️ Fallback model or provider"]
    LF1 --> LF2["↪️ Show retrieved passages<br/>without a generated answer"]
    EF["🧬 Embedding API down"] --> EF1["↪️ Cached embeddings or BM25"]

    style VF fill:#e8e0d4,stroke:#c8b89a
    style RF fill:#e8e0d4,stroke:#c8b89a
    style LF fill:#e8e0d4,stroke:#c8b89a
    style EF fill:#e8e0d4,stroke:#c8b89a

Wrap each dependency with timeouts and circuit breakers, and make the degraded mode explicit to users ("showing matching documents; answer generation is temporarily unavailable") rather than silently answering from the model's own knowledge.

---

Worked Design: Enterprise Knowledge Assistant

Prompt: 10M documents across HR, Legal and Engineering; 5,000 employees; answers with citations; p95 under 3 s; daily updates; department-level access control.

Decision	Choice and reason
Load	5,000 users × 20 queries/day ≈ 100K/day ≈ 3.5 QPS average over 8 working hours, ~20 QPS peak
Chunks and index	~12M chunks × 768 dims ≈ 37 GB float32 → int8 HNSW (~9 GB) with rescoring; hybrid BM25 in the same engine
Access control	department and document ACL groups on every chunk; filter from the SSO identity; fail closed
Retrieval	Hybrid top 50 → reranker top 6 → mid-size LLM with prompt caching of the instructions
Freshness	Nightly batch plus event-driven updates for high-churn spaces; blue/green rebuild for model changes
Quality gates	300-question golden set per department (incl. unanswerable and cross-department probes); retrieval metrics on every index change; groundedness sampled in production
Degradation	BM25-only fallback; passages-only mode if the LLM is unavailable
Estimated latency	~40 ms retrieval + ~150 ms rerank + ~800 ms TTFT → first token ≈ 1 s; full answer ≈ 2.5 s

---

Check Yourself

Exercises

Study Notes

Must-know:

• Design flow: requirements → two pipelines → components → numbers → trade-offs → operations
• Latency is dominated by the LLM; budget each stage; parallelise retrieval branches; stream
• Index sizing: N × d × 4 bytes float32; compress, shard, replicate; blue/green rebuilds for model changes
• Freshness: event-driven delete-then-insert with idempotent workers, or batch
• Multi-tenancy: per-tenant index, namespaces, or shared + mandatory filters; enforce server-side from identity; fail closed; test with probes
• Caches: embedding, retrieval, exact answer, semantic answer (false hits - key by tenant/permissions, conservative thresholds), provider prompt cache
• Degrade explicitly: BM25 fallback, skip reranker, fallback model, passages-only mode

References

• Gao et al., Retrieval-Augmented Generation for Large Language Models: A Survey (2023)
• Barnett et al., Seven Failure Points When Engineering a Retrieval Augmented Generation System (CAIN 2024)
• Google Cloud, RAG infrastructure for generative AI using Agent Platform and Vector Search
• AWS, Amazon Bedrock Knowledge Bases

Last reviewed: 2026-09