Code Lab 01 - Retrieval Evaluation

Measure retrieval instead of guessing: build BM25, dense, hybrid, reranked, learned-sparse and quantized retrievers over the same public benchmark, compute nDCG@10, Recall@k and MRR yourself, and see which "best practices" actually help on this data.

← Back to Overview: RAG · Concepts: Embeddings & Vector Search · Retrieval & Reranking · RAG Evaluation

What's In This Lab

flowchart LR
    C["📚 SciFact corpus<br/>+ 300 labelled claims"] --> B["🔤 BM25"]
    C --> D["🧬 Dense"]
    C --> S["🧮 SPLADE (optional)"]
    B --> H["🔀 RRF"]
    D --> H
    H --> R["🎯 Cross-encoder<br/>rerank top 50"]
    D --> Q["🗜️ int8 / binary<br/>+ rescore (optional)"]
    B & D & S & H & R & Q --> M["📏 nDCG@10, R@10,<br/>R@100, MRR@10"]

    style H fill:#d8dfe8,stroke:#b0bac8
    style R fill:#e8e0d4,stroke:#c8b89a
    style M fill:#dde4dc,stroke:#b0c4b0

Run It

cd 12-RAG/CodeLabs/01-Retrieval-Evaluation
pip install -r requirements.txt

python retrieval_lab.py --limit 50              # quick run on 50 queries
python retrieval_lab.py                         # BM25, dense, hybrid, rerank
python retrieval_lab.py --quantization --splade # everything
python retrieval_lab.py --embed-model BAAI/bge-small-en-v1.5

A reference run (300 queries):

system                                  nDCG@10    R@10   R@100  MRR@10
BM25                                      0.662   0.774   0.876   0.631
Dense                                     0.648   0.788   0.925   0.607
Hybrid (RRF)                              0.690   0.819   0.955   0.655
Hybrid + rerank top-50                    0.686   0.809   0.955   0.655
SPLADE (learned sparse)                   0.710   0.820   0.942   0.679
dense int8 (4x smaller)                   0.622   0.782   0.922   0.576
dense binary + rescore (32x smaller)      0.612   0.754   0.900   0.572

The BM25 and SPLADE numbers are close to published BEIR results for SciFact (about 0.665 and 0.70), which is a useful sanity check that the metrics are implemented correctly.

Walkthrough - What to Look At

1. BM25 beats the small dense model at the top of the ranking (nDCG@10 0.662 vs 0.648). Scientific claims are full of precise terms; a general-purpose 22M-parameter embedding model trained on web data doesn't know them. Dense wins on Recall@100 (0.925 vs 0.876) - it finds more of the relevant documents, just not always at the top.
2. Hybrid beats both on every metric. The two retrievers make different mistakes, and RRF rewards documents both agree on. Recall@100 of 0.955 means a reranker has almost everything it needs in the candidate set.
3. The reranker doesn't help here. ms-marco-MiniLM-L6-v2 was trained on web search queries; scientific claim verification is a different task and domain. A reranker is a model with a training distribution - evaluate it rather than assuming it helps.
4. SPLADE is the best single retriever - learned term weights plus expansion - but encoding the corpus took ~4 minutes on a laptop versus ~25 s for the small dense model, and needs more memory (the vocabulary-sized output per token).
5. Quantization costs a few points with this model: int8 keeps 96% and binary-plus-rescore 94% of dense nDCG@10, for 4× and 32× less memory. Models trained for quantization lose less.
6. Read ndcg_at_k, recall_at_k and mrr_at_k - each is a few lines, and writing them yourself removes the mystery from leaderboard numbers.

Check Yourself

Exercises

References

• Thakur et al., BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models (NeurIPS 2021)
• Wadden et al., Fact or Fiction: Verifying Scientific Claims (SciFact) (EMNLP 2020)
• Lù, BM25S: Orders of magnitude faster lexical search via eager sparse scoring (2024)
• Sentence Transformers documentation

Last reviewed: 2026-09