Obsidian Metadata
RAG Evaluation
RAG has two components that need different evaluation: retrieval (a search problem) and generation.
0. Start with error analysis on end-to-end traces
Look at what was retrieved vs what the model needed, and decide whether failures come from retrieval, generation, or both. Fix retrieval first. (“If your retriever pulls irrelevant context, your generator is doomed from the start.”)
1. Retrieval evaluation (classic IR metrics)
Build a dataset of queries → relevant chunks:
- Manual curation gives the highest quality.
- Synthetic QA (reverse generation): take a chunk, extract a fact, generate a question answered only by that fact. You get query–document pairs without manual annotation.
Given a chunk of text, extract a specific, self-contained fact from it. Then write a question that is directly and unambiguously answered by that fact alone. Return output in JSON format: { "fact": "...", "question": "..." } Chunk: "{text_chunk}" - Adversarial questions: pick target chunk A, find similar chunks B and C by embedding search, and write a question using B/C’s terminology that only A answers. (A: “April 2020, 17% revenue drop, largest since 2008”. B: “significant losses in 2008”. Q: “When did the company experience its largest revenue decline since the 2008 financial crisis?“)
- Filter synthetic questions for realism (few-shot LLM rating, keep 4–5 of 5; Likert is fine here because it’s curation, not measurement).
| Metric | Formula | Use for |
|---|---|---|
| Recall@k | relevant in top-k / total relevant | First-pass retrieval (priority): LLMs can ignore noise but can’t generate from missing content |
| Precision@k | relevant in top-k / k | Reranking |
| MRR | mean of 1 / rank of first relevant | Single-fact lookups |
| NDCG@k | DCG/IDCG with graded relevance, rel_i / log2(i+1) | Graded relevance, ranking quality (supplement with Recall) |
| Two-hop Recall@k | both ground-truth chunks in top-k | Multi-hop queries; classify hop-1 miss / hop-2 miss / rank-out |
Choosing k: factual lookup k = 1–2; synthesis queries k = 5–10.
Chunking is a hyperparameter
Grid search chunk size × overlap, re-indexing each time, and compare Recall@5 / NDCG@5. Try content-aware chunking (sections, paragraphs) and prepend title and section headers before embedding.
2. Generation evaluation
Use Error Analysis → human labels → targeted, validated LLM judges (see Judge Validation).
- Faithfulness / groundedness (A|C): hallucinations (even correct facts not in context count in RAG), omissions, misinterpretations.
- Answer relevance (A|Q): an answer can be faithful and still not answer the question.
Jason Liu’s “only 6 RAG evals” (Q, C, A)
| Tier | Relationship | Question |
|---|---|---|
| 1 | Retrieval precision/recall, MAP@K, MRR@K | Is the retriever finding the right docs? (fast, no LLM) |
| 2 | C|Q context relevance | Do the retrieved chunks address the question’s information need? |
| 2 | A|C faithfulness | Does the answer only claim what the context supports? |
| 2 | A|Q answer relevance | Does it answer what was asked? |
| 3 | C|A context support coverage | Does the context support every claim in the answer? |
| 3 | Q|C question answerability | Can the question be answered from this context at all? (strategic rejection) |
| 3 | Q|A self-containment | Can you infer the question from the answer alone? |
Diagnosis: wrong answer → check A|C. Irrelevant answer → A|Q. Missing info → C|Q or C|A. Domain emphasis differs: medical RAG needs stronger A|C, customer service A|Q, technical docs Q|C.
| Context relevance | Faithfulness | Answer relevance | Diagnosis |
|---|---|---|---|
| High | High | Low | Generator attended to the wrong section of a correct doc |
| High | Low | – | Hallucination or misinterpretation |
| Low | – | – | Retrieval problem: chunking, embeddings, query preprocessing |
Don't use Tier 2/3 judge prompts off the shelf
Run the full judge process (error analysis, prompt iteration, labeled examples, TPR/TNR) and correct estimates for judge error (Judge Bias Correction).
Beyond the framework
Error analysis will surface domain-specific failures that need their own evaluators: a medical RAG confusing adult vs pediatric dosages, a legal RAG mixing up jurisdictions.
Modern IR perspective (Stop Saying RAG Is Dead series)
- Classic IR metrics optimize for finding the #1 result. RAG needs coverage, diversity (corroboration), relevance (Nandan Thakur, FreshStack). Models that top BEIR can fail at real RAG.
- Reasoning retrievers (Rank1), late-interaction models (ColBERT: token-level detail; 150M models beating 7B), multiple representations with routing, and “you probably don’t need a graph DB”.
- Bigger context windows don’t replace retrieval (Judge Context and Context Rot).
Anti-patterns
- One end-to-end correctness metric with no retrieval/generation split.
- Metrics before reading traces.
- Overfitting to synthetic eval data. Check against real queries regularly.
- ROUGE, BERTScore, or cosine similarity as the primary generation metric (fine for retrieval).
See also Is RAG Dead · Document Processing and Chunk Size