Obsidian Metadata
| type | moc |
AI Evals Knowledge Base
A working reference for building evals for AI products. It’s based on Hamel Husain & Shreya Shankar’s AI Evals FAQ and every page it links to: the three-part blog series, the eval-smell, tools, and “revenge” posts, the evals-skills repo, the flashcards, Anthropic’s agent-evals guide, Jason Liu’s RAG evals, and the cited papers. Snapshot taken 2026-09-25.
If you only read three notes
- Eval Playbook: the step-by-step procedure
- Error Analysis: the activity everything else depends on
- Judge Validation: how to trust automated evals
Start by situation
| I’m… | Go to |
|---|---|
| New to evals | Reading Path - New to Evals |
| Unsure what to test | Reading Path - What to Test |
| Distrustful of my eval scores | Reading Path - Trust Your Scores |
| Working on something that feels impossible to evaluate | Reading Path - Hard to Evaluate |
| Spending too much time or money on evals | Reading Path - Reduce Time and Cost |
| Looking for a specific question | FAQ Index |
| Short on time | Principles and Opinions · Source - Evals Flashcards |
Map of the vault
Foundations
Finding failures
- Error Analysis · Theoretical Saturation · Criteria Drift · Reference Answers and Rubrics
- Trace Sampling Strategies · How Many Examples Do You Need · Synthetic Data · Evals with Sensitive Data · Diverse Query Evaluation
- Error Discovery Skill Workflow · What LLMs Can and Can't Automate in Evals
Building evaluators
- When to Build an Automated Evaluator · Eval-Driven Development
- Binary Pass-Fail vs Likert Scales · Generic Metrics
- Code-Based Evals · LLM-as-Judge · Critique Shadowing · Judge Context and Context Rot
- Judge Validation · Judge Bias Correction · Prompt Optimization
People and process
- Benevolent Dictator · Annotation Outsourcing · Team Collaboration on Evals · Making the Case for Evals
- Annotation Tools and Review Interfaces · Designing for Verification · Reviewing Large Traces
Production and operations
- CI Evals vs Production Monitoring · Guardrails vs Evaluators · Capability vs Regression Evals · pass@k and pass^k
- Combining Evals into a Single Metric · Eval Maintenance and Cadence · Maintaining Trust in Evals
- Eval Tooling and Vendors · Internal Eval Platforms · Prompt Management · Model Selection
- Capability Funnel and Experiment Roadmaps
Evaluating specific systems
- RAG Evaluation · Is RAG Dead · Document Processing and Chunk Size
- Agentic Workflow Evaluation · Transition Failure Matrix · Multi-Step Workflow Evaluation
- Multi-turn Conversation Evaluation · Human Handoff Evaluation
- Coding Agent Evaluation · Abstention and Uncertainty
Case studies
- Rechat Lucy · Honeycomb Query Assistant · Nurture Boss · GitHub Copilot · Nova Escola · Teresa Torres Interview Coach
Templates (copy these)
- Error Analysis Worksheet · Synthetic Data Prompts · Judge Prompt Template
- Agent Eval Task Spec · Trace Logging Schema · Eval Audit Checklist
Reference
Sources
- Hamel & Shreya: Source - AI Evals FAQ · Source - Your AI Product Needs Evals · Source - Using LLM-as-a-Judge · Source - A Field Guide to Rapidly Improving AI Products · Source - Its Hard to Eval Is a Product Smell · Source - Selecting the Right AI Evals Tool · Source - The Revenge of the Data Scientist · Source - Evals Skills for Coding Agents · Source - Evals Flashcards · Source - Show Me The Prompt · Source - Stop Saying RAG Is Dead · Source - Context Rot · Source - Case Study Putting Evals Into Production · Source - Do Automated Evals Work · Source - 99 Percent of Executives Are Misled by AI Advice
- External: Source - Demystifying Evals for AI Agents · Source - There Are Only 6 RAG Evals · Source - Who Validates the Validators · Source - GEPA · Source - Play Favorites Self-Bias · Source - Theoretical Saturation (Delve) · Source - Why I No Longer Recommend RAG for Coding Agents
- Video Library (17 video summaries in
Sources/Videos/) · Other Linked References
Conventions
- Folders:
Concepts/(ideas),Evaluating Specific Systems/(per-architecture guides),Case Studies/,Templates/,Guides/(FAQ index and reading paths),Sources/(one summary per linked page),_attachments/(flashcards and diagrams). - Frontmatter
type:is one of moc, playbook, concept, guide, case-study, template, source, reading-path, reference.sources:links back to where each idea came from. - Callouts:
[!important]core rules,[!warning]anti-patterns,[!tip]shortcuts. - The source material is opinionated. Where sources disagree, the notes say so (see Where sources disagree (worth knowing)).