Obsidian Metadata
| type | source |
| url | https://hamel.dev/blog/posts/evals-faq/ |
| authors | Hamel Husain, Shreya Shankar |
| published | 2026-09-18 |
| captured | 2026-09-25 |
Source: AI Evals: Everything You Need to Know (the Evals FAQ)
URL: https://hamel.dev/blog/posts/evals-faq/ · Authors: Hamel Husain & Shreya Shankar
The questions Hamel and Shreya get most often from teaching 5,000+ engineers and PMs in their AI Evals course. “These are sharp opinions about what works in most cases. They are not universal truths. Use your judgment.”
~55 questions in 7 sections. Every question is mapped to vault notes in the FAQ Index.
| Section | Main vault notes |
|---|---|
| Getting Started & Fundamentals | Product Evals vs Model Benchmarks, Traces, Minimum Viable Eval Setup, Making the Case for Evals |
| Error Analysis & Data Collection | Error Analysis, How Many Examples Do You Need, Trace Sampling Strategies, Synthetic Data, Evals with Sensitive Data |
| Evaluation Design & Methodology | Binary Pass-Fail vs Likert Scales, LLM-as-Judge, Judge Validation, Generic Metrics, Combining Evals into a Single Metric |
| Human Annotation & Process | Benevolent Dictator, Annotation Outsourcing, What LLMs Can and Can't Automate in Evals, Designing for Verification |
| Tools & Infrastructure | Annotation Tools and Review Interfaces, Eval Tooling and Vendors, Internal Eval Platforms, Prompt Management |
| Production & Deployment | CI Evals vs Production Monitoring, Guardrails vs Evaluators, Model Selection |
| Domain-Specific Applications | RAG Evaluation, Agentic Workflow Evaluation, Multi-turn Conversation Evaluation, Coding Agent Evaluation, … |
Reading paths the FAQ recommends
| Where you are | Guide |
|---|---|
| I’m new to evals | Reading Path - New to Evals |
| I don’t know what to test | Reading Path - What to Test |
| I don’t trust my eval scores | Reading Path - Trust Your Scores |
| My product feels too hard to evaluate | Reading Path - Hard to Evaluate |
| Evals take too much time or money | Reading Path - Reduce Time and Cost |
Where it says to start
Part 1 Source - Your AI Product Needs Evals → Part 2 Source - Using LLM-as-a-Judge → Part 3 Source - A Field Guide to Rapidly Improving AI Products.