Obsidian Metadata
| type | index |
| source | Source - AI Evals FAQ |
FAQ Index: every question, a one-line answer, and where to read more
All ~55 questions from the AI Evals FAQ (Hamel Husain & Shreya Shankar), grouped in the FAQ’s own sections. → points to the vault note with the full answer. ↗ links the original “focus view” page.
Getting Started & Fundamentals
| Question | Short answer | Read |
|---|---|---|
| What are AI evals? | Tests that systematically measure whether your AI does what you want. Product evals ≠ model benchmarks | → Product Evals vs Model Benchmarks · ↗ |
| What is a trace? | The complete record of one user query through to the final response: messages, tool calls, retrievals, all agents | → Traces · ↗ |
| What’s a minimum viable evaluation setup? | 30 minutes reviewing 20–50 outputs after significant changes, one domain expert, a notebook or small custom UI | → Minimum Viable Eval Setup · ↗ |
| How much budget for evals? | It’s part of development; expect 60–80% of dev time on error analysis and evaluation. Beware 100% pass rates | → When to Build an Automated Evaluator · ↗ |
| Will these methods matter in 5–10 years? | Yes. You still need to verify the right problem is solved, and criteria drift means you must observe outputs | → Product Evals vs Model Benchmarks · ↗ |
| How do I make the case to my team? | Don’t sell “evals”. Show what you find in 50–100 conversations and keep a log of prevented issues | → Making the Case for Evals · ↗ |
Error Analysis & Data Collection
| Question | Short answer | Read |
|---|---|---|
| Why is error analysis so important, and how? | It decides which evals to write. Dataset → open coding → axial coding → iterate to saturation | → Error Analysis · ↗ |
| Need a reference answer or rubric first? | No. Rubrics made too early blind reviewers, and criteria drift. Write the rubric after error analysis | → Reference Answers and Rubrics · ↗ |
| Record problems that aren’t the model’s fault? | Yes. Anything that makes the product less useful. Defer root cause | → Error Analysis · ↗ |
| How many examples do I need? | ~100 traces (≥ 30 self-annotated) for discovery; coverage for code evals; 100–200 labels per judge; 100+ for the repeatable set | → How Many Examples Do You Need · ↗ |
| Surface problematic traces beyond user feedback? | Random sampling → stress tests → existing evals as a screen → outliers, stratification, generic metrics as signals | → Trace Sampling Strategies · ↗ |
| How often to re-run error analysis? | On significant changes; 100+ fresh traces per 2–4-week cycle; 10–20 outliers weekly in between | → Eval Maintenance and Cadence · ↗ |
| “Gold” dataset stale? | Refresh via error analysis; retire always-passing evals; use product metrics for long-term trends | → Eval Maintenance and Cadence · ↗ |
| Best approach to synthetic data? | Failure hypotheses → dimensions → 20 manual tuples → LLM tuples → separate tuple→query step → run the real system | → Synthetic Data · ↗ |
| When is synthetic data unreliable? | Complex domain docs, low-resource languages, unverifiable realism, high stakes, under-represented groups | → Synthetic Data · ↗ |
| Evals with sensitive data? | In order: permitted real data → experts verifying in-product → redaction → synthetic as last resort | → Evals with Sensitive Data · ↗ |
| Diverse user queries? | Let failure patterns, not a predefined query matrix, drive eval design | → Diverse Query Evaluation · ↗ |
| Efficiently sample production traces? | Random, clustering, data analysis, classification, feedback (exploratory → targeted); always keep some random; active learning | → Trace Sampling Strategies · ↗ |
Evaluation Design & Methodology
| Question | Short answer | Read |
|---|---|---|
| Why binary pass/fail instead of 1–5? | Clearer, more consistent, faster, actionable. Decompose into binary sub-checks for gradual progress | → Binary Pass-Fail vs Likert Scales · ↗ |
| Combine evals into a single metric? | Pass-all rate with drill-down; group by theme or severity; avoid weighted composites | → Combining Evals into a Single Metric · ↗ |
| Eval-driven development? | Generally no. Write evals for errors you discover. Exception: crisp known constraints | → Eval-Driven Development · ↗ |
| Build evaluators for every failure mode? | No. Fix spec gaps first; cheap code checks; judges only for persistent failures you’ll iterate on | → When to Build an Automated Evaluator · ↗ |
| Which model for automated evals? | Code first; else a Pass/Fail classifier (LLM, Jev, BERT, LR) validated on human labels; start strong, then downsize | → LLM-as-Judge · ↗ |
| Can I use Jev? | Yes. It’s a classifier, validated like any judge; cheap classifiers make prompt tuning cheaper | → Judge Validation · ↗ |
| How do I trust my automated eval? | Human labels; train/dev/test split; TPR and TNR; check test vs dev for overfitting | → Judge Validation · ↗ |
| Judge won’t agree with humans? | ≥ 50/50 labels; read disagreements (context, instructions, labels); one judge per failure; tune later; held-out test | → Judge Validation · ↗ |
| Use “ready-to-use” metrics? | No, as quality measures. OK as exploration signals | → Generic Metrics · ↗ |
| Similarity metrics (BERTScore, ROUGE)? | Not for LLM outputs generally; useful for retrieval and diversity | → Generic Metrics · ↗ |
| Same model for task and judge? | Usually fine. Alignment on held-out labels is what matters | → LLM-as-Judge · ↗ |
| How much context for a judge? | Only what that failure mode needs; ablate; search tool only if unavoidable | → Judge Context and Context Rot · ↗ |
| Evaluate “knowing what it doesn’t know”? | Balanced answerable and unanswerable set; Pass = answers the former and abstains on the latter | → Abstention and Uncertainty · ↗ |
Human Annotation & Process
| Question | Short answer | Read |
|---|---|---|
| How many annotators? | Usually one benevolent dictator. If several, use Cohen’s Kappa and alignment sessions | → Benevolent Dictator · ↗ |
| Make outputs easier to evaluate? | Design for verification (intermediate, checkable units with provenance), then a frictionless review UI, then debug the review process | → Designing for Verification · ↗ |
| Should PMs and engineers collaborate? | Yes at first for shared context, then a benevolent dictator; judge outcomes, not implementation | → Team Collaboration on Evals · ↗ |
| Can non-experts help? | Yes. Lots of low-hanging fruit; watch experts to build better tools | → Team Collaboration on Evals · ↗ |
| Outsource annotation? | Usually a big mistake; exceptions are mechanical tasks, context-free tasks, hired SMEs | → Annotation Outsourcing · ↗ |
| Review a really large trace? | First upstream failure; progressive disclosure; expert-guided evidence extraction | → Reviewing Large Traces · ↗ |
| What can LLMs automate? | Axial coding drafts, mapping, prompt suggestions, analysis. Not open coding, taxonomy validation, ground truth, or root cause | → What LLMs Can and Can't Automate in Evals · ↗ |
| Stop writing prompts manually? | Be skeptical early; optimizers can’t find new failures; use them for the last mile | → Prompt Optimization · ↗ |
Tools & Infrastructure
| Question | Short answer | Read |
|---|---|---|
| Custom annotation tool or off-the-shelf? | Build custom: the most impactful investment, ~10× faster iteration | → Annotation Tools and Review Interfaces · ↗ |
| What makes a good review interface? | Domain rendering, progress and hotkeys, cluster/filter/search, prioritize flagged traces; keep it minimal | → Annotation Tools and Review Interfaces · ↗ |
| Tooling gaps to fill yourself? | Pattern discovery, AI assistance, custom evaluators, bulk export and write-back APIs | → Eval Tooling and Vendors · ↗ |
| What should an internal platform standardize? | Processes (error analysis, sampling, validation), not metrics; shared storage and labeling | → Internal Eval Platforms · ↗ |
| Favorite eval vendor? | None. Features converge; choose on support; build custom on top | → Eval Tooling and Vendors · ↗ |
| Version and manage prompts? | Git; notebooks or an integrated prompt environment for experiments | → Prompt Management · ↗ |
| System vs user prompt? | Experiment. System = static constitution; user = task, context, examples; when unsure, user | → Prompt Management · ↗ |
Production & Deployment
| Question | Short answer | Read |
|---|---|---|
| CI/CD vs production monitoring? | CI prevents known regressions with small curated deterministic sets; monitoring samples live traffic async with CIs; feed new failures back to CI | → CI Evals vs Production Monitoring · ↗ |
| Guardrails vs evaluators? | Guardrails are inline, fast, low-FP, block; evaluators are async and measure | → Guardrails vs Evaluators · ↗ |
| Can evaluators auto-fix outputs? | Only a subset: depends on latency, cost, and FP/FN trade-offs. Rarely a synchronous LLM judge | → Guardrails vs Evaluators · ↗ |
| Time on model selection? | Little, until error analysis says the model is the problem | → Model Selection · ↗ |
Domain-Specific Applications
| Question | Short answer | Read |
|---|---|---|
| Is RAG dead? | No. Naive vector retrieval for coding agents is; retrieval matters more than ever | → Is RAG Dead · ↗ |
| Evaluate a coding agent? | Public benchmarks if broad; private task benchmark with environment and tests; error analysis; quality metrics; A/B | → Coding Agent Evaluation · ↗ |
| Evaluate a RAG system? | Retrieval with IR metrics first, then generation with validated judges; Jason Liu’s 6 relationships | → RAG Evaluation · ↗ |
| Chunk size for document processing? | Fixed-output → big chunks; expansive-output → small chunks plus map-reduce; tune as a hyperparameter | → Document Processing and Chunk Size · ↗ |
| Debug multi-turn traces? | Goal met? First upstream failure; user-visible first; simplify to single-turn; N-1 or simulated users | → Multi-turn Conversation Evaluation · ↗ |
| Sessions with human handoffs? | Trace through to resolution; log handoff details; track quality and rate | → Human Handoff Evaluation · ↗ |
| Complex multi-step workflows? | Log to the business outcome; outcome and process metrics; segment by stage; transition matrix | → Multi-Step Workflow Evaluation · ↗ |
| Agentic workflows? | End-to-end success first, then step-level diagnostics; tool calls as separate checks; transition matrices | → Agentic Workflow Evaluation · ↗ |