Eval Playbook

Obsidian Metadata

type

playbook

Eval Playbook: building evals for a new AI product, step by step

This combines the FAQ, the three-part blog series, the evals-skills repo, and Anthropic’s agent guidance into one ordered procedure. Each step links to the concept note with the details.

The loop

Instrument → look at data (error analysis) → fix the obvious → build the few evals that matter → validate them → run them in CI and production → feed new failures back → repeat. Expect 60–80% of effort on looking at data, not on building eval infrastructure.

graph LR
  A[0. Instrument traces] --> B[1. Get ~100 traces<br/>real or synthetic]
  B --> C[2. Error analysis<br/>open → axial coding]
  C --> D{3. Triage each<br/>failure mode}
  D -->|spec gap / bug| E[Fix now]
  D -->|persistent + objective| F[4a. Code eval]
  D -->|persistent + subjective| G[4b. LLM judge]
  G --> H[5. Validate vs human labels<br/>TPR / TNR on test]
  F --> I[6. Repeatable eval set<br/>CI gate]
  H --> I
  H --> J[7. Production monitoring<br/>sampled, async, corrected rates + CIs]
  E --> C
  J -->|new failures| C
  I -->|regressions| C

---

Step 0 · Prerequisites (week 0)

• Instrument traces end to end: messages, tool calls with args, results and state changes, retrievals, handoffs, business outcomes, versions, metadata. Use the Trace Logging Schema. See Traces.
• Name the Benevolent Dictator: one domain expert who decides what “good” means.
• Pick a review surface: a notebook or a small custom app built by a coding agent. See Annotation Tools and Review Interfaces and Minimum Viable Eval Setup.
• Before building evals, check whether the product can be designed to be easier to verify (provenance, smaller units). See Designing for Verification.
• If there’s sensitive data, choose a strategy. See Evals with Sensitive Data.

Step 1 · Assemble ~100 diverse traces

• Have production data? Sample for diversity: random plus clusters plus outliers, optionally flagged or feedback traces, and always some random. See Trace Sampling Strategies.
• No data yet? Generate synthetic inputs with dimensions → 20 hand-written tuples → LLM tuples → separate tuple→query prompt, then run them through the real system. See Synthetic Data and Synthetic Data Prompts.
• Don’t generate data for problems you can simply fix in the prompt.

Step 2 · Error analysis (the most important step)

Use the Error Analysis Worksheet or the Error Discovery Skill Workflow.

1. Open coding: the domain expert (or you) writes free-text notes on ≥ 30 traces before any AI suggestions. Note the first upstream failure. Record non-model problems too. Don’t root-cause yet. No rubric yet (Criteria Drift).
2. Axial coding: group notes into a failure taxonomy and count each mode. An LLM can draft the groups; you validate them.
3. Iterate: have an agent propose likely instances across the pool; accept or reject each; re-review earlier traces; continue to Theoretical Saturation (≥ 100 traces).
4. For agents and pipelines, also build a Transition Failure Matrix. For multi-turn, see Multi-turn Conversation Evaluation.

Output: a ranked list of failure modes with counts and example trace IDs.

Step 3 · Triage every failure mode

Use When to Build an Automated Evaluator:

Situation	Action
Spec gap (the prompt never asked for it)	Fix the prompt. No eval, or at most a cheap regression check
Plain bug, missing feature, UX issue	Fix it (it’s still worth recording)
Persistent failure that needs lots of iteration, and is objective	Code eval (Step 4a)
Persistent failure that needs lots of iteration, and is subjective	LLM judge / classifier (Step 4b)
Rare, low impact, easy to fix	Fix and move on

Also write down the Pass/Fail definition for each mode. Now you can write a rubric (Reference Answers and Rubrics).

Step 4a · Code-based evaluators

One check per failure mode (schema, regex, ID cross-check, DB state, execution tests). Test it on Pass, Fail, and borderline examples. See Code-Based Evals.

Step 4b · LLM judges (only where needed)

• One failure mode per judge, binary output, critique before verdict, few-shot examples from train only, and only the context that mode needs. See LLM-as-Judge, Judge Prompt Template, Judge Context and Context Rot.
• Start with the strongest model you can afford, pin the version, and downsize later.

Step 5 · Validate every judge against human labels

• 100–200 expert labels per failure mode, balanced (30–50 Pass and 30–50 Fail in each of dev and test).
• Split train 10–20% / dev 40–45% / test 40–45%. Iterate on dev only.
• Measure TPR and TNR (not accuracy). Target > 90% (minimum ~80%). Inspect disagreements. Split criteria that won’t align.
• Freeze the judge, then run it once on test. If test ≪ dev, you’ve overfit.
• Optional: prompt tuning (GEPA or an agent) against dev, after manual inspection.
• See Judge Validation.

Step 6 · Build the repeatable eval set (CI)

• Include examples from error analysis for important workflows and confirmed failures, plus regression tests for past bugs and edge cases. It usually grows to 100+ examples.
• Favor deterministic checks in CI, because cost per run matters. Gate merges.
• Report a pass-all rate with drill-down, grouped by theme or severity (release-blocking vs tolerable). See Combining Evals into a Single Metric.
• Don’t chase 100%. A saturated eval gives no signal (Capability vs Regression Evals).
• Track results over time outside CI (e.g., in your analytics DB) along with prompt and test versions. See CI Evals vs Production Monitoring.

Step 7 · Production

• Guardrails only for fast, objective, high-impact failures with a very low false-positive rate. See Guardrails vs Evaluators.
• Online monitoring: sample traffic (e.g., Nova Escola uses 2% daily), run judges asynchronously, bias-correct the rates, track CIs, and alert when the lower bound crosses a threshold. See Judge Bias Correction.
• A/B tests for significant changes. Pair them with product metrics (churn, retention, acceptance) for long-horizon tracking. See Three Levels of Evaluation.

Step 8 · Keep the loop running

• Error analysis on every significant change: 100+ fresh traces every 2–4 weeks, plus 10–20 outlier traces weekly, plus after incidents. See Eval Maintenance and Cadence.
• Move new failures from production into CI.
• Re-validate judges when the prompt, model, or data distribution changes.
• Retire or harden evals that always pass. Accept that scores won’t be comparable across eval-set versions.
• Periodically run the Eval Audit Checklist.
• Share findings as stories (“47 issues caught before users saw them”). See Making the Case for Evals.

---

System-specific add-ons

Building…	Also read
RAG	RAG Evaluation (retrieval IR metrics first, then generation judges)
Agents / tool use	Agentic Workflow Evaluation, Agent Eval Task Spec, pass@k and pass^k
Coding agent	Coding Agent Evaluation
Chatbot / multi-turn	Multi-turn Conversation Evaluation, Human Handoff Evaluation
Pipelines / workflows	Multi-Step Workflow Evaluation
Long documents	Document Processing and Chunk Size
Should-refuse behaviors	Abstention and Uncertainty

Top anti-patterns to avoid

1. Building dashboards of generic 1–5 scores (Generic Metrics, Binary Pass-Fail vs Likert Scales).
2. Skipping error analysis or delegating it to an LLM or outsiders (What LLMs Can and Can't Automate in Evals, Annotation Outsourcing).
3. Unvalidated judges; reporting accuracy; leaking dev/test examples into few-shot (Judge Validation).
4. Writing rubrics or evals before looking at data (Criteria Drift, Eval-Driven Development).
5. Building evals for things a prompt fix would solve (Nova Escola).
6. Holistic “is this good?” judges.
7. Making reviewers read raw JSON.
8. Treating model switching as the first lever (Model Selection).
9. Adopting framework or guardrail prompts without reading them (Source - Show Me The Prompt).
10. Chasing 100% pass rates.