Obsidian Metadata
How Many Examples Do You Need?
Building evals is a pipeline, and each stage needs a different amount of data.
| Stage | What to do | How much |
|---|---|---|
| 1. Review the application (Error Analysis) | Read traces, write down failures | Pool of ~100 diverse traces. Annotate ≥ 30 yourself before any agent suggestions. Continue to Theoretical Saturation (≥ 100; go past 100 while still learning) |
| 2a. Code-based evaluator | Objective rule | Coverage, not volume. Pass and Fail examples for every condition plus important edge cases. A one-rule check may need only a few of each |
| 2b. LLM judge / classifier | Needs human judgment | 100–200 labeled examples per failure mode from a trusted expert. Split ~10–20% train / 40–45% dev / 40–45% test, with 30–50 Pass and 30–50 Fail in each of dev and test. < 60 → CIs too wide |
| 3. Repeatable eval set (CI / regression) | Important workflows + confirmed failures | Often grows to 100+. Size is set by coverage: every important workflow and known failure represented, while staying cheap enough to run often |
Other reference points:
- Minimum viable setup: 30 minutes reviewing 20–50 outputs whenever you make significant changes (see Minimum Viable Eval Setup).
- Making the case to your team: look at 50–100 real conversations (see Making the Case for Evals).
- Judge debugging: at least 50 passing and 50 failing labeled examples.
- Production review: 100+ fresh traces per cycle (every 2–4 weeks), plus 10–20 weekly outliers.
- Anthropic (agents): 20–50 simple tasks from real failures is a great start. Early changes have large effects, so small samples suffice. Mature agents need larger, harder sets to detect smaller effects.
- Quality > quantity. You learn more from carefully investigating a few failures than from rushing through many. 100 diverse traces beat thousands of superficial labels.
Related
Judge Validation · Trace Sampling Strategies · CI Evals vs Production Monitoring