Obsidian Metadata
| type | concept |
| sources | Source - AI Evals FAQ, Source - Evals Flashcards |
When to Build an Automated Evaluator
Not every failure mode needs an eval. Do a cost–benefit analysis before building one.
Decision flow
- Have you observed the failure with Error Analysis?
- No → stay away. Evals should target errors you observe or induce, not errors you imagine (see Eval-Driven Development).
- Is it a specification gap? Many teams find the LLM doesn’t meet preferences they never specified: short answers, formatting, step-by-step reasoning. Fix the prompt first. Nova Escola built an eval for “emits two learning goals instead of one”; a one-line prompt fix made the problem disappear.
- How much iteration will fixing it take?
- Little → unclear value. Could a cheap code assertion work? Does it make more sense to just fix it and move on?
- Lots → high value. Evals pay off most when you need to hill-climb against an error. Expensive evals such as an LLM judge may be justified.
- Pick the cheapest evaluator that works (see the cost hierarchy below).
Cost hierarchy
| Evaluator | Cost to build and maintain | Use for |
|---|---|---|
| Assertions / regex / schema / execution tests | Minimal | Objective failures (Code-Based Evals) |
| Reference-based checks | Low | Known correct answers |
| LLM-as-Judge | High: 100+ labeled examples, ongoing weekly maintenance, dev + PM + expert coordination | Persistent generalization failures you’ll iterate on, with subjective qualities |
| Human review | Highest | Calibration, subjective ground truth |
“Only build expensive evaluators for problems you’ll iterate on repeatedly.”
Budget reality
- Evaluation is part of development, not a separate line item, like debugging is part of software development.
- On Hamel and Shreya’s projects, 60–80% of development time went to error analysis and evaluation. Most of that effort is understanding failures (looking at data), not building automated checks.
- Many issues found are simple bugs you fix right away. They need no eval infrastructure.
Don’t use one accuracy bar for every app
Invest in evals in proportion to the stakes. Internal tools (e.g., Slack summaries) need little, and customer-facing features need a lot. Brainstorm worst-case scenarios (a coding agent can help, given traces, an app description, or the code) to decide which evals and guardrails matter (Video - How to Automate AI Evals Correctly).
Beware of 100% pass rates
If you pass 100% of your evals, you’re probably not challenging the system. A 70% pass rate may be more meaningful. Keep evals that catch real issues, not ones that make metrics look good. The “pass 100 tests → 100% accuracy on 100k inputs” advice (Casetext, amplified by YC) is flawed: a perfect score is a red flag for leakage or tests that are too weak (see Source - 99 Percent of Executives Are Misled by AI Advice). Saturated evals should be retired, run less often, or made harder (see Eval Maintenance and Cadence).
Related
Code-Based Evals · LLM-as-Judge · Eval-Driven Development · Eval Playbook