Obsidian Metadata
| type | concept |
| sources | Source - AI Evals FAQ |
Making the Case for Evals
Don’t try to sell your team on “evals”. Show them what you find when you look at the data.
- Do Error Analysis yourself on 50–100 real user conversations. Find the most common failure modes.
- Tell a story with data. Present:
- the top failure modes you discovered,
- how often high-impact errors happen,
- surprising ways users interact with the product,
- bugs found and fixed, framed as “prevented production issues”.
- Frame evaluation as part of development, not optional testing.
- Keep a running log: error caught → what you learned → the fix → likely impact avoided. Share it weekly or monthly. “We caught 47 issues before users saw them” beats an abstract pitch.
- Don’t just show dashboards. Narrate what you’re finding so the team learns along with you.
- After each fix, show the error rate for that specific problem going down.
- Soon the team asks how you’re doing it. Let results, not methods, lead the conversation.
As in classic ML, outcomes are speculative and progress is bounded by iterating on experiments, so share learnings from each one (see Capability Funnel and Experiment Roadmaps).
Budget framing
- Evals are part of development, like debugging, not a line item.
- Expect 60–80% of development time on error analysis and evaluation, mostly looking at data.
- Cheap assertions are almost always worth it. Aligned LLM judges need a cost–benefit case (see When to Build an Automated Evaluator).
- Evals also speed up model upgrades: teams with evals can assess a new model in days rather than weeks (Anthropic).