Obsidian Metadata
| type | concept |
| sources | Source - A Field Guide to Rapidly Improving AI Products, Source - AI Evals FAQ |
Maintaining Trust in Evals
A common pattern: teams build eval systems, then gradually lose faith in them. Sometimes the metrics don’t match what they see in production; other times the evals get too complex to interpret. Either way, they fall back to gut feel and anecdotes.
How successful teams keep trust
- Accept Criteria Drift. Treat criteria as living documents. Reconcile differing stakeholder criteria instead of imposing one blindly.
- Favor binary decisions over arbitrary scales (Binary Pass-Fail vs Likert Scales). A 10% increase in passing outputs means something; +0.5 on a 5-point scale needs interpretation.
- Pair binary judgments with detailed critiques so nuance lives in the critique (Critique Shadowing).
- Measure alignment between automated evals and humans (Judge Validation). People tend to over-trust AI. The MIT “GPT-4 aces the EECS exam” preprint was debunked within hours partly because GPT-4 graded itself. LLMs are also swayed by option ordering and formatting.
Scaling without losing trust
- Start with high human involvement: experts review a significant share of outputs.
- Study alignment patterns: where automated evals agree with humans and where they diverge.
- Strategic sampling: focus human attention on the most informative cases and where alignment is weakest.
- Regular calibration: keep comparing automated and human judgments.
You can’t eliminate humans completely. The judge has to be aligned to something, usually a human. As the system improves, the human effort needed goes down naturally.
Symptoms and fixes (reading guide)
| Symptom | Go to |
|---|---|
| Judge scores don’t match your judgment | Judge Validation |
| Tests pass but users still hit problems | Error Analysis, Eval Maintenance and Cadence (saturated or stale set) |
| Dashboard of 1–5 scores nobody acts on | Generic Metrics, Combining Evals into a Single Metric |
| Reported % looks off vs reality | Judge Bias Correction |