Obsidian Metadata
| type | concept |
| sources | Source - AI Evals FAQ |
Combining Evals into a Single Metric
Each eval returns a binary outcome, and you’ll have many. Stakeholders often want one number.
Recommended: the “pass-all” rate
An example passes only if it passes every check. 80 of 100 examples passing all checks = 80% pass-all rate. Build the dashboard so you can drill down from pass-all to per-check pass rates and see what’s driving failures.
Middle grounds
- Group checks by theme and report a pass-all rate per group. Example: Nurture Boss’s themes were conversation flow, handoffs to humans, and rescheduling (see Nurture Boss).
- Group by severity: one pass-all rate for release-blocking checks and another for tolerable issues. Useful for gating releases.
Weighted scores: discouraged
You can weight checks by importance, but complicated weighted composites have the same problem as Likert scales: a score going from 3.2 to 3.7 feels good while telling you nothing about what improved for users. “Dashboards like this are usually performative and waste everyone’s time.”
Comparability over time
As the eval set changes, scores stop being directly comparable with older ones, and that’s OK. Evals are challenges to hill-climb, and they should change as the product evolves. For long-horizon progress, use product metrics (churn, active users, revenue) alongside evals (see Eval Maintenance and Cadence).
Related
Binary Pass-Fail vs Likert Scales · Judge Bias Correction · pass@k and pass^k