Obsidian Metadata
Binary Pass/Fail vs Likert Scales
Default: every eval returns Pass or Fail
Put the nuance in a written critique next to the verdict, not in a number (see Critique Shadowing).
Why not 1–5 ratings?
- Adjacent points are subjective. What separates a 3 from a 4? Different annotators (and judges) read the scale differently, and the noise flows into your metric.
- Central tendency. Annotators fall back on middle values to avoid hard decisions.
- Statistics. Detecting a difference on a scale needs larger samples.
- Not actionable. People don’t know what to do with a 3.2. You need to be able to say “this failed because…“.
- Scores don’t correlate with what matters. Hamel: every time he analyzed domain-expert judgments, they didn’t correlate with 1–5 “quality” metrics.
- You end up drawing a line anyway. Businesses using 1–5 still ask “where’s good enough?”, which is a binary decision.
- Binary is faster during Error Analysis. No arguing over 3 vs 4.
- Binary is calibratable. You can measure inter-annotator agreement and judge TPR/TNR against a clear boundary (see Judge Validation).
“But I need to track gradual improvement”
Decompose instead of scaling. Rather than rating factual accuracy 1–5, track “4 of 5 expected facts included” as separate binary checks. For severity, use several binary judges (e.g., “factually wrong” and “dangerously wrong”) rather than one ordinal scale.
Common objections (and answers)
| Objection | Response |
|---|---|
| ”The business says these 8 dimensions matter, so we must score all 8 on 1–5.” | Anyone asking for 8 things on 1–5 “doesn’t know what they’re looking for; they’re just guessing.” Let the domain expert make pass/fail calls with critiques to find out what actually matters. Stand your ground. |
| ”We need to explain why it passed or failed.” | That’s what the critique is for. |
| ”My problem is complex.” | Starting simple forces focus on what matters. Add complexity later if needed. |
Numeric labels are advanced and usually unnecessary. Start binary to learn what “bad” looks like.
Exceptions
- Filtering synthetic data for realism: 1–5 LLM ratings are fine there, because the goal is fuzzy ranking for dataset curation, not measuring failure rates (from the
evaluate-ragandgenerate-synthetic-dataskills). - Anthropic’s agent-eval guidance allows partial credit across multi-component tasks. Hamel’s version of the same idea: decompose into multiple binary checks (see Combining Evals into a Single Metric).
Related
LLM-as-Judge · Generic Metrics · Combining Evals into a Single Metric