Obsidian Metadata
| type | source |
| url | https://hamel.dev/blog/posts/llm-judge/ |
| authors | Hamel Husain |
| published | 2024-10-29 |
Source: Using LLM-as-a-Judge for Evaluation: A Complete Guide (Part 2)
URL: https://hamel.dev/blog/posts/llm-judge/ · Hamel Husain (lessons from 30+ companies)
Problem: teams drown in too many metrics, arbitrary 1–5 scales, ignored domain experts, and unvalidated metrics.
Solution: Critique Shadowing. (1) find the principal domain expert → (2) diverse dataset (features × scenarios × personas; real + synthetic) → (3) expert pass/fail + detailed critiques → (4) fix errors → (5) build the judge iteratively with critiques as few-shot → (6) error analysis by dimension and root cause → (7) specialized judges if needed.
Notable details:
- The Honeycomb judge prompt: >90% agreement in 3 iterations (Honeycomb Query Assistant).
- Examples: 30 to discover failure modes; ~100 per failure mode to validate a judge; < 60 gives CIs that are too wide.
- Validation: train 10–20% / dev 40–45% / test 40–45%; TPR/TNR; a judge that always says Pass on 5%-failure data has 95% agreement (Judge Validation).
- Judge-prompt mistakes: no critiques, terse critiques, missing external context, non-diverse examples.
- FAQ: don’t fine-tune judges (fine-tune the product model instead); off-the-shelf judges cause more confusion than value; “do the simplest thing that works”; you can’t fully remove humans.
- “It’s not the judge that created value, after all.” The value is in looking at data.
Resources cited: Who Validates the Validators? · Eugene Yan’s AlignEval and LLM-evaluators survey · Databricks “Grading Notes” · OpenAI cookbook custom judge (Braintrust) · Arize self-improving evals · Dosu/LangSmith continual in-context learning · applied-llms.org.