Obsidian Metadata
| type | reading-path |
| source | https://hamel.dev/notes/llm/evals/start/trust-your-evals/index.html |
Reading Path: I Don’t Trust My Eval Scores
“We have evals, but the scores don’t match our judgment, or tests pass while users still hit problems.”
- Source - Using LLM-as-a-Judge → Critique Shadowing
- Source - Do Automated Evals Work: why automated review still needs human judgment and product context
- Nova Escola (production example) · Source - The Revenge of the Data Scientist (common mistakes)
- Run the Eval Audit Checklist
Metrics and judge decisions: Judge Validation · Judge Bias Correction · Combining Evals into a Single Metric · LLM-as-Judge · Binary Pass-Fail vs Likert Scales · Generic Metrics · Judge Context and Context Rot
Labels and eval data: Benevolent Dictator · How Many Examples Do You Need · Eval Maintenance and Cadence
Production and changes: CI Evals vs Production Monitoring · Trace Sampling Strategies · Guardrails vs Evaluators · Prompt Management · Maintaining Trust in Evals