Obsidian Metadata

type	source
url	https://hamel.dev/notes/llm/evals/flashcards/
authors	Hamel Husain, Shreya Shankar
published	2025-11-30

Source: Evals Flashcards

URL: https://hamel.dev/notes/llm/evals/flashcards/ (downloaded into _attachments/)

#	Card	Key idea	Vault note
0	Cover / TOC	–	–
1	Error analysis	Open → axial coding → saturation	Error Analysis
2	When to write an eval	Observed? → iteration needed? → value	When to Build an Automated Evaluator
3	Don’t use generic eval metrics	–	Generic Metrics
4	AI eval mistakes	Look at data first · measure the judge vs human labels · 100% means saturated or too easy	Eval Playbook
5	Types of automated evals	Code assertions (use whenever possible) · LLM judge (slower, costly, needs alignment) · guardrails (fast, low FP, code or small classifiers)	LLM-as-Judge
6	Don’t use Likert scales	–	Binary Pass-Fail vs Likert Scales
7	How to trust an LLM judge	Train ~20 / dev ~40 / test ~40; TPR and TNR > 90%; no accuracy	Judge Validation
8	Sample traces	Random, clustering, data analysis, classification, feedback	Trace Sampling Strategies
9	Synthetic data	Dimensions → tuples → queries	Synthetic Data
10	Traces for evals	First upstream error · minimal repro · N-1 · perturb · simulate	Traces
11	Transition matrix	From-state × failure-state counts	Transition Failure Matrix
12	Deploy evals	CI/CD vs online monitoring vs guardrails	CI Evals vs Production Monitoring

All cards