Obsidian Metadata
| type | reading-path |
| source | https://hamel.dev/notes/llm/evals/start/getting-started/index.html |
Reading Path: I Don’t Know What to Test
“I’m building an AI product, but I haven’t figured out which failures to measure or what good performance looks like.”
- Source - A Field Guide to Rapidly Improving AI Products
- Video: Intro to Error Analysis With Just Spreadsheets, or read Error Analysis
- Error Discovery Skill Workflow (plus Shreya’s How to Automate AI Evals (Correctly))
- Template: Error Analysis Worksheet
Sampling and generating data: Trace Sampling Strategies · How Many Examples Do You Need · Synthetic Data
Reviewing and labeling: Evals with Sensitive Data · Diverse Query Evaluation · Reference Answers and Rubrics · Benevolent Dictator · Team Collaboration on Evals
Deciding what to test: LLM-as-Judge · Coding Agent Evaluation · When to Build an Automated Evaluator · Eval-Driven Development