Obsidian Metadata
| type | source |
| subtype | video |
| url | https://youtu.be/qH1dZ8JLLdU |
| speakers | Hamel Husain, Shreya Shankar |
| duration | 41:36 |
Video: Intro to Error Analysis: Creating Custom Data Annotation Apps
A lightning lesson. Shreya vibe-codes an annotation app live and open-codes outputs from a toy AI recruiter that writes personalized outreach emails from Kaggle résumés (Llama 3 70B).
Motivation
- People jump to frameworks and tools and rarely look at data. Greg Brockman’s line about manual data inspection having the best value-to-prestige ratio is cited. Hamel goes further and calls it the highest-ROI activity in AI products.
- Without looking at data you play whack-a-mole: fix one thing, break another.
Error analysis as stages
- Look at data and write free-form notes per output (“write your vibes down”).
- Group the notes into categories (an LLM can help).
- Build a pivot table: which outputs fall into which category, with counts.
- Improve (prompts, fixes, maybe new metrics), then repeat. Teams often do several passes before deploying.
Building the tool (prompt design tips)
- One input/output per page: résumé on the left, email in the center, open-ended feedback box on the right.
- Left/right hotkeys, a persistent table of all notes so far, and a save/export button. Flask + HTML was enough, with no React.
- It came together in ~10 minutes. Later additions they wanted: search within the résumé for claims the AI made, and show the job post too.
- Why not a spreadsheet or Label Studio? You’d have to mentally render an email stored in a cell. A custom UI can render emails as emails, code as code, and plots as plots. Use whatever framework you (and the AI) know.
Failure notes they wrote
Wrong greeting (no recipient name), leftover placeholder in the sign-off, role confusion (email written as if from the candidate), irrelevant or inferred skills that aren’t on the résumé, “although your background…” phrasing that reads like a rejection, overused openers like “impressed by your experience”, verbosity, repeated budget mention. Shreya also noticed her own criteria drift (an irrelevant hobby might be fine as a PS).
Coaching points
- Don’t root-cause while annotating. Record what went wrong, not why.
- Go slowly on the first outputs. Minutes per item is fine.
- Note the single smoking-gun problem. You don’t need every nit.
- Don’t let an LLM do the first pass. You’ll anchor on its opinion. Read at least ~10 yourself first.
- Early on, focus on critical feedback. Affirmative feedback matters later for judges, or to preserve behaviors you didn’t specify.
- Use voice dictation (MacWhisper, Wispr Flow) to cut the typing strain.
- How many? Start with 20 (a “Jedi mind trick” to get started). Shreya does ~30. Stop when 5–10 more reveal nothing new.
- For axial coding, paste notes into an LLM and say they’re “open codes” in order (later notes reflect updated thinking). Ask for categories, then classify notes into them.
- If you see failures everywhere in 20 examples, go back to prompt engineering before building judges.
→ Error Analysis · Annotation Tools and Review Interfaces · Criteria Drift