Obsidian Metadata
| type | source |
| subtype | video |
| url | https://youtu.be/N-qAOv_PNPc |
| speakers | Teresa Torres, Hamel Husain |
| duration | 1:10:21 |
Video: From Noob to Automated Evals in a Week (as a PM), Teresa Torres
Teresa (product-discovery coach, author of Continuous Discovery Habits) built an AI interview coach for her customer-interviewing course and learned evals by applying each course lesson to it. Full write-up: Teresa Torres Interview Coach.
The product
Students submit an interview transcript. The coach scores four rubric dimensions (story-based opening question, setting the scene, building the timeline, redirecting generalizations) as “keep practicing / getting it / great”, each with a transcript excerpt plus a tip. It started as one Claude project prompt and grew into a 5–9-call workflow.
Her path
- Traces in Airtable, a tool she already knew, to avoid taking on too much new learning at once. An Airtable interface served as her first annotation UI (trace, notes, failure modes, “reviewed” flag). Her first pass showed the coach was much worse than her vibe checks had suggested.
- Jupyter notebook (a suggestion from her husband) plus ChatGPT as a tutor, even though she’d never used Python. Rule: never run code she doesn’t understand, going line by line.
- First two evals targeted the most harmful failures for a teacher: the coach suggesting leading questions, and suggesting general questions (“tell me about your typical…”).
- Leading questions → an LLM judge, iterated over input strategies (all questions at once, one at a time, full response vs questions only).
- General questions → a code assertion on red-flag keywords (“usually”, “typically”, …). It was very highly aligned, which took domain expertise to realize. Hamel: an engineer would likely have built a judge.
- Helper functions pre-extract the coaching tips and questions, so each judge sees minimal context. They’re cached so iterating on a judge doesn’t rerun the expensive calls (context engineering plus cost and speed).
- A results grid (eval × trace) sized the problems (~20–25% for the two big ones). Digging in revealed a bug in a helper and a case where the eval was right and her human labels were wrong, because writing the eval forced a clearer rubric.
- A fast feedback loop: for every model, prompt, temperature, or workflow change, run the evals on dev for prod vs change and make a judgment call on mixed results.
Today
- VS Code plus Claude Code as a pair programmer (not vibe coding: plans in planning mode, small diffs). An eval directory, experiments, an A/B runner, and evals separated from the analysis notebook. Some evals run as production guardrails or on a sample of traces.
- The notebook has widgets: pick a transcript and see what the helpers extracted plus every eval verdict.
- About half a dozen vibe-coded annotation UIs, one per annotation shape (whole trace vs single question), with progress bars and hotkeys. Sometimes she labels by having Claude Code show one item at a time in the terminal (~100 in 10 minutes).
- Shipped as a beta inside Vistaly (SOC 2 compliant), prompted partly because students started uploading real customer interviews.
Standout story
Excerpts repeated across dimensions with contradictory feedback showed up in 81% of transcripts. She first spent a day and a half on a question classifier, which went badly. The real fix was a workflow change: pass already-used excerpts to later calls. That brought it to 3%. She first had to build an eval that could even detect the (non-identical) repeats, then A/B-test a workflow change. All of it took three days.
Lessons
- Splitting one prompt into per-dimension calls improved quality but introduced new errors (each dimension “greedy” about its own topic).
- Error-analyze the whole trace (what the customer sees) to prioritize, and use per-dimension annotation only when working on a specific eval.
- Keep adding small annotation-speed features (e.g., pills for already-used failure modes).
- Discovery-to-evals analogies: start with outcomes, measure before changing, and bring the customer closer to the problem. She wants finer-grained user feedback UX (flag a specific excerpt’s tip as wrong) and to derive synthetic-data dimensions from customer research.
- Take breaks. The simpler solution often shows up away from the keyboard.
→ Teresa Torres Interview Coach · Code-Based Evals · Judge Context and Context Rot