Obsidian Metadata
| type | case-study |
| sources | Video - From Noob to Automated Evals in a Week (Teresa Torres) |
Case Study: Teresa Torres’s AI Interview Coach
A PM and educator with almost no recent coding experience went from “what’s an eval?” to a production beta in about four months, applying each course lesson to a real product.
| Aspect | What she did | Vault link |
|---|---|---|
| Product | Grades customer-interview transcripts on 4 rubric dimensions, with excerpt + tip feedback. Grew from 1 prompt to 5–9 LLM calls | – |
| Traces and first annotation UI | Airtable (a known tool, to limit how much was new at once) | Annotation Tools and Review Interfaces |
| Analysis | A Jupyter notebook with ChatGPT/Claude as a tutor, never running code she didn’t understand | Minimum Viable Eval Setup |
| Highest-harm failures | The coach suggesting leading questions and general questions | Error Analysis |
| Leading questions | LLM judge, iterating on input granularity | LLM-as-Judge |
| General questions | Keyword code assertion, very highly aligned. Spotting that took domain expertise | Code-Based Evals |
| Context engineering | Cached helper functions extract tips and questions, so judges see minimal input and iteration is cheap | Judge Context and Context Rot |
| Surprises | A results grid exposed a helper bug and a case where the eval was more right than her labels, which pushed her to sharpen the rubric | Criteria Drift |
| Big win | Repeated excerpts across dimensions went from 81% to 3% through a workflow change (pass used excerpts forward), not a prompt tweak. It needed a new eval just to detect the problem, then an A/B test | Multi-Step Workflow Evaluation |
| Process lesson | Error-analyze whole traces (what the customer sees) to prioritize. Per-dimension annotation only for targeted evals | Error Analysis |
| Now | Claude Code pair-programming, evals separated from the analysis notebook, some evals as production guardrails or sampled monitors, and many small vibe-coded annotation UIs | CI Evals vs Production Monitoring |
Why it’s a model example (per Hamel): she measured before fixing (81% → 3%), picked cheap evaluators where possible, worked like a data scientist investigating disagreements, and kept the domain expert (herself) at the center.