Video - How to Automate AI Evals Correctly

Obsidian Metadata

type	source
subtype	video
url	https://youtu.be/tqUDjc1HzO4
speakers	Shreya Shankar, Hamel Husain
duration	27:19

Video: How to Automate AI Evals (Correctly), Shreya Shankar

This is part 1 of the AI Product Engineering series. Shreya argues that you can’t farm evals out to a coding agent end-to-end, but AI can help you apply your judgment faster. She demos her error-discovery skill live on AI-generated essays, looking for signs of “AI slop”.

Framework: analyze → measure → improve (iterative)

• Analyze (error analysis) is the hardest step because “what’s bad” is know-it-when-you-see-it.
• Measure the prevalence of each failure mode. A Pareto pattern is common: ~80% of issues come from ~20% of the modes.
• Improve: prompt instructions, model swaps, fine-tuning.
• AI is weakest early in this loop (taste-heavy analysis) and strongest later (measurement, hill-climbing, prompt optimization).

Mistake 1: “Claude, check out my traces and evaluate my app”

• The agent will find some issues, but it misses everything that lives in your head and isn’t written into the traces or prompt.
• It doesn’t find the highest-priority issues and may invent a vague “biggest issue”.
• No reuse: every run starts over. You want to persist intermediates (failure-mode definitions and taxonomy) in files or code so later runs build on them.
• Fix: keep your workflow and use AI for specific parts, like building the review UI and organizing annotations.

The skill, as shown in the demo

1. Understand the data (fields, structure).
2. Design a visual encoding (e.g., color by role).
3. Build a review app with three views: trace viewer, map (clusters), and progress (failure-mode treemap).
4. Cluster and sample representative items. Clustering is a guess, so don’t assume it’s perfect. Embeddings aren’t always the right representation, so the skill lets the agent pick features.
5. Human-in-the-loop: the reviewer highlights text and writes notes in place. The agent monitors annotations and builds the taxonomy in the background, aiming for breadth (cover all modes) and depth (many examples per mode).

• It took ~5 minutes to generate. The UIs are rough, and she keeps folding UI feedback back into the skill.

Mistake 2: only looking at each trace once

• Re-reading already-reviewed traces surfaces new failure modes, because more traces in your head spark new patterns. Example: she noticed late that nearly every essay overused the word “matters”, including ones she’d already reviewed.
• Think of it as an outer loop over data and an inner loop of hypotheses within each data point.
• Have the agent apply every new annotation to previously reviewed and unreviewed traces, then accept or reject its suggestions.
• She doesn’t let the agent add new failure modes to the taxonomy. Validating the agent’s taste felt worse than stating her own and letting the agent scale it.

Mistake 3: a uniform accuracy bar for every AI app

• Internal tools (Slack summaries) don’t need heavy eval investment. Customer-facing features do.
• Use a coding agent to brainstorm worst-case scenarios from traces, the app description, or the codebase (e.g., sabotaged citations, leaking a journalist’s private notes). This helps decide which evals and guardrails matter.

Other points

• In their benchmark, general coding agents (Claude Code, Codex) were often more exhaustive than dedicated vendor discovery tools (see Source - Do Automated Evals Work). Hamel: vendor tools are “someone’s prompt” plus a harness, and you can close the gap by adding your own domain context.
• No tool yet elicits your taste “20-questions style”. There will always be a human element.

→ Related: Error Discovery Skill Workflow · Error Analysis · What LLMs Can and Can't Automate in Evals