Obsidian Metadata

type	source
subtype	video
url	https://youtu.be/mF4CaijvJos
speakers	Lucas Machado Rocha (Nova Escola), Hamel Husain
duration	33:39

Video: How To Build AI Evals: Nova Escola end-to-end

Lucas (a former course student) walks backward from where Nova Escola is now to how it got there. This extends Nova Escola.

Where they ended up

Pedagogical (non-engineering) staff run evals through Claude Cowork connected to their eval project folder (history, CLAUDE.md, memory) and to Hamel’s eval skills. Engineers use Claude Code.
Example session: pick the next failure mode (“the activity is unfocused, touching everything and nothing”). Claude proposes hypotheses from judge false positives and drafts a new judge prompt using write-judge-prompt. The skill then caught that training examples had leaked into the dev split. After fixing that, validate-evaluator showed the new prompt was actually worse, which is still useful to know.
They distribute skills, plugins, and connectors company-wide through an internal plugin marketplace, including the evals skills. They use write-judge-prompt, validate-evaluator, and build-review-interface, but do error analysis themselves and have enough real data that they don’t need synthetic generation.
Langfuse is the observability store: traces, datasets, prompts. Its MCP gives natural-language access. Before the MCP existed, they wrapped the API in a skill. They don’t use Langfuse to calibrate judges, because tools tend to assume you’re already calibrated and just hand you a score.
The product lets staff annotate chatbot answers (thumbs plus a note) and syncs that back to Langfuse.
Annotations must include reasoning. Without it, AI can’t tell why a case is a false positive or false negative.

Technical/data: getting full traces, turns, tool calls, retrievals, context. Fix: observability first.
Methodology: open and axial coding, judge building, splits, balanced calibration data. “There’s a trap hiding in each one.” Fix: the course, plus skills that package the method.
Judgment (the one that matters): is this lesson plan engaging, does it build the right knowledge? The first two barriers kept getting in the way of this one.

A huge manual spreadsheet of rubric criteria rather than observed failure modes, and a lot of labeling on things that weren’t problems (e.g., titles). Fixing that saved resources.
The “two learning goals instead of one” issue was a prompt fix, not an eval.
Two annotators disagreed worse than a coin flip, so they rewrote the rubric with the pedagogy team and relabeled. Each round took 1–2 months. They’re still weighing a benevolent dictator against multiple experts, since education is nuanced. Hamel pushes hard for finding the one person who knows what good looks like.

Four judges calibrated to production-worthy TPR/TNR. Some criteria still can’t get a trustworthy judge.
Two roles: (1) a CI gate on every PR (is the new bot version better or worse on each criterion?), and (2) daily batch monitoring of sampled production lesson plans.
The curriculum-alignment judge shows the most trouble and is the next hill-climbing target, because if alignment is wrong, everything else falls apart.
Evals also exposed coverage gaps. Computational-thinking lesson plans broke the judges, which hadn’t been calibrated on that subject.
Evals sped up a refactor to Anthropic managed agents because Claude “knows what good looks like” from the eval set.

Sampling: 2% daily was too little coverage per grade and subject, and the scores were erratic. He raised the sample until results stabilized.
Batch mode keeps judge cost down.
The judge model is pinned to the one used for calibration (a strong model), so the measured TPR/TNR still hold. The generator is a smaller model.
Impact: the early bot invented nonexistent curriculum codes that teachers trusted, which was a brand risk. Evals directed a small team (29 people, 6 in tech) to fix it. The failure modes that matter change over time.
Hamel: the value is mostly in the journey of building evals. If you did it right, production dashboards shouldn’t surprise you. Customize skills. Off the shelf they’re “somebody else’s prompt”.