Obsidian Metadata
| type | case-study |
| sources | Source - Case Study Putting Evals Into Production |
Case Study: Nova Escola Lesson Planner
Context: Brazilian nonprofit education platform (~1M monthly users, 200k teachers). Lucas Machado Rocha used evals to optimize a lesson planner.
Mistakes (and lessons)
- Wrote a rubric before doing error analysis. Most criteria never showed up as failures, so the team labeled data it didn’t need (see Reference Answers and Rubrics).
- Built an eval for a problem that needed a simple fix. The assistant sometimes emitted two learning goals instead of one. Changing the prompt stopped it (see When to Build an Automated Evaluator).
- Two annotators agreed less often than chance. “Good” wasn’t defined clearly enough, so they rewrote the rubric with pedagogical experts and relabeled (see Benevolent Dictator).
What worked
- After error analysis identified the failure modes, the team used the evals skills to automate writing judges and validating them against human labels.
- A daily eval suite on 2% of production traffic catches regressions early (see CI Evals vs Production Monitoring).
More from Lucas’s talk
Details from Video - How To Build AI Evals (Nova Escola):
- Non-engineers run evals through Claude Cowork plus the eval skills over a project folder. Langfuse is the trace, dataset, and prompt store (via MCP). The skills are distributed through an internal plugin marketplace.
- 4 judges are calibrated and used both as a CI gate on PRs and in daily batch monitoring. The curriculum-alignment judge is the weakest and the next target.
- 2% sampling was too erratic per grade and subject, so they raised it until scores stabilized. The judge model is pinned to the calibration model.
- The skill caught train examples leaking into dev. After the fix, the “improved” judge prompt turned out to be worse.
- The early bot invented nonexistent curriculum codes that teachers trusted, a brand risk that evals helped eliminate.