Obsidian Metadata
What LLMs Can (and Can’t) Automate in Evals
“Use LLMs to scale what you’ve learned, not to avoid looking at data.”
✅ Where LLMs help (with oversight)
- First-pass axial coding. After you’ve open-coded 30–50 traces, have an LLM propose groupings. Review and refine them.
- Mapping annotations to failure modes. “Given this annotation: [open_annotation] and these failure modes: [list], which apply?”
- Suggesting prompt improvements from recurring problems. Review before adopting.
- Analyzing annotation data. LLMs or AI notebooks (Julius, Hex) find patterns like “lag reports ×3 during peak hours” or “slow responses mostly on mobile”.
- Searching for more instances of a failure mode you’ve defined (active learning; see Error Discovery Skill Workflow).
- Generating synthetic inputs, writing boilerplate and plumbing, building review UIs, and brainstorming assertions.
- Optimizing a judge prompt after the task and labels are defined (see Prompt Optimization).
❌ Don’t outsource to an LLM
- Initial open coding. This is how you discover new failure types and build intuition. Example: users keep saying “lag”, but the LLM lumps it under generic “performance issues” instead of a “latency” category, and you miss the recurring complaint.
- Validating failure taxonomies. “App crashes after login” and “login takes too long” are different problems.
- Ground-truth labeling for judge validation. Hand-validate every label.
- Root-cause analysis of subtle, workflow-specific patterns (e.g., bugs only when users paste from Excel).
Evidence: “Do Automated Evals Work?” (Parlance Labs, 2026)
On 100 real apartment-leasing traces with 39 human-labeled failures:
| System | Recall | Precision | Discoveries | False positives |
|---|---|---|---|---|
| Braintrust Loop | 87.2% | 79.1% | 20 | 9 |
| Arize AX Alyx | 74.4% | 91.0% | 19 | 3 |
| LangSmith (chat agent) | 79.5% | 77.5% | 20 | 9 |
| Codex (GPT-5.5 High) | 84.6% | 82.8% | 20 | 11 |
| Factory Droid (GPT-5.5 High) | 84.6% | 83.3% | 17 | 10 |
| Claude Code (Opus 4.8) | 79.5% | 77.4% | 17 | 14 |
- ✅ All caught obvious failures (unsupported scheduling claims, repeated confirmation requests, broken follow-up promises, contradicting tool output), and each found 17–20 minor issues humans missed.
- ❌ Every system missed failures that “look correct” without product context: giving up on sales objections, Markdown in SMS, the voice agent interrupting callers, missed required handoffs.
- ❌ No tool interviewed the user about requirements. They all treated error analysis as one-shot.
- Recommended instead: annotate yourself and let the AI learn from your annotations continuously.
- “If a tool could find and fix every issue on its own, it would do the same for your competitors.”
Beware “stacked abstractions”
Be very skeptical of features where an AI writes the rubric and then immediately scores outputs against it. That hides flaws behind a high score (see Eval Tooling and Vendors).
Related
Error Analysis · Annotation Outsourcing · Prompt Optimization