Obsidian Metadata
| type | source |
| url | https://parlance-labs.com/blog/posts/auto-evals/index.html |
| authors | Antaripa Saha, Hamel Husain |
| published | 2026-07-11 |
Source: Do Automated Evals Work? (Parlance Labs)
URL: https://parlance-labs.com/blog/posts/auto-evals/index.html
Experiment: 100 real Nurture Boss traces, 39 failures labeled by a domain expert, labels masked. Braintrust Loop, Arize Alyx, LangSmith (chat agent; Engine surfaced too few issues), Codex, Factory Droid, and Claude Code all got the same light prompt (surface recurring failures, cluster them, cite trace IDs, note acceptable/inconclusive traces, suggest evaluators).
Results: recall 74–87%, precision 77–91%, each found 17–20 valid issues humans missed, with 3–14 false positives. Loop had the highest recall; Alyx the highest precision (strong on ungrounded claims); LangSmith was best at citing exact evidence but inflated severity; coding agents were competitive on finding quality, while platforms won on workflow (linking, review, re-runs).
Every system missed failures that look correct in the trace but violate product goals: sales objections, Markdown in SMS, voice interruptions, missed required handoffs. No tool asked the user questions.
Recommendation: don’t run error analysis one-shot. Annotate yourself and let the agent learn continuously from your annotations (Shreya’s Monitor-tool workflow). “If a tool could find and fix every issue on its own, it would do the same for your competitors.”
→ What LLMs Can and Can't Automate in Evals, Error Discovery Skill Workflow