Obsidian Metadata
| type | source |
| url | https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents |
| authors | Mikaela Grace, Jeremy Hadfield, Rodrigo Olivares, Jiri De Jonghe (Anthropic) |
| published | 2026-01-09 |
Source: Demystifying Evals for AI Agents (Anthropic)
URL: https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents
Vocabulary: task · trial · grader (checks) · transcript/trace/trajectory · outcome (final environment state) · evaluation harness · agent harness/scaffold · suite.
Why evals: avoid reactive loops and “flying blind”; specify success early; adopt new models in days; baselines for latency, tokens, cost; a high-bandwidth product↔research channel. Claude Code evals went from concision and file edits to over-engineering. Descript: “don’t break things, do what I asked, do it well”. Bolt: static analysis plus browser agents plus judges.
Graders: code (string/regex, fail-to-pass tests, static analysis, outcome verification, tool-call params, transcript stats) · model (rubric, NL assertions, pairwise, reference-based, multi-judge) · human (SME, crowd, spot checks, A/B, IAA). Scoring can be weighted, binary all-must-pass, or hybrid.
Capability vs Regression Evals; per-agent-type advice (coding, conversational, research, computer use); pass@k and pass^k.
Roadmap (steps 0–8): start early with 20–50 tasks from real failures · begin with manual checks and bug tracker · unambiguous tasks with reference solutions (0% pass@100 means a broken task) · balanced sets (should/shouldn’t) · stable, isolated environments · thoughtful graders (outcome not path, partial credit, calibrate judges, “Unknown” escape, isolated per-dimension judges, hack-resistant; CORE-Bench 42% → 95% after grading fixes) · read transcripts · watch for saturation · long-term ownership (evals team for infrastructure, product teams contribute tasks) · “eval-driven development”.
Holistic view: automated evals + production monitoring + A/B + user feedback + transcript review + human studies, the Swiss-cheese model.
Frameworks: Harbor, Braintrust (autoevals), LangSmith, Langfuse, Arize Phoenix/AX. “Only as good as the eval tasks you run through them.”
→ Agentic Workflow Evaluation, Coding Agent Evaluation, Eval-Driven Development (compare viewpoints)