Obsidian Metadata

type	concept
sources	Source - AI Evals FAQ, Source - Demystifying Evals for AI Agents

Eval Maintenance and Cadence

How often to re-run error analysis

• Re-run on significant changes: new features, prompt updates, model switches, major bug fixes.
• Goal: 100+ fresh traces per review cycle. Typical cycles are every 2–4 weeks.
• Between cycles, review 10–20 traces weekly, focused on outliers: unusually long conversations, many retries, traces flagged by monitoring.
• New systems: weekly analysis until failure patterns stabilize. Mature systems: maybe monthly, unless usage changes.
• Always analyze after incidents, spikes in user complaints, or metric drift. Scaling usage introduces new edge cases.
• After the first couple of iterations, lean toward error-focused sampling (search for more examples of a found error), but always include some random sampling.

When the “gold” dataset goes stale

• Eval datasets naturally go stale as the product and users change. Use regular Error Analysis to find new problems and update examples and reference answers.
• Like unit tests, evals catch problems that come back after a fix, but evals cost much more to maintain and run. Weigh each eval’s cost against the value of its signal.
• If everything keeps passing, the eval is no longer useful. Retire it or run it less often. (Anthropic calls this eval saturation: an eval at 100% tracks regressions but gives no signal for improvement, and near saturation large capability gains show up as small score increases. Qodo, for example, underrated Opus 4.5 because its one-shot evals were saturated.)
• Scores won’t be comparable with older ones as the set changes. That’s fine; the challenges should evolve. Track long-term progress with product metrics (churn, active users, revenue).

Maintaining evaluators

• Re-validate judges after changing prompts or models, or when inputs shift. Pin model versions.
• Keep checking human–judge agreement periodically (see Maintaining Trust in Evals).
• Re-audit after significant pipeline changes (see Eval Audit Checklist).

Ownership (Anthropic)

An eval suite is a living artifact that needs clear owners. What worked for them: a dedicated evals team owns core infrastructure, while domain experts and product teams contribute most tasks and run evals themselves. Let PMs, CSMs, and salespeople contribute eval tasks as PRs, and actively enable them. Owning evals should be as routine as maintaining unit tests.

Related

Error Analysis · Capability vs Regression Evals · CI Evals vs Production Monitoring