Obsidian Metadata
| type | concept |
| sources | Source - AI Evals FAQ, Source - Evals Flashcards, Source - Demystifying Evals for AI Agents |
CI Evals vs Production Monitoring
CI evals protect against known regressions before deployment. Online monitoring finds failures in production traffic and estimates how often they occur.
| CI / CD | Online monitoring | Guardrails | |
|---|---|---|---|
| Goal | Prevent regressions | Discover new failures and track performance | Enforce safety, block high-impact errors |
| When | Pre-merge (pull request) | Async (post-response) | Synchronous (pre-response) |
| How | Unit tests / assertions, LLM judge | Assertions, LLM judge, A/B testing | Assertions, small classifiers |
| Data | Curated test cases | Sampled production traffic | 100% of live traffic |
| On failure | Block merge | Trigger an alert | Block response, retry, or fall back |
Evals in CI
- Small, purpose-built datasets (often ~100+ examples) covering core features, regression tests for past bugs, and known edge cases.
- They run often, so cost per test matters. That’s why the set is curated. Favor assertions and deterministic checks over LLM judges.
- Orchestrate with whatever has the least friction in your stack (GitHub Actions, GitLab pipelines). Store metrics and prompt/test versions outside CI for tracking over time. Rechat used Metabase on top of existing analytics.
- Pass rate is a product decision. You don’t need 100%.
Online monitoring
- Sample live traces and run evaluators asynchronously.
- Production usually has no reference outputs, so you rely more on reference-free evaluators like LLM judges (more expensive).
- Track confidence intervals. If the lower bound crosses your threshold, investigate. Correct judge rates for known judge error (see Judge Bias Correction).
- Example: Nova Escola runs a daily eval suite against 2% of production traffic to catch regressions early.
Connect the two
When monitoring (plus Error Analysis) reveals a new failure pattern, add representative examples to the CI dataset. That prevents regressions on newly discovered issues.
Anthropic’s “Swiss cheese” view
No single layer catches everything. A full picture combines:
| Method | Strength | Weakness |
|---|---|---|
| Automated evals | Fast, reproducible, no user impact, run every commit | Up-front cost, maintenance, false confidence if unrepresentative |
| Production monitoring | Real behavior at scale, ground truth | Reactive, noisy, needs instrumentation |
| A/B testing | Measures real outcomes, controls confounds | Slow, needs traffic, weak on “why” |
| User feedback | Surfaces the unanticipated | Sparse, self-selected, skews severe |
| Manual transcript review | Builds intuition, catches subtle issues | Doesn’t scale, inconsistent coverage |
| Systematic human studies | Gold-standard, handles subjectivity | Expensive, slow |
Automated evals come first (pre-launch and CI), monitoring post-launch, A/B tests for significant changes, and feedback plus transcript review are ongoing (sample transcripts to read weekly).
Related
Guardrails vs Evaluators · Three Levels of Evaluation · Capability vs Regression Evals · Eval Maintenance and Cadence