Obsidian Metadata

type	template
sources	Source - Evals Skills for Coding Agents

Eval Audit Checklist

Adapted from the eval-audit skill. Inspect the actual artifacts (traces, evaluator configs, judge prompts, labeled data, dashboards), not just the docs. Report findings ordered by impact.

1. Error analysis

• Has systematic error analysis been done on real (or synthetic) traces? Are there labeled traces, failure definitions, review notes?
• Were failure categories observed (“missing query constraints”, “wrong client tone”) or brainstormed (“hallucination score”, “toxicity”, “coherence”)?
• Fix: Error Analysis / error-discovery (or Synthetic Data first if there are no traces).

2. Evaluator design

• Are evaluators binary Pass/Fail (no Likert, letter grades, or thresholdless scores)?
• Does each judge target one specific failure mode (not “is this helpful?”)?
• Is code used for objective checks (format, schema, keywords, constraints)?
• Are similarity metrics (ROUGE, BERTScore, cosine) not the primary generation evaluator?
• Fix: Binary Pass-Fail vs Likert Scales, LLM-as-Judge, Code-Based Evals, Generic Metrics.

3. Judge validation

• Are judges validated against human labels (confusion matrix, TPR/TNR)?
• Is alignment reported as TPR/TNR rather than accuracy, % agreement, or Kappa?
• Is there a proper train/dev/test split, with no few-shot leakage from dev/test?
• Are production rates bias-corrected with CIs?
• Fix: Judge Validation, Judge Bias Correction.

4. Human review process

• Are domain experts labeling (not outsourced generalists)?
• Do reviewers see full traces (not just final outputs)?
• Is data rendered in native format (not raw JSON or spreadsheet cells)?
• Fix: Benevolent Dictator, Annotation Tools and Review Interfaces.

5. Labeled data

• ~100 traces reviewed for error analysis (saturation)?
• ~50 Pass + ~50 Fail per judge for validation?
• Sampling includes random plus clustering / outliers / classifier flags / feedback?
• Fix: Trace Sampling Strategies, How Many Examples Do You Need.

6. Pipeline hygiene

• Is error analysis re-run after significant changes (model, prompt, feature, incident)?
• Are judges re-validated and datasets refreshed? Is the judge model version pinned?
• Are saturated (always-passing) evals retired or hardened?
• Fix: Eval Maintenance and Cadence.

If there’s no eval infrastructure

1. error-discovery on a sample of real traces.
2. No prod data? generate-synthetic-data → run through the pipeline → error discovery.
3. Don’t build evaluators, judges, or dashboards before error analysis.

Finding format

### [Problem Title]
**Status:** Problem exists / OK / Cannot determine
[1–2 sentences on the specific problem found]
**Fix:** [Concrete action + link]