Error Discovery Skill Workflow

Obsidian Metadata

type	concept
sources	Source - Evals Skills for Coding Agents

Error Discovery Skill Workflow (human-in-the-loop, agent-assisted)

The error-discovery skill from ai-evals-course/evals-skills encodes the current recommended way to use a coding agent for Error Analysis without removing the human. Install:

npx skills add https://github.com/ai-evals-course/evals-skills
# single skill: --skill error-discovery

Prompt: “Can you help me do error analysis on traces.jsonl?”

Phases

1. Understand the domain and data. Read 5–10 records. Inventory fields (primary content vs metadata, varying vs constant). Identify the structure: single text, input/output pair, multi-turn trace (roles, tool call/result pairs, thinking blocks), code, structured output, composite. Identify dimensions of variation.
2. Design the visual encoding (Gestalt principles): hue = role or category; opacity reduced only for truly redundant content (identical boilerplate), never muted by role; spacing = hierarchy (4–8 / 16–24 / 32–48 px); typography = content type (monospace code, italic thinking); containers group a tool call with its result; header-only outlier badges (“longer than 95%”).
3. Build the review app. Python stdlib http.server plus a single HTML file; data in error_discovery_data/ (samples.json, annotations.json, graph.json, patterns.json, suggestions.json). Three views: Content (inline highlight → margin notes, free-text only, agent suggestions shown with dashed style and accept/dismiss), Map (PCA/UMAP scatter with cluster hulls), Progress (failure-mode treemap plus a suggestion queue with bulk accept/dismiss).
4. Cluster and select the initial sample. KMeans with 6–10 clusters; 15–25 items; 60–70% cluster representatives + 30–40% random; diversity over prevalence.
5. Interactive review loop:
   • Monitor annotations.json (poll every ~2s, e.g. with Claude Code’s Monitor tool).
   • Categorize each note into a running taxonomy {mode: {description, count, example_ids, quotes}} and push it to the UI.
   • Depth: after the human has annotated ~5 distinct records, spawn one subagent per failure mode to scan all records (including already-reviewed ones, to catch misses caused by Criteria Drift) and push suggestions.
   • Breadth: after each batch, sample uncovered clusters plus random picks.
   • Encourage re-review: “You’ve found 3 new modes since records 0–5; worth a second pass.”
   • Report convergence and suggest stopping when discovery slows (Theoretical Saturation).

Key principles

1. The human notices; the agent organizes. Don’t make the human categorize.
2. The agent proposes coverage.
3. Live feedback loop.
4. Always include random samples.
5. Multiple passes (criteria drift).
6. Agent suggests, human confirms. Favor recall over precision.
7. Breadth, then depth, repeat.

Other skills in the plugin

Skill	Purpose
evals-start	Router to the right skill
eval-audit	Audit an existing pipeline (see Eval Audit Checklist)
generate-synthetic-data	Dimension/tuple generation (see Synthetic Data)
write-code-eval	Code checks (see Code-Based Evals)
write-judge-prompt	Judge prompts (see LLM-as-Judge)
validate-evaluator	TPR/TNR, splits, bias correction (see Judge Validation)
evaluate-rag	RAG retrieval and generation (see RAG Evaluation)
build-review-interface	Annotation UI (see Annotation Tools and Review Interfaces)

The skills are a starting point. Tune them to your data and domain.