Obsidian Metadata
| type | template |
Error Analysis Worksheet
A spreadsheet or notebook layout for one Error Analysis cycle. Copy it into a sheet or DataFrame.
Session header
- Date:
YYYY-MM-DD· Reviewer (benevolent dictator):__· Pipeline/prompt version:__· Model:__ - Trigger: ☐ scheduled cycle ☐ prompt change ☐ model switch ☐ new feature ☐ incident / complaint spike
- Sampling mix:
__random ·__cluster reps ·__outliers (length/retries/latency) ·__flagged by evals ·__negative feedback
Sheet 1: Open coding (one row per trace)
| trace_id | source (real/synthetic) | channel / persona / feature | pass_fail | first_upstream_failure_note (free text) | other_notes | defer? |
|---|---|---|---|---|---|---|
Rules: annotate ≥ 30 yourself before any AI suggestions · note what went wrong, not why · include non-model problems (missing features, UX) · write the first upstream failure.
Sheet 2: Axial coding (failure taxonomy)
| failure_mode | description (Pass/Fail boundary) | count | % of failures | example trace_ids | severity (blocking?) | fix type |
|---|---|---|---|---|---|---|
| prompt / code bug / retrieval / product-UX / needs eval |
Sheet 3: Decisions
| failure_mode | action | evaluator? | type (code / judge / none) | owner | status |
|---|---|---|---|---|---|
| fix now → re-check | no | – | |||
| persistent → hill-climb | yes | judge |
Saturation tracker
| batch # | traces reviewed (cumulative) | new failure modes found | modes changed |
|---|---|---|---|
| 1 | 30 | ||
| 2 | 60 | ||
| 3 | 100 |
Stop when new batches stop producing new or changed modes (see Theoretical Saturation).
Pivot (optional)
Pivot failure_mode × (feature | persona | channel) to find where failures cluster. For agents, also fill a Transition Failure Matrix.