Obsidian Metadata
| type | concept |
| sources | Source - AI Evals FAQ, Source - Evals Flashcards |
Transition Failure Matrix
For agents and pipelines with many steps (Plan → Search → Code → Finalize, or ParseReq → IntentClass → DecideTool → GenSQL → ExecSQL → …), it’s hard to see which step fails most. A transition failure matrix shows the hotspots.
How to build one
- Define states: every step or state the agent can be in.
- Create a grid: rows = last successful state (“from”), columns = state where the first failure occurred (“to”).
- Count failures: for each failing trace, find the last successful transition before the error and increment that cell.
Reading it: in the text-to-SQL example, GenSQL → ExecSQL = 12 failures vs DecideTool → PlanCal = 2, so investigate SQL generation/execution first.
Bryan Bischof (“Failure is a Funnel”, Data Council 2025) compares matrices across experiments and exploits the inherently sequential text-to-SQL workflow for deeper analysis. How you organize the matrix depends on your application.
Using matrices to compare experiments
Bryan Bischof compares before/after matrices for each treatment. Two experiments may improve different funnel stages, and the funnel tells you which matters more. Upstream failures count more, because nothing downstream can succeed. Fixing them may reveal new downstream failures. Linearize agent “spaghetti” into the paths you care about (Video - Stop Managing AI Projects Like Traditional Software).
A related tool is a tool-selection confusion matrix (intended vs actual tool). Frequently confused tools overlap and need clearer partitioning (Video - The Evals That Made GitHub Copilot).
Where to use it
- After Error Analysis identifies failing workflows, as step-level diagnostics in Agentic Workflow Evaluation.
- Multi-Step Workflow Evaluation: early-stage failures cascade, so upstream fixes have the most impact.
- Multi-turn Conversation Evaluation involving tools or several agents.
Related
Agentic Workflow Evaluation · Capability Funnel and Experiment Roadmaps