Obsidian Metadata
| type | guide |
| sources | Source - AI Evals FAQ |
Multi-Step Workflow Evaluation
- Log the entire workflow from initial trigger to final business outcome: LLM calls, tool usage, human approvals, database writes. You need this to diagnose failures.
- Use both outcome and process metrics.
- Outcome: does the final result meet requirements? (Was the business case complete? Accurate? Properly formatted?)
- Process: efficiency (step count, time taken, resources used). Process failures are more deterministic and easier to debug, so tackle them first.
- Segment error analysis by stage. Early failures (understanding input) differ from middle ones (data processing) and late ones (formatting output). Early-stage improvements have more impact because errors cascade in LLM chains.
- Use a Transition Failure Matrix (last successful state vs first failure) to find hotspots.
- Frame progress as a capability funnel (valid → executes → relevant → matches intent → optimal).