Obsidian Metadata
| type | concept |
| sources | Source - AI Evals FAQ |
Reviewing Large Traces
Traces get large when an agent runs for a long time or retrieves a lot of context.
- Focus on the first upstream failure. Errors compound, so prioritizing early ones saves time (see Traces).
- Progressive disclosure in the review tool: show the most relevant information first (e.g., the conversation) with tool outputs collapsed until needed.
- If a single trace is still too large, work with the domain expert to identify what they need to check, then build a tool that extracts the relevant evidence and links back to its location in the trace or document. Example: for an answer about a long contract, show the relevant clauses with links to their original pages. Always validate the extraction with a domain expert.
- Quality > quantity. Carefully investigating a few failures teaches more than rushing through many.
For judges facing huge traces, see Judge Context and Context Rot (give the judge only what it needs; give it a search tool only if you absolutely must).
Related
Designing for Verification · Annotation Tools and Review Interfaces · Agentic Workflow Evaluation