Obsidian Metadata
| type | guide |
| sources | Source - AI Evals FAQ, Source - Evals Flashcards |
Multi-turn Conversation Evaluation
Start simple
- Did the whole conversation meet the user’s goal? One pass/fail judgment.
- Look at the entire trace and focus on the first upstream failure.
- Read the user-visible parts first to see whether something went wrong. Only then dig into tool calls and intermediate steps.
Multi-agent trace logging
Assign a session/trace ID to each user request and log every message with its source (which agent or tool), the trace ID, and its position in the sequence. That lets you reconstruct the full path across agents.
Annotation strategy
Annotate only the first failure at first. Downstream failures often cascade from it, so fixing upstream can fix the rest. With experience, annotate independent failure modes in the same trace to speed things up.
Simplify when possible
Reproduce each failure with the simplest possible test case.
A shopping bot gives the wrong return policy on turn 4. First try the single-turn version: “What is the return window for product X1000?” If that still fails, the problem isn’t conversation context. It’s likely retrieval or knowledge, which is much easier to debug.
“Not every multi-turn failure requires multi-turn analysis.”
Test-case generation
| Approach | How | Trade-off |
|---|---|---|
| N-1 testing | Replay the first N-1 turns of a real conversation and test the next turn | Usually better: real prefixes. Less flexible; assumes the system isn’t changing fast |
| User simulation | Another LLM plays the user | Flexible and realistic when done well, but hard to do well |
| Perturbation | Modify reproduced traces in valid ways (e.g., rephrase the user) | Extra coverage |
When the conversation involves tools or multiple agents, use a Transition Failure Matrix to find hotspots.
Related
Traces · Agentic Workflow Evaluation · Human Handoff Evaluation