Obsidian Metadata
| type | concept |
| sources | Source - AI Evals FAQ |
Evaluating Systems with Diverse User Queries
Complex applications field wildly different queries, from “What’s the return policy?” to “Compare pricing trends across regions for products matching these criteria”. That creates confusion about how to design eval criteria.
“Error analysis is all you need.” Let the evaluation strategy emerge from observed failure patterns, not from predetermined query classifications. Don’t build a huge matrix of every query type you can imagine.
- During Error Analysis you’ll often find that failures cut across query categories. For example, temporal reasoning struggles regardless of whether a query is a lookup or an aggregation, and multi-source combination fails in consistent ways.
- Query category might turn out to be a good grouping, but you don’t know that until you’ve analyzed the data.
Related
Error Analysis · Synthetic Data (use dimensions for generation coverage, not as your eval structure)