Obsidian Metadata
| type | guide |
| sources | Source - AI Evals FAQ, Source - Demystifying Evals for AI Agents |
Evaluating Abstention (“Know What It Doesn’t Know”)
Many apps need the model to refuse when it lacks sufficient information, without refusing questions it should be able to answer.
Build a two-sided eval set
- Answerable questions: a correct, verifiable answer exists in the provided context or general knowledge.
- Unanswerable questions designed to tempt hallucination: false premises, information explicitly missing from context, topics far outside its knowledge.
A roughly balanced mix is a good start. Diversity and difficulty matter more than the exact ratio.
Scoring
A binary check of judgment. Pass = answers the answerable ones and refuses the unanswerable ones. Fail = a fabricated answer to an unanswerable question (poor calibration). Over-refusal on answerable questions is also a failure.
This mirrors Anthropic’s advice on balanced problem sets. One-sided evals produce one-sided optimization (e.g., testing only “should search” yields an agent that searches for everything). Related findings: in context-rot experiments, some model families abstain under distractors while others hallucinate (see Judge Context and Context Rot). Jason Liu’s Q|C (question answerability) covers this for RAG.
The research term is “Abstention Ability”. Search arXiv for the latest techniques.