Obsidian Metadata
| type | concept |
| sources | Source - AI Evals FAQ |
Evals with Sensitive Data
“There is no replacement for looking at real interactions.” When traces are sensitive, try these options in order of preference:
- Find real data you’re allowed to inspect. A customer may share a subset, or test users may let you review their sessions. Even limited access shows how people really use the product.
- Work with domain experts who are allowed to see it, and make the product easy for them to verify as part of their normal work (see Designing for Verification). Example: a medical research assistant shows the clinician the evidence behind each claim and flags conflicting sources. The clinician’s corrections and resolutions become eval data (subject to the same storage and sharing restrictions). Learn how experts check answers, and give them evidence links and smaller units to review. “Was the final answer helpful?” tells you too little.
- Redact or edit traces so they can be shared. If sensitive information can’t be stored, redact before logging. Redaction tools miss things, so check their output. When editing (removing PII, changing sensitive details), make sure the edits preserve the behavior you need to evaluate.
- Synthetic data is the last resort. It helps find initial problems but gives limited evidence about real user behavior (see When synthetic data is unreliable).