Obsidian Metadata
| type | concept |
| sources | Source - AI Evals FAQ, Source - The Revenge of the Data Scientist |
Should You Outsource Annotation?
Outsourcing Error Analysis is usually a big mistake. The core of evaluation is building product intuition that only comes from systematically analyzing your own system’s failures. Be extremely skeptical of delegating it.
Dangers
- Breaks the feedback loop between observing a failure and knowing how to improve the product.
- Superficial labeling. Even well-defined metrics need nuanced judgment that outside teams (or generalist devs and IT staff) lack.
- Loss of tacit knowledge. The principal domain expert holds knowledge and user understanding that no rubric fully captures, and preferences they can’t articulate upfront.
- Annotation conflicts. Without shared context, external annotators create more disagreement than they resolve.
“Labeling seems unglamorous, so it gets delegated… A data scientist would insist that domain experts label the data, stay skeptical of the labels, and look at the data.” — Source - The Revenge of the Data Scientist
Build internal capability instead
- Appoint a Benevolent Dictator.
- If you need multiple annotators, run a structured alignment process: rubric → independent labels → Cohen’s Kappa → alignment sessions → iterate.
Handling capacity constraints
- Smart sampling: 100 diverse traces reviewed thoroughly beat thousands labeled superficially.
- Think-aloud protocol: one hour of an expert narrating their review.
- Lightweight custom tools to raise throughput (see Annotation Tools and Review Interfaces).
Legitimate exceptions
- Purely mechanical tasks (is this a phone number? a valid email?), after an internal process has defined the rubric.
- Tasks without product context, e.g., translation (needs linguistic expertise, not product knowledge).
- Hiring external SMEs to be your domain experts isn’t outsourcing. AnkiHub hired 4th-year medical students to evaluate their medical RAG.
Related
Benevolent Dictator · What LLMs Can and Can't Automate in Evals