Obsidian Metadata
| type | concept |
| sources | Source - AI Evals FAQ, Source - Using LLM-as-a-Judge |
Benevolent Dictator (Principal Domain Expert)
For most small and medium-sized companies, appoint a single domain expert as the definitive voice on quality: a “benevolent dictator”. They can take input from others, but they drive the process.
Examples of principal domain experts
- A psychologist for a mental-health assistant
- A lawyer for legal document analysis
- A customer-service director for a support bot
- A lead teacher / curriculum developer for an education tool
- In a small company, often the CEO/founder. If you’re an independent developer, you (but be honest about your expertise). If you must rely on leadership, regularly check their assumptions against real user feedback.
Why one person
- Sets the standard. They define what’s acceptable and help you tell whether you’re building something users want.
- Surfaces unspoken expectations. Experts often can’t state their preferences upfront. Pass/fail decisions with critiques draw them out.
- Consistency. Opinions vary across an org. One standard removes annotation conflicts and the “too many cooks” paralysis.
- Ownership. An expert who helped shape the AI is invested in it and more likely to approve it.
If you feel you need five subject-matter experts to judge a single interaction, your product scope is probably too broad.
Anti-pattern
Developers acting as the domain expert themselves, or picking a convenient proxy such as their boss, is “a recipe for disaster”. What matters is that the principal domain expert is satisfied.
When you do need multiple annotators
Larger orgs, or products spanning multiple domains or cultures (for example a multinational), may need several annotators. Then:
- Draft an initial rubric with clear Pass/Fail definitions and examples.
- Have annotators label the same examples independently before discussing them.
- Measure inter-annotator agreement (IAA) with a chance-corrected metric such as Cohen’s Kappa. Use Kappa for human–human agreement, not for judge-vs-truth (see Judge Validation).
- Run alignment sessions on the disagreements: which part of the rubric caused it, and what rule would make the next decision clear?
- Update the rubric with a definition, rule, or example that covers the disputed case, then relabel the affected examples.
- If they still disagree, a domain expert decides and records the reason.
- Iterate until agreement is consistently high.
Real example: Nova Escola used two annotators who agreed less often than chance. The fix was to rewrite the rubric with their pedagogical experts and relabel.
Start with a benevolent dictator whenever feasible. Add complexity only when absolutely necessary.
Making the expert’s time efficient
- Build a custom review tool that shows all context on one screen, including the outcome (the confirmation, the email, the DB update).
- Think-aloud protocol (from usability testing): the expert talks through their reasoning while reviewing a handful of traces. One hour can yield deep insight.
- Ask about outcomes, not implementation. “Has an appointment been made?” rather than “Did the tool call succeed?”
- Smart sampling. 100 diverse traces beat thousands of superficial labels.
- Hiring external SMEs to act as your in-house domain experts is fine; it isn’t outsourcing. Example: AnkiHub hired 4th-year medical students.
Related
Error Analysis · Annotation Outsourcing · Team Collaboration on Evals · Critique Shadowing