Obsidian Metadata
| type | concept |
| sources | Source - AI Evals FAQ |
What Should an Internal Eval Platform Standardize?
It’s tempting to start with tools, infrastructure and a shared metrics catalog. Teams then adopt whatever the platform offers without checking whether it helps them find and fix problems.
Standardize the process, not the metrics
- Encourage every team to do Error Analysis and sample data well.
- Teams use their failures to decide which automated checks to build, then validate them against human labels.
- Let each team develop its own metrics and, when needed, its own tools.
Give teams flexibility on tools
AI coding agents make custom software cheap. Annotation tools usually need data-specific interfaces: reviewing OCR’d documents is different from reviewing chat conversations.
What a platform can provide
- Shared storage for results.
- Support for collaborative labeling.
- Start by serving one team and one use case well, then expand as you learn which needs are truly shared. Standardization pays off less when teams have very different needs and can build tools cheaply.
Don’t
- Offer generic metrics (helpfulness, coherence) as a shortcut. Strongly advised against.
- Compare eval scores across projects unless the checks and test data are actually comparable.
Related
Eval Tooling and Vendors · Generic Metrics · Eval Maintenance and Cadence