Obsidian Metadata
| type | concept |
| sources | Source - AI Evals FAQ, Source - Evals Skills for Coding Agents |
Product Evals vs Model Benchmarks
AI evals are tests that tell you whether an AI system is doing what you want. They give feedback when the product drifts from user needs or business goals, and the failures they catch become data you can use to improve the system.
More formally, evaluation is the systematic measurement of quality. Each eval checks one behavior on relevant examples and returns a score or a structured review. Most products need several evals because they fail in several different ways.
The word “evals” usually means one of two things:
| Model benchmarks | Product evals | |
|---|---|---|
| Question | Which general-purpose model is strongest? | Does my product do what I want, on my data? |
| Examples | GPQA Diamond (science reasoning), Terminal-Bench (CLI agents), MMLU (broad knowledge), SWE-bench, HumanEval | ”Did the cancellation agent select the correct order and wait for cancel_order to succeed before confirming?” |
| Scope | The model alone | The model, prompts, retrieval, tools, and application code |
| Use | Picking a promising starting model | Catching failures that matter to users and the business |
| Access to your systems | None | Full |
A high GPQA or Terminal-Bench score tells you little about whether your order-cancellation agent works, because those benchmarks can’t see your systems.
This vault is about product evals. Their loop is:
- Analyze traces to discover real failure modes.
- Turn the important failures into targeted evals (Code-Based Evals or LLM-as-Judge).
- Use the results to guide changes.
- Re-run the evals to confirm the system improved.
Mechanisms for implementing product evals
- Code assertions → Code-Based Evals
- Human review → Error Analysis, Benevolent Dictator
- LLM judges / classifiers → LLM-as-Judge
- Online experiments → A/B testing
Which one to use depends on the failure you’re measuring.
Where model benchmarks still help
- Picking a starting model.
- Coding agents that handle a very wide variety of tasks. Treat public benchmarks the way you would for a foundation model, then add a private benchmark.
Will this still matter in 5–10 years?
Yes. Even a perfect model has to be checked against the right problem. Prompt tricks will go out of date, but you’ll still need to understand failure modes. An LLM can’t read your mind, and research (Shankar et al.) shows people must observe LLM behavior before they can state their requirements (Criteria Drift). For both sides of the debate, see “The model is the product” vs “The model is NOT the product” in Video Library.