Generic Metrics

Obsidian Metadata

type	concept
sources	Source - AI Evals FAQ, Source - A Field Guide to Rapidly Improving AI Products, Source - The Revenge of the Data Scientist

Generic Metrics (and Similarity Metrics)

"No. Generic evaluations waste time and create false confidence when you use them as quality measures."

“The abuse of generic metrics is endemic. Many eval vendors promote off-the-shelf metrics, which ensnare engineers into superfluous tasks.” (from the course)

Generic metrics are the helpfulness, coherence, quality, “hallucination score”, toxicity, and verbosity scores that ship in eval libraries. Similarity metrics are BERTScore, ROUGE, BLEU, and cosine similarity to a reference.

Why they mislead

1. They measure abstract qualities that may not matter for your use case. Good scores don’t mean the system works.
2. False sense of progress. Teams think they’re data-driven because they have dashboards. One team celebrated a 10% “helpfulness” gain while users still struggled with basic tasks. “Like optimizing page load time while checkout is broken.”
3. Fragmented attention. Too many metrics means optimizing many dimensions at once: “when everything is important, nothing is.”
4. They don’t catch real failures. A real-estate CRM assistant suggesting unavailable showings (assertion) or confusing client personas (judge) won’t show up in similarity or verbosity scores.
5. They don’t correlate with domain-expert judgment in Hamel’s analyses.

The “dashboard that foreshadows failure”: rows of 1–5 scores for helpfulness, conciseness, coherence, and so on.

What to do instead

1. Error Analysis to find the real failures.
2. Define binary failure modes from real problems (Binary Pass-Fail vs Likert Scales).
3. Build custom evaluators (Code-Based Evals or LLM-as-Judge) and validate them against humans (Judge Validation).

Application-specific metrics look like “Calendar Scheduling Failure” or “Failure to Escalate To Human”, not “Helpfulness 3.7”.

Legitimate uses

• Exploration signals. Experienced practitioners use generic metrics to find interesting traces to review (e.g., sort by low coherence or high verbosity), not as quality measures (see Trace Sampling Strategies).
• Retrieval / search / recommendation. Cosine similarity between embeddings is fine for semantic closeness in retrieval, so similarity metrics help when optimizing and debugging RAG retrieval.
• Output diversity. Average pairwise similarity (lower = more diverse), e.g., to check synthetic data variety.

For platforms

Internal eval platforms should strongly avoid offering generic metrics as a shortcut (see Internal Eval Platforms). When assessing a vendor, look for axial-coding and error-analysis support, not a metrics catalog (see Eval Tooling and Vendors).

Related

Error Analysis · Binary Pass-Fail vs Likert Scales · Source - The Revenge of the Data Scientist