Obsidian Metadata

type	concept
sources	Source - Demystifying Evals for AI Agents, Source - AI Evals FAQ

Capability vs Regression Evals

	Capability (“quality”) evals	Regression evals
Question	”What can this agent do well?"	"Does it still handle everything it used to?”
Expected pass rate	Low at first: a hill to climb	~100%: a drop means something broke
Built from	Tasks the agent struggles with	Tasks the agent reliably solves

Capability evals that reach high pass rates “graduate” into the regression suite, run continuously to catch drift. “Can we do this at all?” becomes “Can we still do this reliably?”
Run regression evals alongside capability hill-climbing so changes don’t break other things.
This lines up with Hamel’s guidance: CI sets = regression-style (curated, cheap, deterministic); a ~70% pass rate on challenge sets is healthier than 100% (see When to Build an Automated Evaluator, Eval Maintenance and Cadence).
Descript runs two separate suites: quality benchmarking and regression testing.

thought umwelt