Obsidian Metadata

type	concept
sources	Source - Theoretical Saturation (Delve), Source - AI Evals FAQ

Theoretical Saturation

A concept from grounded theory (Glaser & Strauss, 1967). It’s the point where collecting and analyzing more data no longer teaches you anything new about your topic. In grounded theory, collection and analysis alternate until you reach saturation. Once more data only confirms what you already know, you stop.

Applied to evals

It’s the stopping rule for Error Analysis: keep reviewing traces until new traces stop revealing new failure modes or changing existing ones.

Practical heuristics from the FAQ:

• Annotate at least 30 traces yourself first.
• Review at least 100 traces in total, and continue past 100 while you’re still learning.
• Hamel’s older footnote: “keep reading logs until you feel like you aren’t learning anything new.”
• Track convergence. Are new records mostly repeats of known modes, or do they reveal new ones? When the discovery rate drops, stop or narrow your focus (from the error-discovery skill’s review loop).

Saturation is about discovery, not measurement

Reaching saturation tells you the failure taxonomy is roughly complete. It doesn’t tell you how often each failure happens. That needs representative sampling and validated evaluators (see Judge Validation, Trace Sampling Strategies).

Saturation is also local in time. After significant product changes, the space of failures moves, so re-run error analysis (see Eval Maintenance and Cadence).

Related

Error Analysis · Criteria Drift · How Many Examples Do You Need