| Abstention ability | A model’s ability to decline answering when it lacks information | Abstention and Uncertainty |
| Active learning | Having the system choose the most informative items for a human to label next | Trace Sampling Strategies |
| Agent harness / scaffold | The system that lets a model act as an agent (tools, orchestration). Agent evals test harness + model together | Agentic Workflow Evaluation |
| Axial coding | Grouping open-coding notes into a failure taxonomy and counting each category | Error Analysis |
| Benevolent dictator | The single domain expert who decides what quality means | Benevolent Dictator |
| Capability eval | An eval with a low initial pass rate that gives you a hill to climb | Capability vs Regression Evals |
| Capability funnel | Progressive levels of utility (valid → executes → relevant → intent → optimal) | Capability Funnel and Experiment Roadmaps |
| Code-based eval / assertion | A deterministic check (regex, schema, DB state, tests) | Code-Based Evals |
| Cohen’s Kappa | Chance-corrected agreement between human annotators | Benevolent Dictator |
| Context rot | Performance degradation as input length grows | Judge Context and Context Rot |
| Criteria drift | Grading outputs changes the criteria you grade by | Criteria Drift |
| Critique | Written reasoning that accompanies a pass/fail verdict; also used as few-shot examples | Critique Shadowing |
| Critique shadowing | Hamel’s process for building a judge that mirrors a domain expert’s critiques | Critique Shadowing |
| Dev set | Labeled examples used to iterate on a judge. Never placed in the prompt | Judge Validation |
| Dimensions / tuples | Axes of variation, and value combinations, used to structure synthetic data | Synthetic Data |
| Error analysis / error discovery | Systematic trace review → failure taxonomy | Error Analysis |
| Eval saturation | When an eval passes (nearly) everything and stops giving improvement signal | Eval Maintenance and Cadence |
| Evaluator | Any check that scores output quality, usually offline or async | Guardrails vs Evaluators |
| Failure mode / taxonomy | A distinct, recurring way the product fails; the list of all of them | Error Analysis |
| First upstream failure | The earliest error in a trace, which often causes later ones | Traces |
| Generic metrics | Off-the-shelf scores (helpfulness, coherence, ROUGE…) | Generic Metrics |
| GEPA | A reflective, Pareto-based prompt optimizer | Source - GEPA |
| Grader | Logic that scores part of a trial (code, model, or human) | Agentic Workflow Evaluation |
| Guardrail | A fast, inline, synchronous check that blocks, retries, or falls back | Guardrails vs Evaluators |
| Integrated prompt environment | An admin mode of the real product UI for editing prompts | Prompt Management |
| LLM-as-judge | An LLM used as a binary classifier for one failure mode | LLM-as-Judge |
| N-1 testing | Replaying the first N-1 turns of a real conversation and testing turn N | Multi-turn Conversation Evaluation |
| Open coding | Free-text, journal-style notes on traces | Error Analysis |
| Outcome | Final environment state after a trial (vs the transcript) | Traces |
| Pass-all rate | Share of examples passing every check | Combining Evals into a Single Metric |
| pass@k / pass^k | At least one success in k trials / all k trials succeed | pass@k and pass^k |
| Principal domain expert | See benevolent dictator | Benevolent Dictator |
| Product evals | Evals of your system on your tasks (vs model benchmarks) | Product Evals vs Model Benchmarks |
| Regression eval | An eval expected to pass ~100%, guarding against backsliding | Capability vs Regression Evals |
| Rogan–Gladen correction | Adjusting an observed judge pass rate for the judge’s TPR/TNR | Judge Bias Correction |
| Test set | Held-out labels used once for the final judge estimate | Judge Validation |
| Theoretical saturation | The point where more data yields no new failure modes | Theoretical Saturation |
| Think-aloud protocol | An expert narrates their reasoning while reviewing | Benevolent Dictator |
| TPR / TNR | True positive rate / true negative rate of a classifier or judge | Judge Validation |
| Trace | The full record of one request/session, including all steps | Traces |
| Train set | The small labeled pool few-shot examples are drawn from | Judge Validation |
| Transition failure matrix | Last successful state × first failure state counts | Transition Failure Matrix |
| Trial | One attempt at an eval task | pass@k and pass^k |
