Obsidian Metadata
Code-Based Evals (Assertions)
Deterministic checks for objective, rule-based failures. Use them whenever possible. They’re fast, cheap, deterministic, interpretable, easy to debug, and need no human labels to trust (only Pass and Fail test cases for the check itself).
Examples
| Failure mode | Possible check |
|---|---|
| Invalid output structure | Parse (JSON/schema) and check required fields |
| Missing or forbidden text | String / regex match (e.g., leaked UUIDs, competitor names) |
| Citation not in retrieved docs | Compare cited IDs with retrieved IDs |
| Bad tool call | Validate args against the tool schema, or run in a safe sandbox |
| Wrong value | Compare with a reference value |
| Contact creation works | Create a contact, then query the DB: exactly one matching record with the requested fields |
| Listing search | Scenario “one match” → len(listings)==1; “multiple” → >1; “none” → ==0 |
| Code generation | Execute; run unit tests; static analysis (ruff, mypy, bandit) |
| Agent outcome | State check on the environment (reservation row exists, order status changed) |
| Transcript constraints | n_turns <= 10, token or latency budgets |
Rechat’s “no exposed UUID” test
const noExposedUUID = message => {
// Remove all text within double curly braces
const sanitizedComment = message.comment.replace(/\{\{.*?\}\}/g, '')
// Search for exposed UUIDs
const regexp = /[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}/ig
const matches = Array.from(sanitizedComment.matchAll(regexp))
expect(matches.length, 'Exposed UUIDs').to.equal(0, 'Exposed UUIDs found')
}CRM results fed to the LLM contain UUIDs that users should never see. The prompt says not to include them, and the regex checks that it doesn’t.
How to write one (write-code-eval skill)
- Start from a failure mode found in Error Analysis. Write one check per failure mode, like a unit test that asserts what should hold for each trace.
- State the rule and identify the trace fields or reference data it needs. If the rule needs interpretation, use an LLM-as-Judge.
- Implement it in your project’s language and framework. Return a result and a reason.
- Test known passes and failures, including borderline cases, and inspect mistakes on real traces. If the rule is a proxy for human judgment, compare it with human labels.
- For a failure with objective and interpretive parts: check the objective part with code and use a judge for the rest.
Organizing assertions (Rechat lessons)
- Break the LLM’s scope into features × scenarios and write scoped tests per scenario. Add generic tests that apply everywhere (no UUIDs, valid JSON).
- Reuse assertions beyond tests: data cleaning, and automatic retries at inference time, where the assertion error helps course-correct.
- Make them fast enough to run on every code change.
- Update them constantly from failures observed in data. Rechat has hundreds.
- You don’t need 100%. The pass rate is a product decision about which failures you’ll tolerate. “One signal you are writing good tests is when the model struggles to pass them.”
- Brainstorm assertions with an LLM, but ground them in real traces.
Caveat from Anthropic: don’t over-specify paths
Checking that an agent followed an exact sequence of tool calls is often too rigid, because agents find valid paths you didn’t anticipate. Grade what the agent produced (the outcome), not the path, unless the path itself is the requirement (e.g., authorization before an action; see Agentic Workflow Evaluation).
Related
When to Build an Automated Evaluator · LLM-as-Judge · Three Levels of Evaluation · CI Evals vs Production Monitoring