Thought UmweltThought Umwelt

thought umwelt

Judge Prompt Template

3 min read

Obsidian Metadata

typetemplate
sourcesSource - Evals Skills for Coding Agents, Source - Using LLM-as-a-Judge

Judge Prompt Template

Use this for one failure mode found in Error Analysis. Validate the judge with Judge Validation before trusting it.

Pre-flight checklist

  • The failure mode was observed in error analysis (not brainstormed)
  • It can’t be checked with code (regex / schema / DB lookup / execution); see Code-Based Evals
  • The prompt already asks for the desired behavior (otherwise fix the prompt first)
  • ≥ 20 Pass and ≥ 20 Fail human labels exist (target 100–200 total)
  • Labels are split into train / dev / test, and few-shot examples come only from train
  • I know exactly which trace fields the judge needs (and nothing more)

Template

You are an evaluator assessing whether {{AI system description}} {{ONE specific criterion}}.
 
## Context you will receive
- {{field 1, e.g., client persona}}
- {{field 2, e.g., generated email}}
- {{any external context a human reviewer would need: user metadata, system state, retrieved docs}}
 
## Definitions
PASS: {{precise definition drawn from your failure-mode description}}
- {{concrete sub-rule}}
- {{concrete sub-rule}}
 
FAIL: {{precise definition}}. Examples:
- {{observed failure pattern 1}}
- {{observed failure pattern 2}}
 
If the provided information is insufficient to decide, answer "Unknown".
 
## Examples
### Example 1: PASS
{{input fields}}
Critique: {{detailed critique citing specific evidence; understandable by a new employee}}
Result: Pass
 
### Example 2: FAIL
{{input fields}}
Critique: {{detailed critique}}
Result: Fail
 
### Example 3: PASS (borderline)
{{input fields}}
Critique: {{why this edge case still passes}}
Result: Pass
 
## Now evaluate
{{input fields for the item under review}}
 
First write a detailed critique explaining your reasoning, then give the verdict.
Respond as JSON: {"critique": "<detailed assessment>", "result": "Pass" | "Fail" | "Unknown"}

Notes

  • Enforce the JSON with the provider’s schema features (response_format, tool definitions) or Instructor / Outlines.
  • Critique before verdict.
  • 2–4 examples is typical; include a borderline one.
  • Start with the strongest model you can afford; pin a dated snapshot.
  • For severity, use multiple binary judges (“factually wrong”, “dangerously wrong”), not a scale.

Filled example (tone / persona)

You are an evaluator assessing whether a real estate assistant's email uses the appropriate
tone for the client's persona.
 
PASS: The email matches the expected communication style for the client persona:
- Luxury Buyers: formal language, emphasis on exclusive features, premium market positioning, no casual slang
- First-Time Homebuyers: warm and encouraging tone, educational explanations, avoids jargon, patient and supportive
- Investors: data-driven language, ROI-focused, market analytics, concise and professional
FAIL: The email uses a tone mismatched to the client persona. Examples:
- Using casual slang ("hey, check out this pad!") for a luxury buyer
- Using heavy financial jargon for a first-time homebuyer
- Using overly emotional language for an investor
 
### Example 2: FAIL
Client Persona: Luxury Buyer
Email: "Hey! Just found this awesome place you might like. It's got a pool and stuff, super cool neighborhood..."
Critique: The greeting "Hey!" is informal. Phrases like "awesome place," "got a pool and stuff," and
"super cool" are casual slang inappropriate for a luxury buyer. The email reads like a text message,
not a professional communication for a high-end client.
Result: Fail

Graph View