Thought UmweltThought Umwelt

thought umwelt

Critique Shadowing

4 min read

Obsidian Metadata

typeconcept
sourcesSource - Using LLM-as-a-Judge, Source - A Field Guide to Rapidly Improving AI Products

Critique Shadowing

Hamel’s name for his end-to-end process for building an LLM-as-Judge that matches a domain expert. The judge “shadows” the expert’s critiques.

graph TB
    A[Start] --> B[1 Find Principal Domain Expert]
    B --> C[2 Create Dataset]
    C --> D[3 Domain Expert Reviews Data]
    D --> E{Found Errors?}
    E -->|Yes| F[4 Fix Errors]
    F --> D
    E -->|No| G[5 Build LLM Judge]
    G --> H[Test Against Domain Expert]
    H --> I{Acceptable Agreement?}
    I -->|No| J[Refine Prompt]
    J --> H
    I -->|Yes| K[6 Perform Error Analysis]
    K --> L{Critical Issues Found?}
    L -->|Yes| M[7 Fix Issues & Create Specialized Judges]
    M --> D
    L -->|No| N[Material Changes or Periodic Review?]
    N -->|Yes| C

Steps

  1. Find the principal domain expert (see Benevolent Dictator).
  2. Create a diverse dataset along dimensions such as features × scenarios × personas, from real or synthetic inputs. Use your real APIs and databases so scenarios actually trigger.
  3. Expert makes pass/fail judgments plus detailed critiques. The single question: “Did the AI achieve the desired outcome?” Don’t do root-cause analysis yet. Remove all friction from review.
  4. Fix obvious errors before building a judge, then go back to step 3. Finding errors early is the whole point.
  5. Build the judge iteratively. Use expert critiques as few-shot examples and iterate until agreement is acceptable. Measure with TPR/TNR, not raw agreement (see Judge Validation).
  6. Error analysis with the judge. Apply it to unseen data, compute failure rates by dimension, then classify root causes by hand.
  7. Create specialized judges (or code assertions) only where needed.

The process never truly ends. Repeat periodically and whenever something material changes (e.g., a model update).

Why critiques matter

Every judgment gets a binary verdict plus a critique explaining the reasoning.

  • Captures nuance: “mostly right, but…”
  • Guides improvement with specific feedback.
  • Forces articulation. Experts often haven’t internalized all their criteria. Deciding and explaining clarifies them (see Criteria Drift).
  • Feeds the judge. Critiques become few-shot examples in the judge prompt. Hamel sees 15–20% higher human–LLM agreement with example critiques than without.
  • Fuels synthetic data / fine-tuning. Critiques can be used to filter data, generate better outputs, or simulate chain-of-thought.

Critique quality bar

A critique should be detailed enough that a new employee could understand it, and good enough to use as a few-shot example. Being too terse is the most common mistake.

Example critiques (from the LLM-judge guide)

InteractionVerdictCritique
U: “Cancel my flight next week.” AI: “Canceled. Confirmation email sent.”PassAchieved the goal, but should have asked for confirmation before an irreversible action. Passes despite that oversight; a confirmation step would improve safety.
U: “What’s the capital of France?” AI: “Paris.”PassCorrect and prompt. Could offer more engagement, but met the primary need.
U: “Can’t log in; says password incorrect.” AI: “Click ‘Forgot Password’.”FailGeneric answer, no acknowledgment of frustration, no check for account lockout. Poor support experience.
U: “Book a table for two, Italian, 7 PM.” AI: “Booked at Luigi’s, conf #12345.”PassCompleted with all details. Could ask about seating or dietary preferences; minor.
U: “Help with my billing statement.” AI: “Provide your account number and full name.”FailAsked for sensitive info without secure identity verification. Critical data-protection oversight.

Another example (Field Guide): “The AI successfully provided the market analysis requested (PASS), but included excessive detail about neighborhood demographics that wasn’t relevant to the investment question. This makes the response longer than necessary and potentially distracting.”

When it fails

  • The AI is overscoped (e.g., a SaaS chatbot that “does anything”).
  • The process isn’t followed: no principal expert, poor critiques.
  • Alignment expectations are unrealistic. Address the root cause instead of forcing alignment, and lean on human annotation where needed.

Shortcut version

If you’re a solo developer who is also the domain expert, data already exists, and looking at data is cheap, jump straight to step 3 and do error analysis without a judge at first. You can never skip looking at data.

"It's not the judge that created value, after all"

“Creating a LLM judge is a nice ‘hack’ I use to trick people into carefully looking at their data!” — Hamel

LLM-as-Judge · Judge Validation · Honeycomb Query Assistant · Judge Prompt Template

Graph View