Thought UmweltThought Umwelt

thought umwelt

Judge Bias Correction

3 min read

Obsidian Metadata

typeconcept
sourcesSource - Evals Skills for Coding Agents, Source - AI Evals FAQ

Judge Bias Correction (Rogan–Gladen)

Raw judge scores on unlabeled production data are biased by the judge’s known errors. Once you know the judge’s TPR and TNR from the test set (see Judge Validation), you can correct the observed rate to estimate the true rate. The FAQ: “Once you know your judge’s True Positive and True Negative rates, you can correct its estimates to determine the actual failure rate in your system.”

Formula (positive = Pass, as in the validate-evaluator skill)

  • p_obs = fraction of unlabeled traces the judge scored Pass
  • TPR, TNR = from the test set
  • θ̂ = corrected estimate of the true success rate. Clip to [0, 1].
  • Invalid when TPR + TNR − 1 ≈ 0, i.e. the judge is no better than random.

Worked example: TPR = 0.92, TNR = 0.88. Of 500 prod traces, 400 are judged Pass, so p_obs = 0.80. θ̂ = (0.80 + 0.88 − 1) / (0.92 + 0.88 − 1) = 0.68 / 0.80 = 0.85. The true success rate is ~85%, not the raw 80%.

If you use the FAQ convention (positive = failure)

Apply the same formula with p_obs = fraction flagged as Fail, and TPR/TNR defined for failure detection. The result is the corrected failure rate.

Always report a confidence interval

A corrected 85% could easily be 78–92% with a small test set. Bootstrap over the labeled test set:

import numpy as np
def bootstrap_ci(human, judge, p_obs, n_boot=2000):
    human, judge = np.array(human), np.array(judge); n = len(human); est = []
    for _ in range(n_boot):
        i = np.random.choice(n, n, replace=True); h, e = human[i], judge[i]
        tp = ((h=='Pass')&(e=='Pass')).sum(); fn = ((h=='Pass')&(e=='Fail')).sum()
        tn = ((h=='Fail')&(e=='Fail')).sum(); fp = ((h=='Fail')&(e=='Pass')).sum()
        tpr = tp/(tp+fn) if tp+fn else 0; tnr = tn/(tn+fp) if tn+fp else 0
        d = tpr + tnr - 1
        if abs(d) < 1e-6: continue
        est.append(np.clip((p_obs + tnr - 1)/d, 0, 1))
    return np.percentile(est, 2.5), np.percentile(est, 97.5)

Or use the judgy package (pip install judgy):

from judgy import estimate_success_rate   # labels as 0/1, 1 = Pass
theta_hat, lower, upper = estimate_success_rate(test_labels, test_preds, unlabeled_preds)

Practical notes

  • Improving TPR narrows the CI more than improving TNR (in the Pass-positive convention), because a low TPR shrinks the denominator and magnifies estimation error.
  • In production monitoring, track the CI. If the lower bound crosses your threshold, investigate (see CI Evals vs Production Monitoring).
  • Anti-patterns: reporting raw judge rates without correction, and reporting point estimates without CIs.

Judge Validation · LLM-as-Judge · Combining Evals into a Single Metric

Graph View