Obsidian Metadata
| type | source |
| subtype | video |
| url | https://youtu.be/KEdoIbBu2Ko |
| speakers | Alex Volkov (Weights & Biases / ThursdAI), Hamel Husain |
| duration | 1:02:52 |
Video: Reasoning Models & LLM-as-a-Judge
Alex tests the hypothesis that reasoning models (LRMs) are better LLM judges against published research as of about mid-2025. His answer: “It depends. Meta-evaluate.”
Where reasoning judges help
- Verifiable domains (math, code). A judge’s ability to verify a solution is highly correlated with its ability to solve the problem (a JudgeBench ablation). On JudgeBench, reasoning models beat non-reasoners by 10–20 points. A useful trick is to have the judge solve first, then judge.
- Thinking budgets (Gemini token budgets, OpenAI reasoning effort) add a knob: you can meta-evaluate the same judge at different budgets and trade cost against alignment.
- Visible reasoning traces help you debug judges and spot bias, when providers expose them.
Where they don’t (or it’s unclear)
- Creative writing / multi-criteria scoring (JudgeMark): reasoning can overthink and hurt results.
- Survey results on alignment with human preferences were mixed and even revised between paper versions. Check the latest arXiv version before citing.
- They still show biases: position (they prefer later options), bandwagon, authority, distraction, self-preference and family bias (no better than LLMs), and a new “superficial reflection” bias, where text like “wait, let me think” in the evaluated content sways verdicts. Irrelevant injected info can derail their reasoning. Many studies tested only one model family, so don’t over-generalize.
- Slow, so generally ruled out for online evals and guardrails. Fine offline.
- Composable judge panels (e.g., Verdict) reached comparable accuracy at about a third of the cost in one report.
Practical guidance
- Higher eval scores ≠ a better judge. Alex swapped GPT-4o for o1, saw a higher score, and found Cohen’s κ against humans unchanged. Always meta-evaluate judges against human labels (in this vault’s terms: TPR/TNR on a held-out set, see Judge Validation).
- Prompting techniques for reasoning judges: role assignment, task decomposition, self-reflection on its own reasoning, bias-aware instructions, and provider prompt improvers (e.g., the Anthropic console’s option to tailor a prompt for thinking models).
- Reasoning is becoming a toggle on hybrid models, so the practical question is “reasoning on, or off, and at what budget?”, answered by meta-evaluation.
→ LLM-as-Judge · Judge Validation · Source - Play Favorites Self-Bias