Obsidian Metadata
| type | source |
| url | https://arxiv.org/abs/2508.06709 |
| authors | Evangelia Spiliopoulou, Riccardo Fogliato, et al. |
| published | 2025-08-08 |
Source: Play Favorites: A Statistical Method to Measure Self-Bias in LLM-as-a-Judge
URL: https://arxiv.org/abs/2508.06709
A statistical framework that separates self-bias (a judge rating its own outputs too favorably) from genuine quality differences, using an independent third-party judge (humans) as the reference. On >5,000 prompt-completion pairs with expert annotations and nine judges, some models (e.g., GPT-4o, Claude 3.5 Sonnet) systematically favor their own outputs and show family bias.
FAQ stance: bias exists, but for scoped binary classification judges what matters is measured alignment with humans on a held-out set. Using the same model as the app is usually fine; switch only if alignment stalls. → LLM-as-Judge