The industry replaced human graders with an AI judge. Now someone has to grade the judge.
Strong LLM judges agree with people over 80% of the time, about as often as people agree with each other. An ICLR 2025 paper shows where that shortcut hits a hard limit: right where teams lean on it most.
LLM-as-a-judge is now the default way AI teams check their own outputs. One model scores another model's answer against a written rubric. The case for it goes back to a 2023 paper from the team behind Chatbot Arena: GPT-4 as a judge agreed with human preferences over 80% of the time, about the same rate humans agree with each other. That's close enough to human review, at a fraction of the cost, that most eval pipelines now assume it holds.
A paper at ICLR 2025 draws a hard line around when it doesn't. Florian Dorner, Vivian Nastl and Moritz Hardt showed that when the judge is no more accurate than the model it's grading, no debiasing method can cut the human labels you need by more than half. That limit lands right where teams reach for a judge most, when they're testing a model new enough that nothing clearly outperforms it.
The 2023 paper also named the ways a judge goes wrong. Position bias, where it favors whichever answer it saw first. Verbosity bias, where longer reads as better. Self-enhancement, where it rates its own model family higher. None of these show up as errors. They show up as scores, and a confident wrong number is exactly what downstream decisions get built on.
So the setup I'd ship is a stack. Deterministic checks for anything code can verify. An LLM judge against a rubric for anything that needs reasoning. And a human on whatever slice either layer flags as uncertain. A judge multiplies human labels. It doesn't retire them.
I learned a version of this running QA on LLM training data pipelines. A quality score is only as trustworthy as whoever produced it, so the reviewers get checked along with the work. The industry is rebuilding that same discipline now, one layer up, with a model sitting in the reviewer's seat.
The mechanics are simple. Before a judge ships, have people label a sample of real outputs against the same rubric. Measure how often the judge agrees with them, usually with Cohen's kappa. Set the bar by the stakes. A judge grading marketing tone can live with looser agreement than one grading what a model tells a customer about their account. Ship a judge nobody checked and you've shipped a rubric with a vibe attached.
Calibration isn't a one-time check either. Judge models get updated, prompts drift, and the outputs being graded change. Re-run the human comparison every time you swap the judge, and on a fixed schedule regardless. Track agreement over time as its own metric.
That said, this overhead doesn't belong on every rubric equally. I've written before that every agent needs a blast radius before it ships. Judge calibration needs the same proportionality. Size the sample and the recheck schedule to what breaks if the judge is wrong.
Someone still has to own that call, and it shouldn't default to the model team. The team that wrote the acceptance criteria the judge now enforces should certify the judge is enforcing them correctly, with a number they'd defend in a room. Evals answered "does this feature work?" Calibration answers "is the thing checking it telling the truth?" Skip the second question and the blind spot just moves one layer further from where anyone is looking.