LLM-as-a-Judge: How It Works and When It Fails

Grading LLM output by hand does not scale. The moment you have more than a handful of test cases, reading every answer on every change becomes the bottleneck, so most teams quietly stop grading and ship on feel. LLM-as-a-judge fixes the throughput problem by having a strong model grade the answers for you. It is the highest-leverage technique in evaluation, and also the easiest to trust blindly and get wrong.

#How it actually works

You give a capable model three things: the original question, the answer you want graded, and a rubric that spells out what good looks like. The judge returns a score and, ideally, a short reason. That reason is not decoration. It is how you catch a judge grading on the wrong thing, like rewarding a confident tone over a correct fact.

The most reliable shape is not an absolute score. Asking a judge to rate one answer from one to ten produces numbers that drift run to run. Asking it which of two answers is better, a pairwise comparison, is steadier, because relative judgments are easier than absolute ones. When you can, judge by comparison and build a ranking from there.

#The biases that quietly wreck it

Judges have consistent, well-documented failure modes. They show position bias, favoring whichever answer they saw first. They show verbosity bias, reading longer as better. And they show self-preference, rating answers from their own model family higher. None of these announce themselves. An uncalibrated judge hands you confident, consistent, wrong scores, which is worse than no scores because you trust them.

#How to calibrate a judge you can trust

Never trust a judge you have not checked against humans. Take thirty or so cases, grade them yourself, then have the judge grade the same set and measure how often it agrees with you. If agreement is poor, your rubric is too vague, so tighten it. Randomize the order of answers to neutralize position bias. Force a structured rubric with explicit criteria rather than a vibe score. Once the judge tracks your own judgment on a set you both graded, you can turn it loose on thousands of cases you will never read.

#The bottom line

Used with care, an LLM judge is the thing that makes nuanced evaluation possible at all, scaling human-quality judgment to a volume no person could read. Used carelessly, it launders a bias into a metric your whole team starts trusting. The difference is entirely in the calibration step, and it takes an afternoon. Do that afternoon before you wire a judge into anything that blocks a release.