What is LLM-as-a-judge?

LLM-as-a-judge is the practice of using a language model to evaluate another model's output, in place of or alongside a human reviewer. Instead of having a team manually read thousands of responses, a second LLM (often larger or more expensive than the one being evaluated) receives the question, the answer and a judging criterion, and returns a score or a verdict. It has become the standard tool for evals at scale: the human-review bottleneck on high volumes disappears, at the cost of introducing a judge that has limits of its own. The typical use case is a team that needs to evaluate a chatbot's or an AI agent's answers across thousands of conversations a day, a volume no team of human reviewers can cover systematically. It does not fully replace the human eye, it moves it downstream: whoever supervises the process spot-checks the cases where the automated judge is uncertain.

See also: Evaluator independence · The condition that makes the verdict on an AI system credible: whoever grades it must have no stake in the outcome.

Where it works well

It works best when the task is comparative rather than absolute: asking "which of the two answers is better" is more reliable than asking "how much is this answer worth out of 10", because relative comparison reduces judgment variance. It also works well with an explicit rubric, concrete criteria to check one by one (did it cite the source, did it follow the required format, did it not invent data) instead of a generic holistic judgment, which the judge model tends to render vaguer and less reproducible.

The rubric is not neutral

Giving the judge a rubric helps, but it is not a clean fix: the rubric itself carries a bias. An August 2026 study, Judging LLM-as-a-Judge: Concerning Rubric Artifacts (arXiv 2609.02942), shows that the judge's score can be partly predicted from the rubric text alone, without ever seeing the answer being evaluated: how the criterion is worded already contains the verdict, the phenomenon the literature calls a rubric artifact. Two more words are worth knowing before a vendor uses them on you: a rubric bank, a set of criteria selected and calibrated per task rather than written by hand once (the CalibratedRubric paper, arXiv 2607.29252, measures how much criterion selection moves agreement with human judgment), and a self-evolving rubric, one rewritten automatically as new failure modes show up. Then there is deferral: cases where the small judge is uncertain get routed to a larger judge or to a person, so you pay for the expensive model only where it counts. These are still research ideas rather than common practice, so the operational caution is a single one: treat the rubric as part of the instrument to be calibrated, not as the neutral yardstick everything else is calibrated against.

Known biases and how to calibrate them

LLM-as-a-judge has documented, recurring biases: it tends to prefer longer answers even when they are not better (verbosity bias), to favor the first option shown when content is equivalent (position bias), and to rate output generated by its own model family more favorably (self-preference bias). None of these goes away by picking a more powerful judge model: it is mitigated by randomizing option order, using granular rubrics, and above all periodically calibrating the judge against a sample of human judgments. Without that calibration, LLM-as-a-judge measures how consistent the automated judgment is with itself, not how correct it is: a reliable evals setup treats it as a tool to be tuned, not as a source of truth.