Reliability without Validity: A Systematic, Large-Scale Evaluation of LLM-as-a-Judge Models Across Agreement, Consistency, and Bias

This paper evaluates language-model judges across reference agreement, repeatability and bias checks. Its useful central point is straightforward: a judge can produce the same verdict repeatedly while applying an undesirable rule. For example, always choosing the first answer is perfectly repeatable but fails when the same answers change position.

The study compares an overall cohort of 21 judges across three benchmarks and recommends a validation protocol that includes repeated runs and paired answer-order swaps. It also argues for chance-corrected agreement rather than relying on a raw agreement percentage alone.

There is a material caveat. The paper's exact-match measure excludes human ties, while its discussion compares that measure with kappa and attributes the gap to chance correction. Its numerical explanation for a balanced three-label example is also inconsistent with the stated kappa formula. This breakdown preserves the useful protocol while making those limitations explicit.

Treat the June 2026 report as a methodological reading, not a current purchasing leaderboard. Its measurements describe a particular spring evaluation window, rubric and set of model configurations. A production judge needs validation against the decisions your application actually asks it to make.

An automated judge may decide whether a RAG answer is grounded, whether an agent completed a task or whether a new prompt is better. If that judge favors answer position or a familiar style, an optimization loop can improve the score without improving the customer's outcome.

Repeated runs catch instability but cannot establish correctness. Agreement with human labels is also only as useful as the label quality and task coverage. A judge that agrees on routine cases may still miss a rare but expensive failure.

Use LLM-as-a-judge together with evaluation gates. Qualify the evaluator as a component of the system: define the rubric, validate the labels, test transformations that should not change a verdict, and retain human review for consequential disagreements.