Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena

By 2023, chat models were judged mostly on benchmarks like MMLU, where a question has one right letter. Those benchmarks could not tell an instruction-tuned assistant from its base model, even though people strongly preferred the assistant. Open-ended answers needed human judgment, and human judgment is slow and costly.

The team behind the Vicuna model asked whether a strong model could stand in for the human. They built two datasets with human preferences to check against. MT-bench is 80 hard two-turn questions in eight categories, with about 3,000 votes from 58 expert raters, mostly graduate students. Chatbot Arena is a public site where people chat with two anonymous models side by side and vote for the better one; the paper used 3,000 of its 30,000 votes.

They then tested GPT-4, GPT-3.5 and Claude as judges in three formats: compare two answers, grade one answer on a scale, or compare against a reference answer. The headline result is that GPT-4's verdicts matched expert humans as often as experts matched each other: 85% against 81% when ties are left out.

The more useful half of the paper is the list of ways the judges fail. They prefer whichever answer is shown first. They prefer longer answers even when the extra length adds nothing. They mark wrong math answers as correct even when they can solve the problem themselves. Each comes with a measured mitigation. This paper is why "use a model as the grader" became standard practice, and why careful teams still check the grader.

Almost every customer project ends up grading open-ended outputs with a model: summaries, answers from documents, agent replies. This paper is the manual for doing it responsibly.

Read the agreement number carefully. The famous "over 80%" agreement excludes ties. Count ties and verdicts that flipped when the answers were swapped, and GPT-4 agreed with experts 66% of the time, while experts agreed with each other 63%. Both framings are true. The second is the one to remember when a judge's score is the only thing standing between you and a launch decision.

Judges are weakest on close calls. Agreement with humans rose from about 70% when two models were close in quality to nearly 100% when one was much better. Most decisions on a project are close calls: prompt A against prompt B, this month's model against last month's. Those are exactly the comparisons where a single judge's verdict is least reliable.

Every bias has a cheap fix. Judge both orders and call it a tie if the verdict flips. Tell the judge length is not quality and test it with padded answers. For anything with a correct answer, have the judge solve it first, or give it the reference. None of this costs more than a second judge call.