RouteLLM: Learning to Route LLMs with Preference Data

Large models answer hard questions better, but the price gap between a frontier model and a small one can exceed 50× per token. Many queries, such as greetings, simple lookups and short rewrites, are answered just as well by the small model. A router that sends each query to the cheapest model that will handle it well can cut cost without hurting quality, provided it can tell easy from hard before any model answers.

RouteLLM, from Berkeley and Anyscale (Isaac Ong, Amjad Almahairi, Ion Stoica and colleagues), frames this as learning from preferences. Chatbot Arena collects human votes on which of two models' answers was better for a real user query. From about 65,000 such comparisons across 64 models, grouped into 10 strength tiers, the authors train a win-prediction model: given a query, the probability that a strong model's answer would beat a weak model's. A threshold on that probability decides routing, so one setting trades cost for quality. Each query goes to exactly one model, unlike cascades that try the cheap model first and escalate, which add latency.

They compare four router designs: a similarity-weighted Elo ranking computed at query time, matrix factorization over query and model embeddings, a BERT classifier, and a classifier fine-tuned from Llama 3 8B. Arena data alone worked for open-ended chat but routed math and multiple-choice questions no better than random, because those look unlike Arena traffic. Augmenting with about 120,000 GPT-4-judged comparisons (costing about $700), or with roughly 1,500 gold-labeled MMLU validation items, fixed that.

Routing between GPT-4 and Mixtral-8x7B, the best router reached 95% of GPT-4's MT Bench score while sending 13.4% of queries to GPT-4, against 49% for a random router: 3.66× fewer expensive calls. The same routers, without retraining, worked for Claude 3 Opus with Sonnet and for Llama 3.1 70B with 8B.

Routing is one of the most direct levers on a customer's AI bill. This paper shows it works, and also why a router trained on someone else's traffic can fail on yours with no visible error.

Most savings come from knowing your query mix. Routers trained on general chat routed math and exam questions no better than random. A router has to be trained or at least validated on the customer's queries. The fix was small: under 2% in-domain labeled data restored good routing on MMLU.

LLM judges make router training cheap. Labeling which model's answer is better can be automated with a strong model as judge, checked against a sample of human labels. The authors built 120,000 labels for about $700.

Define quality relative to the strong model. The paper's metric, the share of the quality gap between weak and strong models recovered, makes the trade-off concrete: "keep 95% of the strong model's quality at the lowest cost" is a target a customer can approve.

A router is cheap to run. The most expensive router added at most 0.4% to GPT-4 generation cost, and the fastest handled over 150 requests per second on one L4 GPU.

Routers outlive model upgrades, partly. Routers generalized to new strong/weak pairs without retraining, because they learn what makes a query hard. Still re-validate when either model changes.