IN THIS BREAKDOWN · 10 SECTIONS
You can read 20% of this breakdown.
The paper in one minute
Query routing sends each request to one of several models: easy ones to cheap models, hard ones to an expensive reference model. In practice the savings are small. When the reference is GPT-4o and the alternatives are 7-8B or small mixture models, one answer from a small model is seldom as good, so a router that wants to protect quality sends nearly everything to GPT-4o. In the paper's tests, two prior routers achieved essentially 0% cost reduction.
Best-of-n sampling changes the comparison. Generate several answers from a cheap model and keep the best; the chosen answer is often as good as the big model's, and with the paper's prices five answers from Phi-3-mini still cost far less than one from GPT-4o (input tokens are charged once when several outputs are requested together). The catch is picking the best answer: an LLM judge is too slow and expensive per request.
BEST-Route (Best-of-n Enhanced Sampling and Test-time Route Optimization) has two parts. A proxy reward model, a 300M DeBERTa-large, is trained with a pairwise ranking loss to order answers the way an 8B judge (armoRM) does, using the worst, median and best of 20 sampled answers per training query. A multi-head router on a 44M DeBERTa-small backbone has one head per (small model, n) pair, each predicting the probability that the best of n answers is at least as good as GPT-4o's single answer. At serving time a user-chosen threshold sets the trade-off: among (model, n) options whose predicted probability clears it, BEST-Route takes the cheapest (using token prices and the model's average output length); if none does, it calls GPT-4o once.
On a new 10,000-query dataset of instructions, coding and safety prompts with eight models, BEST-Route reduced cost by 60% with a 0.80% quality drop measured by armoRM, where the best baseline lost 5.08%. Adding a coding specialist let it be 20% cheaper than GPT-4o on coding while scoring slightly higher.
Research notes: the problem, approach and results
lower cost than always using GPT-4o, with a 0.80% drop in reward-model quality score
smaller quality drop than the best prior router (N-label) at the same 60% cost cut
cheaper per query: best-of-5 from Phi-3-mini vs one GPT-4o answer at the paper's prices (500 tokens in, 300 out; our arithmetic)
Query routers save money by sending easy queries to small models, but a single small-model answer rarely matches a frontier model's, so in practice routers send most traffic to the expensive model and save little. Best-of-n sampling can lift a small model's quality, but judging samples with a large LLM is too slow and costly for serving.
Train a small proxy reward model (300M) to rank answers the way an 8B judge does, so best-of-n is cheap. Train a multi-head router (44M) to predict, for each small model and each sample count n, the probability that the best of n matches one answer from the reference model. At serving time, choose the cheapest (model, n) whose predicted match probability clears a threshold; otherwise call the reference model once.
On a 10,000-query mix of QA, coding and safety prompts with GPT-4o as reference and six cheaper models, BEST-Route cut cost 60% with a 0.80% quality drop, against 5.08% for the best prior router. Routing overhead was 0.04 s plus 0.58 s for the reward model at n = 20.
Why it matters to an FDE
Many customer workloads use a frontier model for every request because a single cheaper answer isn't good enough. BEST-Route's lesson is that "cheaper model" and "more test-time compute" combine: several samples from a small model, filtered by a small judge, can match the big model for a large share of traffic.
The judge sets the ceiling. Best-of-n only helps if the scorer picks the truly better answer. A small, fast reward model trained to agree with a trusted judge on the customer's data is the enabling piece.
Price structure matters. The method depends on paying for the prompt once while sampling several outputs, which self-hosted engines and many APIs allow. Long prompts with short answers favor it most.
Quality is measured by a model. The paper's "under 1% loss" is in reward-model score, not human ratings or task accuracy; validate on the customer's own success measure before switching traffic.
Understand the mechanism. Try it on a real problem.
Unlock this breakdown and 149 more papers.
Premium opens every paper in full, plus every answer, concept, course and guide on the site. One payment for six months of access.
6 months · One payment · No auto-renewal
- ◇1 architecture diagram
- </>1 runnable code experiment
- ▦2 results tables
- ?5 interview questions with answers
- 03Key takeaways
The router and judge design, cost estimation, what drives savings, limits and three misreadings
Free account - 04How it works
The proxy reward model, the multi-head router, the cost estimate and the serving algorithm
Premium - 05The numbers that matter
Main results, coding with a specialist, out-of-distribution, metrics and overhead
Premium - 06Try it yourself
Price best-of-n against GPT-4o, recheck Table 1, and simulate routing with and without sampling
Premium - 07Where it breaks
Judge-based metrics, price assumptions, latency, data needs and slips
Premium - 08Using it on a deployment
Building the judge, collecting router labels, choosing thresholds and validating on real outcomes
Premium - 09Interview angles
Five questions this paper prepares you for, with answer hints
Premium - 10Read next
Where to go from here on this site
Premium