IN THIS BREAKDOWN · 10 SECTIONS
You can read 17% of this breakdown.
The paper in one minute
In early 2023, a company could call a dozen LLM APIs whose prices differed by two orders of magnitude: 10 million input tokens cost $30 on GPT-4 and $0.20 on GPT-J hosted by Textsynth. The paper's example: a customer-service bot answering 360,000 questions a month with 1,800-token prompts on GPT-4 costs about $21,000 a month.
The authors sort ways to spend less into three families. Prompt adaptation shrinks what you send: fewer in-context examples, or several queries sharing one prompt. LLM approximation replaces the expensive call: cache answers to similar queries, or fine-tune a small model on the big model's outputs. LLM cascade chooses which model answers each query.
FrugalGPT implements the cascade. Queries go through an ordered list of APIs. After each answer, a small scoring model (a DistilBERT regressor trained on labeled examples) estimates whether the answer is correct; if the score clears that step's threshold, the answer is returned, otherwise the next API is asked. The list (up to three APIs) and the thresholds are chosen on labeled training data to maximize accuracy under a budget, with a search pruned to model combinations that disagree often enough to be worth combining.
On HEADLINES (gold-price direction from news headlines), the learned cascade was GPT-J, then J1-Large, then GPT-4, with thresholds 0.96 and 0.37. It reached GPT-4's accuracy for $0.6 instead of $33.1, and at one-fifth of GPT-4's cost it was 1.5% more accurate. The reason is complementarity: on about 6% of HEADLINES queries, GPT-4 is wrong and a cheap model is right. Across three datasets, savings at equal accuracy were 59% to 98%.
Research notes: the problem, approach and results
lower cost than GPT-4 at GPT-4's accuracy on HEADLINES, a gold-price news classification task ($33.1 to $0.6)
cost saved while matching the best single API across HEADLINES, OVERRULING and CoQA
price spread for input tokens across the 12 APIs in March 2023 ($0.2 to $30 per 10M tokens)
LLM APIs differed in price by up to two orders of magnitude in 2023 (GPT-4 cost 150 times GPT-J per input token), and high-volume applications sent every query to the best, most expensive model.
Three ways to cut cost: shorter or shared prompts, cheaper approximations (caching, fine-tuning a small model on a big one's answers), and an LLM cascade. FrugalGPT implements the cascade: query a list of APIs in order, score each answer with a small trained model, return it once the score passes a learned threshold; the list and thresholds are optimized on labeled data under a budget.
On three classification-style datasets with 12 APIs, FrugalGPT matched the best single model's accuracy at 59-98% lower cost (GPT-4 on HEADLINES: $33.1 to $0.6), or improved on it by up to 4-5% at the same cost.
Why it matters to an FDE
Cost is usually what stops an LLM pilot from scaling. The cheapest fix rarely involves new infrastructure: route easy requests to cheap models and keep the expensive one for the rest.
Start with labeled data from the customer's traffic. The cascade's thresholds and the scorer are trained on labeled examples from the same distribution. Without them, savings are guesses.
The scorer is the product. Everything depends on a cheap model that can tell when a cheap answer is wrong. In the paper's setting answers are short labels, which makes scoring easy; for open-ended answers it is much harder.
Use all three levers. Prompt trimming and caching often save more with less risk than routing, and they compound with a cascade.
Understand the mechanism. Try it on a real problem.
Unlock this breakdown and 149 more papers.
Premium opens every paper in full, plus every answer, concept, course and guide on the site. One payment for six months of access.
6 months · One payment · No auto-renewal
- ◇1 architecture diagram
- </>1 runnable code experiment
- ▦2 results tables
- ?5 interview questions with answers
- 03Key takeaways
Pricing quirks, complementarity, the cascade's parts, its limits and three misreadings
Free account - 04How it works
The cost model, the cascade loop, the scorer, the optimizer and the experimental setup
Premium - 05The numbers that matter
Prices, savings at equal accuracy, the HEADLINES case and complementarity
Premium - 06Try it yourself
Redo the cost arithmetic, price one query across 12 APIs, and train and stress a cascade
Premium - 07Where it breaks
Short-answer tasks, labeled data needs, scorer drift, latency, stale prices and slips
Premium - 08Using it on a deployment
Collecting labels, building a scorer, setting thresholds, and monitoring for silent failure
Premium - 09Interview angles
Five questions this paper prepares you for, with answer hints
Premium - 10Read next
Where to go from here on this site
Premium