IN THIS BREAKDOWN · 10 SECTIONS
You can read 24% of this breakdown.
The paper in one minute
Calling an API correctly from a request like "transcribe this recording with a Torch Hub model" requires picking the right library, the right repository and model name, and the right arguments. General models in 2023 often invented names or chose the wrong framework. Gorilla tries to fix this with fine-tuning and retrieval.
The authors scrape model cards from three hubs into APIBench: every Torch Hub API (94 calls), TensorFlow Hub v2 after filtering (626) and the 20 most-downloaded HuggingFace models in each task category (925), for 1,645 APIs. Each card becomes a JSON record with domain, framework, functionality, API name, call, arguments, requirements, example code and description. GPT-4, given three hand-written examples and one API's documentation, writes 10 user instructions that call for that API without naming it: 16,450 instruction-API pairs.
Gorilla is LLaMA-7B fine-tuned on those pairs as one-turn conversations, in two variants: plain, and "retriever-aware", where the training prompt includes the relevant API documentation. At inference, the retriever-aware model gets the top document from BM25, from an OpenAI embedding index, or from an oracle that always returns the right one.
Calls are scored by parsing them into an abstract syntax tree and matching the call and key arguments against the database. A call that matches no API is a hallucination; one that matches the wrong API is an error.
Zero-shot, Gorilla matched the reference more often than GPT-4, GPT-3.5, Claude and LLaMA on all three hubs and hallucinated less. With the oracle document, the retriever-aware model reached 67% on Torch Hub, 91% on HuggingFace and 94% on TensorFlow Hub, and could follow documentation changes at test time. With real retrievers its accuracy dropped a lot, and the authors conclude that retrieval quality decides whether retrieval helps.
Research notes: the problem, approach and results
Torch Hub calls matching the reference, Gorilla fine-tuned vs GPT-4 zero-shot, on a test set of 186 (Table 1)
Gorilla on HuggingFace without retrieval vs the retrieval-trained model given BM25's top document (Table 2)
differences quoted in the text that do not match Table 2, all on HuggingFace (our check)
Language models are poor at writing correct API calls from a natural-language request when there are thousands of overlapping APIs: they invent model names or repositories, or pick the wrong library. Prompting cannot fit every API's documentation into context.
Build APIBench from 1,645 model-hub APIs, generate 10 synthetic user instructions per API with GPT-4, and fine-tune LLaMA-7B on the pairs, optionally with retrieved documentation appended. Score calls by matching their abstract syntax tree against a database, separating hallucinated APIs from wrong ones.
Fine-tuned Gorilla beat GPT-4 zero-shot on all three hubs (for example 59% against 39% on Torch Hub) and hallucinated less. Trained with documentation and given the right document, it reached 67% to 94%. With an imperfect retriever, scores fell sharply, sometimes below no retrieval at all.
Why it matters to an FDE
Fine-tuning on a customer's API surface works. A small model trained on synthetic instructions for a fixed set of internal APIs can call them more reliably than a large general model that has never seen them. For a customer with a stable, large internal API catalog, this is a realistic path, and the data recipe (documentation in, generated instructions out) is cheap.
Retrieval quality decides the outcome. The retriever-aware model with the correct document was the best configuration in the paper. With BM25's top document on HuggingFace, the same model scored 17%, below the plain model's 72% without any retrieval. Before adding documentation retrieval to a tool-calling system, measure whether the retriever returns the right API's documentation, and fall back when it is unsure.
Evaluate on unseen requests and unseen APIs. The test instructions were split from the same generated pairs as the training data, so every test API had training examples. That measures memorization of a fixed catalog, which is fine if the catalog is fixed. If new APIs appear weekly, hold out whole APIs.
Separate hallucinated tools from wrong tools. The paper's split between calls to APIs that do not exist and calls to the wrong real API is a useful pair of metrics. They need different fixes: constrained decoding or validation against a registry for the first, better retrieval or disambiguation for the second.
Validate calls before executing them. AST matching against a registry is also a runtime guard: parse the generated call, check it against known APIs and argument schemas, and reject or repair anything that does not match.
Understand the mechanism. Try it on a real problem.
Unlock this breakdown and 290 more papers.
Premium opens every paper in full, plus every answer, concept, course and guide on the site. One payment for six months of access.
6 months · One payment · No auto-renewal
- ◇1 architecture diagram
- </>1 runnable code experiment
- ▦2 results tables
- ?5 interview questions with answers
- 03Key takeaways
APIBench, the training recipe, AST scoring, results with and without retrieval, two misreadings
Free account - 04How it works
Dataset construction, instruction generation, two training variants, retrieval and AST matching
Premium - 05The numbers that matter
Table 1 by hub, hallucination rates, Table 2's retriever effects, constraints and test-set size
Premium - 06Try it yourself
Recover the test-set size, recheck the text against Table 2, size the leakage, run an AST matcher
Premium - 07Where it breaks
Seen APIs at test time, generated instructions, mismatched text, baseline scoring, one-call tasks
Premium - 08Using it on a deployment
When to fine-tune on an API catalog, how to retrieve docs, how to validate calls before running
Premium - 09Interview angles
Five questions this paper prepares you for, with answer hints
Premium - 10Read next
Where to go from here on this site
Premium