An LLM Compiler for Parallel Function Calling

Ask an agent "would Stanford and UCLA together have more Nobel laureates than Berkeley?" and a ReAct agent will search for Stanford, wait, think, search for UCLA, wait, think, search for Berkeley, then do the arithmetic. The three searches do not depend on each other, but each one waits for the last, and each wait includes a full model call over a growing history. Sehoon Kim, Amir Gholami and colleagues at UC Berkeley noticed that this is the problem compilers solved long ago: find the dependencies, then schedule independent work in parallel.

LLMCompiler has three parts. A function-calling planner reads the request and writes a list of tasks, each a tool call whose arguments may reference earlier tasks' outputs as $1, $2 and so on. That list is a directed acyclic graph. A task fetching unit watches the graph, replaces references with actual values as they arrive, and dispatches every task whose dependencies are done. An executor runs dispatched tasks in parallel. For problems where the right steps depend on intermediate results, a joiner can either answer or ask the planner to replan with what is known so far.

On HotpotQA, a movie-recommendation task and the authors' own ParallelQA set, LLMCompiler with GPT models was 1.8× to 3.7× faster and 3.4× to 6.7× cheaper than ReAct, with accuracy within a few points, and beat OpenAI's parallel function calling, released at the same time, on latency and cost. It also ran the Game of 24 faster than Tree of Thoughts and, with replanning, beat ReAct and other agents on WebShop success rate.

Latency is often the complaint that kills an agent rollout: users will not wait 20 seconds for an answer a person could look up in ten. Much of that time is the agent waiting on tool calls one at a time. This paper shows how much of it is avoidable.

Look for independent calls. Fetching the same data for several entities, querying several systems for one customer, checking several policies: these fan out naturally. Running them in parallel can cut response time by more than half.

Plan once, not per step. A single planning call that writes the whole graph is cheaper than a model call between every tool call, which is where the 3× to 7× cost savings came from.

Keep a replanning path. Some workflows cannot be planned in full up front, such as shopping through search results. A planner that can revise its graph after seeing results handles these without falling back to one step at a time.

Parallel calls change load. Fanning out ten calls at once hits downstream systems harder than ten calls over ten seconds. Agree rate limits and timeouts with the system owners, and set deadlines so one slow call does not hold the whole answer.

Accuracy is not the selling point. With GPT models, accuracy stayed about the same. Sell this as speed and cost, measured on the customer's own requests.