Parrot: Efficient Serving of LLM-based Applications with Semantic Variable

LLM applications rarely make one call. A document summarizer maps over chunks and reduces; a chat search rewrites the query, retrieves, checks safety and answers; a multi-agent coder runs architect, coders and reviewers in rounds. Each step is a separate completion request to a public LLM service, which knows nothing about how they connect.

The authors identify three costs of that blindness. Round trips: for dependent steps the client must receive one output, build the next prompt and send it, paying network latency and a fresh place in the queue each time; in one production chain application, 30% to 50% of call latency was outside the GPU. Wrong objectives: the service minimizes each request's latency, but a map stage should be batched for throughput because only the final answer's latency matters. Missed sharing: long system prompts and growing conversation histories repeat across requests (94% of tokens in a production chat search), but finding shared prefixes by token matching across a cluster is too expensive.

Parrot adds one abstraction. A Semantic Variable is a marked region of a prompt template, an input or an output, such as input:task or output:code (written between double braces in Parrot's templates). Applications submit all their templated requests up front and fetch final outputs with a goal (latency or throughput). From the variables the service builds a request graph and can: run dependent requests itself as soon as inputs are ready; deduce objectives, marking parallel steps that feed a latency-critical output as a task group to batch together; and hash prompts at variable boundaries to find shared prefixes, co-locate those requests and use a kernel that reads the shared KV once.

With LLaMA 7B and 13B against vLLM-based baselines, Parrot was 11.7x faster on a MetaGPT coding workflow, sustained 12x the request rate for four apps sharing prompts, and was 2.37x faster on map-reduce summaries.

Customers building agents often see slow end-to-end times and blame the model. Parrot's measurements say much of the time can sit between calls: client round trips, re-queuing behind other tenants and per-request scheduling that ignores the workflow.

Measure time outside the model first. Log, per workflow, the time each step spends waiting to be sent, in transit and in the provider's queue. If it's a large share, moving orchestration next to the model or batching steps helps more than a faster model.

Self-hosted stacks can exploit the workflow. With an open model on your own engine, you control the API: submit a whole stage at once, keep dependent steps server-side and group parallel calls. Prefix-sharing engines (SGLang, vLLM's prefix caching) do part of this already.

Templates matter for caching. Prompt structure (static role, quasi-static examples, dynamic input) decides how much KV can be shared. Putting the stable parts first and keeping them byte-identical lets any prefix cache work.