ThunderAgent: A Simple, Fast and Program-Aware Agentic Inference System

A coding or research agent loops: think on the GPU, call a tool (run tests, query an API, browse), append the result, think again. Its context only grows, so each turn could reuse almost all of the previous turn's KV cache. Today's stacks pair an inference engine (vLLM, SGLang) with a tool orchestrator (Kubernetes) and treat each turn as an independent request.

The authors measured three problems with that split on GLM-4.6 agents. KV cache thrashing: under memory pressure the engine evicts caches of agents waiting on tools to make room for others; when the tool returns, the whole history must be prefilled again, raising per-request latency up to 7.14x. Memory imbalance: routers that pin an agent's requests to one node for cache locality let nodes drift apart, up to 51% in a 90-minute rollout. Tool lifecycle blindness: finished agents' sandboxes and images stay on disk until it fills, and sandbox setup time grows with concurrency.

ThunderAgent adds a scheduling layer that knows about programs. Each request carries a program ID; the scheduler tracks each program's context length, phase (reasoning or acting) and status (active, paused, terminated). Every 5 seconds it checks each backend. If active programs need more KV than the backend holds, it pauses programs, acting ones and shorter ones first, freeing their caches; when room returns it restores waiting programs, reasoning ones and shorter ones first. Paused programs wait in one global queue and can resume on any node, which fixes imbalance. Acting programs' memory claims decay over time, so long tool calls give up their cache before short ones. Tool environments are prepared ahead of a program's restore and deleted when it ends.

On H100s, serving throughput was 1.48-3.58x vLLM's across SWE-agent, OpenHands, ToolOrchestra and a science agent; RL rollout was 1.79-3.92x faster than vLLM with SGLang's router; disk use fell 4.2x. It is open source and has been adopted in SkyRL.

Agent workloads look different from chat: long, growing contexts, long pauses in tools, and many concurrent sessions in batch jobs, evaluations and RL. A stack tuned for chat can lose most of its throughput to recomputation as concurrency rises.

Throughput can fall as you add agents. In the paper's vLLM baseline, adding parallel workflows past memory capacity reduced total throughput. Find the knee before scaling a customer's agent fleet, or use a scheduler that caps active programs automatically.

Measure prefix-cache hit rate per agent turn. Agents should hit near 100% because each turn extends the last. A falling hit rate under load means caches are being evicted during tool calls.

Tool infrastructure is part of inference cost. Sandbox setup time and leaked environments limit sustained throughput just as GPU memory does; cleanup tied to workflow completion is the fix.