Scaling Long-Horizon LLM Agent via Context-Folding

A ReAct agent appends every thought, tool call and tool output to one growing context. On long tasks, such as deep research with dozens of searches or a coding task that reads many files, that context becomes huge, and models use long contexts poorly and slowly. Weiwei Sun, Jiecao Chen and colleagues at ByteDance Seed, CMU and Stanford propose context folding: let the agent manage its own context with two tools.

branch(description, prompt) starts a sub-task in a separate working context. return(message) ends the branch: all of its intermediate steps are removed from the context, and only the message, a short account of what was found, is appended to the main thread. A search-heavy subtask like "find the publication with these authors" can take thousands of tokens inside its branch and leave a two-line result behind. Because the branch starts from the main thread's prefix, the KV cache can roll back to that point on return, so folding is cheap at inference.

The authors use a plan-execute pattern: the main thread plans and launches branches; branches execute, and cannot create further branches. Prompting alone helps a little, but the agent tends to do heavy work in the main thread or forget to return. So they train it with FoldGRPO, which applies the fold during reinforcement learning rollouts and adds token-level penalties: tokens in the main thread are penalized once it passes half of its 32K budget, a branch judged off-task by GPT-5-nano is penalized, and failed tool calls are penalized.

On BrowseComp-Plus (deep research over a fixed corpus) and SWE-Bench Verified, a Seed-OSS-36B folding agent with a 32K window and up to 10 branches scored 0.620 and 0.580. That beat the same model as a ReAct agent with a 327K context, trained with the same RL, which scored 0.540 and 0.574, and a summarization agent. During training the agent learned to call more tools and create more branches, and its main thread shrank to about 8,000 tokens.

Every long-running agent you deploy hits the context problem: logs, documents, search results and diffs pile up. Context folding is a clean pattern you can apply even without training.

Do heavy reading in a side context. Run "search the knowledge base", "read these five files" or "inspect this log" as a sub-task whose working context is thrown away, keeping only a written result. This is how many coding agents already use sub-agents.

The result message is the memory. Whatever the branch does not write into its return message is gone. Prompt for results that state what was found, what was not found, and where it came from, so the main thread can decide what to do next.

Folding saves money as well as context. Every call re-reads the context. Keeping the main thread small cuts input tokens over a long task several-fold, and the KV prefix stays cacheable.

Training made the difference, and it is not portable. The best results came after reinforcement learning on the target task family. With prompting alone, the folding agent was worse than the long-context agent. Expect a gap if you use the pattern with an off-the-shelf model.