Recursive Language Models

Models now accept hundreds of thousands of tokens, but accepting is not using. Quality drops as inputs grow, and it drops fastest on tasks where the answer depends on most of the input. GPT-5 finds a single needle in a million tokens, but on OOLONG-Pairs, where the answer depends on comparing pairs of lines across the whole input, it scored an F1 of 0.1. The usual fix for long agent histories, compaction (summarizing older context), assumes some details can safely be forgotten.

MIT CSAIL proposes Recursive Language Models (RLMs). Do not put the long prompt in the model's context at all. Put it in a variable inside a Python REPL. The root model is told only the prompt's length and a short preview, and how to reach it. It then writes code: peek at a section, split the input into chunks, loop over them calling a model (or another RLM) on each slice, collect results in variables, and finally set an answer variable. From outside, it behaves like a language model, string in, string out, with no practical limit on input or output length.

With GPT-5 and Qwen3-Coder on four long-context tasks, RLMs beat the base models, compaction agents, retrieval-equipped coding agents and Claude Code, often by double digits, at comparable average cost. They handled inputs above 10 million tokens. The authors also fine-tuned Qwen3-8B on 1,000 RLM trajectories and improved it by a median 28%.

The appendix of things that did not work is as useful as the results: the same prompt behaved differently across models, one model tried to make thousands of sub-calls, and sequential sub-calls made everything slow.

Customers bring long inputs: a year of support tickets, a full codebase, every contract in a data room. "The model has a million-token window" is not a plan. This paper gives you a better question and a different architecture.

Ask how the task scales with the input. Finding one fact is constant work regardless of length; the authors' S-NIAH task. Labeling and counting every line is linear work. Comparing pairs of items is quadratic. The more the work grows with the input, the earlier a model's context breaks down. A request like "find every pair of contracts with conflicting renewal terms" is quadratic, and no context window will save it.

Treat the data as an environment. RAG pulls snippets into the window; agents fill the window with tool outputs; compaction shrinks the window. All three keep the window as the bottleneck. RLMs move the data out entirely and let code decide what each model call sees. That is a familiar pattern for engineers: map over chunks, store intermediate results, reduce.

Budget for variance and guardrails. Average costs were comparable to the alternatives, but the spread was wide; in some cells the standard deviation exceeded the mean. A model writing loops that call models can make thousands of calls. Production versions need hard limits on sub-calls, depth and spend, a sandboxed execution environment, and logging of every trajectory.