Lost in the Middle: How Language Models Use Long Contexts

By 2023 language models accepted 16,000, then 100,000 tokens of input, and the natural move was to give them more: more retrieved documents, longer chat histories, whole contracts. The authors asked a simple question nobody had measured carefully. Does a model use everything in its context equally well?

They built a test where only one thing changes. A question from Natural Questions comes with 10, 20 or 30 Wikipedia passages; exactly one contains the answer, and the others are relevant-looking distractors from a search engine. They move the answer passage from first to last and measure accuracy at each position, across six models including GPT-3.5-Turbo and Claude-1.3, plus their extended-context versions.

Every model showed the same shape: a U. Accuracy was highest when the answer was in the first passage, recovered somewhat when it was last, and sagged in the middle. For GPT-3.5-Turbo with 20 passages, accuracy fell from 75.8% at the start to 53.8% in the middle. With no passages at all, answering from memory, the same model scored 56.1%. Burying the answer in the middle was worse than giving the model nothing.

Two more findings complete the picture. The 16K and 100K versions of each model did no better than the 4K and 8K versions when the input fit both. And in a realistic retrieval setup, going from 20 to 50 retrieved passages added only about 1.5 points while making the prompt two and a half times longer.

"The model has a million-token window, so just put everything in" is one of the most common requests on enterprise projects. This paper is the evidence for why that is a design decision, not a free win.

Position is a variable you control. A model's accuracy on the same evidence moved by 20 points depending only on where the evidence sat. Your prompt template decides positions: system instructions first, retrieved passages in some order, history, then the question. Put the most important material at the start or end, and keep the question near the evidence.

More context can lower accuracy. Every extra passage pushes the useful ones further from the edges and gives the model more to sort through. The retrieval experiment shows retrieval recall still climbing at 50 documents while model accuracy had already flattened. Past a point you pay more for tokens and latency and get nothing back.

Bigger windows did not mean better reading. Extended-context models matched their shorter siblings. A larger window raises the ceiling on what you can send; it does not improve how well the model uses what you send. Test long-context claims on the customer's data, with the evidence deliberately placed in the middle.