The Context Window

TL;DR: The context window is the model's working memory measured in tokens, and the prompt plus the generated answer must both fit inside it. Treat it as a budget you allocate, not free space you fill, because more tokens cost more money, add latency, and can bury the one fact that mattered in the middle where the model attends to it least.

What it actually is

A language model does not read characters or words. It reads tokens, and it can only hold a fixed number of them in view at any one time. That ceiling is the context window. If a model advertises a 128k context window, that is the total number of tokens it can process in a single forward pass.

The point most people miss: input and output draw from the same pool. If your window is 128k tokens and your prompt is already 126k, you have at most 2k tokens left for the answer, no matter how much the model wants to say. Run past the limit and the request either errors or the API silently truncates the oldest tokens, which is worse because you get a confident answer built on a prompt that quietly lost its first half.

A concrete scenario

You are building a support assistant. The system instructions are 800 tokens. A retrieved knowledge-base article is 6,000 tokens. The customer's chat history is 4,000 tokens. You want room for a 1,500-token reply. That is roughly 12,300 tokens of budget you have explicitly chosen to spend. Now a teammate adds "just include the last 50 messages and the top 20 articles to be safe." Suddenly you are at 90k tokens per call. The reply quality does not improve. The bill and the latency both climb, because cost and time scale with tokens processed, and the model now has to find the relevant article inside a pile of mostly-irrelevant ones.

This is the practical reason RAG and chunking exist. You cannot fit a 500-page manual in the window, and even if you could, you should not want to. Retrieve the three passages that matter and spend your budget on those.

The long-context traps

Bigger windows tempt you to stop being selective. Two failure modes bite:

• Lost in the middle. Models reliably attend to the start and end of a long context and get measurably worse at using facts buried in the middle. A relevant document at position 60 of 100 may be functionally invisible. Liu et al. (2023, 'Lost in the Middle: How Language Models Use Long Contexts') measured it on multi-document question answering and key-value retrieval: accuracy was highest with the relevant passage at the beginning or end of the input and significantly lower with it in the middle, including for models built explicitly for long context. Put the most important context near the top or bottom.
• Cost and latency scale with tokens. Attention work grows faster than linearly with sequence length, so a 100k-token prompt is not just "more"; it is disproportionately slower and pricier per call. A long window is a capability, not a default setting.

The spine below is how you spend that budget deliberately, in the order you decide it, ending at a number you chose rather than a limit you discovered.

In the flowchart three arrows enter the window and one leaves it, and the one that leaves draws from the same box. That is the production bug in one picture: an input that fills the box leaves the output nowhere to go.

Why interviewers probe this

The answer that loses arrives as good news: "the new model has a million tokens, so we can paste everything in." The candidate who says it has read about the window and never paid for one. It is a fast proxy for whether you have actually shipped an LLM feature. The candidate who says "I'll just use a model with a huge context window" without mentioning cost, latency, or lost-in-the-middle has read about the window but never paid for one. The follow-up they hold in reserve is "your prompt is 200k tokens and the model is 128k, what do you cut and how?" The strong answer names a concrete budget allocation: pin the system prompt, retrieve the top-k passages instead of dumping the corpus, summarize or truncate old history, and reserve a fixed slice for the output.

The support assistant from the scenario, sized twice:

Same reply quality, roughly seven times the cost and the latency, and the one relevant article now sits somewhere in the middle of nineteen irrelevant ones.

Common misconceptions

• "The context window is just the prompt size." It is prompt plus completion. Forgetting to reserve room for the answer is a common production bug.
• "A token is a word." Tokens are sub-word pieces. A rough rule is about 0.75 words per token for English, but code, JSON, and other languages tokenize very differently, so always measure rather than guess.
• "A bigger window means I can stop doing RAG." Stuffing everything in is slower, costlier, and triggers lost-in-the-middle. Retrieval is about precision, not just fitting under the limit.
• "Hitting the limit fails loudly." Many setups truncate silently. You can get a plausible answer from a prompt that lost its most important half.

Key takeaways

• The context window is a fixed token budget that input and output share; reserve space for the completion explicitly.
• Cost and latency rise with token count, and attention cost grows faster than linearly, so long prompts are expensive in both.
• Models attend worst to the middle of a long context; place critical information at the start or end.
• The window's size is why RAG and chunking exist: retrieve the few passages that matter instead of paying to process a corpus.
• Write the budget down before the first call: system, retrieved, history, reserved output. A number you chose beats a limit you discovered.