TL;DR: The context window is the token budget shared by prompt, history, retrieved docs, tool results, and output in a single call. The senior move is the gap between advertised size and usable size: cost and latency scale with what you stuff in, and recall degrades in the middle.
How to approach it
The wrong move arrives as good news: "the new model has a million-token context, so we can drop RAG and paste the corpus in." The candidate who agrees has just signed the customer up for the largest bill of the year and given away permissions and citations in the same breath. Give the one-line definition, then spend your time on the gap between the spec sheet and production reality: cost scaling, latency scaling, and degraded attention over long inputs. That gap is where FDE work actually happens.
A strong answer
The context window is the maximum number of tokens the model can attend to in a single call; system prompt, conversation history, retrieved documents, tool results, and the generated output all share it. Modern models range from roughly 32k to 200k tokens, with some advertising 1M or more. For scale: 100k tokens is roughly a 200-page book.
The practical limits matter more than the number:
- Cost scales linearly with what you stuff in. Filling a 200k window costs 200k input tokens per call. A chat app that naively resends full history grows its per-message cost every turn, a classic bill surprise FDEs get called in for.
- Latency scales with input. Prefill time grows with prompt length; pushing 150k tokens can add seconds to time-to-first-token. Long context is rarely compatible with snappy UX.
- Attention quality degrades. Models exhibit "lost in the middle": recall is strongest for content near the start and end of the context and weaker in the middle. Needle-in-a-haystack benchmarks look great; real multi-fact reasoning over 100k tokens does not. The effect has a paper: Liu et al. (2023, 'Lost in the Middle: How Language Models Use Long Contexts') tested multi-document question answering and key-value retrieval and found accuracy highest when the relevant passage sat at the beginning or end of the input and significantly lower when it sat in the middle, including for models built explicitly for long context. Treat advertised context as a ceiling, not a working budget.
- It is not memory. The model is stateless between calls; "remembering" prior turns means re-sending them. Everything it appears to remember, you are paying to repeat. Anything outside the window effectively does not exist, which is why conversation summarization and external memory patterns exist.
The three forces pull against each other as you fill the window:
In the second chart the shaded band is the argument. Cost and latency climb the whole way to the advertised ceiling, and recall turns down before you get there, so the usable budget ends where the red line starts falling, not where the spec sheet ends.
It helps to see the budget worked once, the way you would size it in a design review. Say you have a 128k window and you are building doc-QA. Reserve 4k for the output (the model needs room to answer), spend 2k on the system prompt and tool definitions, 6k on conversation history, 300 on the user's question. That leaves roughly 115k for retrieved context, which at 600-token chunks is about 190 chunks, far more than retrieval quality can usefully rank. In practice you pass 10 to 30 good chunks and the window is nowhere near full, which is the point: the constraint that binds first is almost never the window, it is cost, latency, and the model's ability to use what you passed.
The cost mechanics deserve one honest set of numbers, because "resending history" sounds abstract until you price it. Take a support chat where each turn adds about 500 tokens and the app naively re-sends everything (illustrative rates of $3 per million input tokens):
| Turn | Context re-sent | Cumulative input paid | Cost so far |
|---|---|---|---|
| 1 | 500 | 500 | ~$0.0015 |
| 10 | 5,000 | ~27,500 | ~$0.08 |
| 50 | 25,000 | ~637,000 | ~$1.91 |
The per-conversation cost grows with the square of its length, because turn N re-pays for all N-1 turns before it. One long-running customer conversation can quietly cost hundreds of times a short one. This is why prompt caching, history summarization, and trimming exist, and why "our LLM bill doubled but traffic didn't" is one of the most common calls an FDE gets.
The FDE-relevant conclusion: a big context window does not eliminate RAG. Even with 1M tokens, you can't fit a 10M-document corpus, you don't want to pay to re-send a corpus per query, and retrieval gives you access control and citations. Long context changes the retrieval budget (you can pass 20 generous chunks instead of 3 stingy ones), not the architecture.
What interviewers probe next
- "Customer says: the new model has 1M context, can we delete our RAG pipeline?" No: cost per query, latency, permissions, freshness, and citation needs all still point to retrieval; long context just relaxes chunk-selection pressure.
- "What do you do when a conversation exceeds the window?" Sliding window over recent turns plus a running summary of older ones; keep the system prompt and key facts pinned.
- "How would you verify a model actually uses content deep in the context?" Build a small eval placing known facts at varying depths and measure recall by position, rather than trusting the benchmark.
The four things people reach for when the window gets bigger, and why each one costs money:
| What people reach for | Why it fails | What to say instead |
|---|---|---|
| "1M context means no more RAG" | A 10M-document corpus does not fit, re-sending a corpus per query is the bill, and permissions and citations live in retrieval | "Long context relaxes the retrieval budget; it does not replace retrieval" |
| "The advertised size is the working budget" | Recall sags in the middle of long inputs while cost and latency climb with every token | "The spec is a ceiling; size the working budget below it" |
| "The model remembers the conversation" | It is stateless; every turn re-sends history, so cost grows with the square of conversation length | "Sliding window plus a running summary, key facts pinned" |
| "Just pass more chunks" | Past 10 to 30 good chunks the limit is retrieval quality and the model's use of what you passed | "Fewer, better chunks, and measure recall by position" |
Common mistakes
- Stopping at the definition. The question is easy; the differentiation is entirely in the production caveats.
- Confusing context window with training data ("it knows our docs now") or with persistent memory across sessions.
- Not knowing that output tokens share the window; a maxed-out prompt leaves no room for the answer.
- Quoting context sizes confidently but having no idea what 100k tokens costs or how it affects latency. Interviewers at OpenAI and Anthropic reliably pull this thread.
What to actually do
Size the window in a design review the way the worked example does: reserve the output first, then the system prompt, history and question, and only then retrieval. Cap retrieved chunks at what retrieval can rank well, 10 to 30, and place the most important ones first and last. Price the conversation at turn 50, not turn 1, and put prompt caching or a summary strategy in before launch rather than after the bill. Build a small recall-by-position eval on the customer's own documents before believing any needle-in-a-haystack number.
Key takeaways
- The window is one shared token budget: prompt, history, tools, and output all compete for it.
- Advertised context is a ceiling, not a working budget; recall sags in the middle (lost in the middle).
- Long context relaxes the retrieval budget but does not replace RAG, because of cost, permissions, freshness, and citations.