TL;DR: Lay out the spine (ingest, chunk, embed, index, retrieve, rerank, generate), then make the senior move: most failures live in chunking and retrieval, and the one diagnostic question "was the answer in the retrieved context?" splits a retrieval bug from a generation bug.
How to approach it
The move that fails this question is an eager one: the customer says the answers are wrong, and the candidate starts rewriting the prompt. Before touching anything, split the failure, because a chunk that was never retrieved and a chunk that was retrieved and ignored have nothing in common, and the fix for one does nothing for the other. Don't recite a blog post. Lay out the spine in one breath, ingest to chunk to embed to index to retrieve to assemble context to generate to measure, then signal that you know where the bodies are buried: "most failures live in chunking and retrieval, not generation, so I'll spend my time there." Asking one clarifying question ("is this for a chatbot or a batch workflow? Latency budget?") already separates you from candidates who jump straight into vector databases.
A strong answer
Ingestion pulls documents from sources (drive, wiki, tickets), normalizes them to text, and tracks metadata: source, timestamp, ACLs, doc ID. Chunking splits documents into retrievable units, typically 200–800 tokens with 10–20% overlap, ideally respecting structure (headings, paragraphs) rather than blind character counts. Each chunk is embedded (a model like text-embedding-3 or Cohere embed) and stored in a vector index (pgvector, Pinecone, Elasticsearch with dense vectors) alongside keyword/BM25 indexing for hybrid retrieval.
The two containers are the thing to hold on to. Everything in the top one happens once, when a document lands; everything in the bottom one happens again on every single question, which is why cost and latency live down there and quality decisions live up top.
At query time: embed the query, retrieve top-k (k≈20–50), optionally rerank down to 3–8 chunks with a cross-encoder, assemble them into the prompt with instructions like "answer only from the provided context, cite sources," and generate. Log everything, the query, retrieved chunks, and answer, because those logs become your eval set.
The decision diamond at the end of the flowchart is the whole diagnostic. One yes/no on "was the answer in the context" sends you down to a generation bug or across to a retrieval bug, and every fix on one branch is wasted effort on the other.
Where it breaks, in rough order of frequency: (1) retrieval misses, the right chunk exists but doesn't surface, often because of vocabulary mismatch or bad chunk boundaries splitting an answer across chunks; (2) ingestion rot, stale or deleted documents still served, missing connectors, lost tables in PDFs; (3) generation errors, the model ignores retrieved context or blends it with parametric knowledge. The diagnostic question for any failure is "was the answer in the retrieved context?" If yes, it's a generation problem; if no, retrieval or ingestion.
Both of the big retrieval failures deserve a concrete picture, because they are what you will actually debug in week two of a deployment. Vocabulary mismatch: the HR handbook says "annual leave entitlement" and the employee asks "how much PTO do I get?"; the embedding gets you partway, but if the corpus is full of internal jargon, the exact-term half of hybrid search is what rescues the query. Boundary splitting: the eligibility rule is a heading on one page and its three exceptions are on the next, your chunker cut at the page break, and now the chunk that surfaces says employees are eligible while the exceptions live in a chunk that never ranks. The model answers confidently from half a policy. Nobody wrote a bug; the document was simply cut in the wrong place, which is why chunking is a quality decision and not a preprocessing chore.
The failure classes summarize into a table worth having in your head when someone says "the answers are wrong":
| Failure class | What the log shows | First fix to try |
|---|---|---|
| Retrieval miss | Correct chunk absent from top-k | Hybrid search, better chunk boundaries, query rewriting |
| Ingestion rot | Chunk retrieved but stale or orphaned | Incremental sync, deletion tombstones, source-of-truth audit |
| Generation error | Correct chunk present, answer ignores or contradicts it | Grounding instructions, citation requirement, smaller k with better chunks |
The reflexes that read as tutorial-deep, and the operational move that replaces each:
| What people reach for | Why it fails | What to say instead |
|---|---|---|
| "Rewrite the prompt" when answers are wrong | If the correct chunk never reached the context, no prompt can recover it | "Pull twenty failures and check the passage against top-k first" |
| "Split every 500 tokens" | Cuts an eligibility rule away from its exceptions; the model answers from half a policy | "Chunk on structure, 10 to 20% overlap" |
| "Pure vector search is enough" | Part numbers and codenames carry no meaning for a general embedder to capture | "Hybrid: BM25 plus dense, merged" |
| "Which vector database is best?" | Vendor choice almost never decides quality; chunking and retrieval do | "Say so, then talk about the golden set" |
Close with measurement: a small golden set (50–100 real questions with known source documents) lets you compute retrieval recall@k separately from answer quality, so you fix the right stage.
What interviewers probe next
"Your customer says answers are wrong, which stage do you check first?" Retrieval, and with a measurement rather than a hunch: take twenty failed questions, check whether the correct passage appeared in the retrieved top-k, and you have split the problem in one afternoon. Prompt work before that check is polishing the wrong stage.
"Why hybrid search instead of pure vectors?" Embeddings capture meaning, and an internal part number or project codename has no meaning for a general-purpose embedder to capture. "POL-88231" and the customer's jargon live in lexical space, so BM25 rescues exactly the queries vectors fumble. Production systems run both and merge; neither alone survives an enterprise corpus.
"How do you keep the index fresh?" Incremental sync driven by the source's change feed where one exists, tombstones so deletions actually disappear from the index rather than lingering as orphaned chunks, and a freshness check in the eval set: a handful of questions whose answers changed recently, so staleness shows up as a failing eval instead of an embarrassed customer.
Each probe is checking whether your pipeline knowledge is operational or decorative.
Common mistakes
Spending five minutes on embedding math and zero on failure modes. Treating chunking as an afterthought ("just split every 500 tokens"). Never mentioning evals or logging: per the rubric at AI labs, "evals first, unprompted" is the single strongest signal even on an easy question. And naming a vector database as if the choice of vendor were the interesting decision; it almost never is.
What to actually do
Log the query, the retrieved chunks and the answer from day one; those logs are the eval set. Build a golden set of 50 to 100 questions from real traffic with the source passage labeled for each. Measure recall@k separately from answer correctness. When a customer reports wrong answers, pull twenty failures and check whether the passage appeared in the retrieved top-k before anyone opens the prompt. Run hybrid retrieval and chunk on document structure by default, and treat the vendor decision as the last one you make, not the first.
Key takeaways
- Recite the spine in one breath, then immediately point to chunking and retrieval as the failure hot spots.
- The split-the-failure question ("was the answer in the context?") is the line that reads as operational, not tutorial.
- Bring up a golden set and recall@k unprompted; vendor choice is the least interesting decision here.