TL;DR: Treat chunking as a retrieval-recall problem: start with a measurable fixed-size baseline, then move to structure-aware plus parent-child retrieval (embed small chunks, return the parent section), and prove every change with recall@k on a labeled golden set.
How to approach it
Frame chunking as a retrieval-recall problem, not a text-splitting problem: a chunk is the unit your system can find and the unit the model gets to read. Ask what the documents are, since a 200-page contract, a Slack thread, and a wiki page want different treatment, and what questions users ask, because chunk size should roughly match answer size.
A strong answer
Start with the baseline: fixed-size chunks of 300–800 tokens with 10–20% overlap, split on sentence boundaries. It's dumb but measurable, and you should always be able to say how much better anything else is.
Then the upgrades. Structure-aware chunking splits on document structure (headings, sections, list items) so a chunk is a semantically complete unit; for HTML/Markdown this is nearly free, for PDFs you need layout parsing (unstructured.io, Docling) and table extraction, since a table sliced mid-row is unrecoverable. Semantic chunking breaks where embedding similarity between consecutive sentences drops, useful for unstructured prose. Parent-child (small-to-big) retrieval is often the biggest practical win: embed small chunks (100–300 tokens) for precise matching, but return the parent section (1,000–2,000 tokens) to the model, so retrieval precision and generation context stop fighting each other. Finally, contextual enrichment: prepend each chunk with document title, section path, and date ("ACME MSA 2024 > Termination > 11.2"), or LLM-generated chunk summaries, which fixes the classic failure where a chunk says "the fee shall be 2%" and nothing identifies which fee or which contract.
Be ready to defend the number itself, because "why 500 and not 200 or 1500?" is the follow-up that separates reasoning from recitation. Three forces set it. Query granularity: a chunk should be about the size of the answer to a typical question, and most factual questions are answered in a paragraph, not a page. Embedding dilution: an embedding is one vector summarizing the whole chunk, so a 1,500-token chunk covering four topics matches all four weakly instead of one strongly. Prompt budget: k chunks times chunk size has to leave room for instructions and the answer. Small chunks win on match precision and lose context; big chunks win on context and lose precision; 300 to 800 with structure-aware boundaries is where those curves cross for most corpora, and parent-child exists precisely so you stop having to choose.
To make the boundary problem concrete, here is the failure in miniature. A contract reads: "11.2 Either party may terminate with 30 days notice. Notwithstanding the foregoing, during the first contract year termination requires 90 days notice and executive approval." A naive splitter cuts after the first sentence. Now the chunk that ranks for "how do we terminate?" says 30 days, the exception lives in a chunk that starts mid-thought with "Notwithstanding the foregoing" (which embeds as legal fog and ranks for nothing), and the system confidently gives customers the wrong notice period. Structure-aware chunking keeps clause 11.2 whole; contextual enrichment stamps it "ACME MSA 2024 > Termination > 11.2" so it is citable when it surfaces. That one example is worth reciting in interviews because it shows the failure, the mechanism, and both fixes in four sentences.
How do you know it's good? Build a golden set of 50–100 real questions labeled with the document (and ideally the passage) containing the answer. Measure recall@k and MRR for each chunking strategy with everything else held constant. Chunking changes are cheap to A/B offline (re-chunk, re-embed, rerun the eval), and a strong candidate quotes a realistic effect while saying where the number comes from. On heterogeneous corpora, moving from naive fixed-size to structure-aware plus parent-child is worth planning around a 10 to 20 point lift in recall@10. Treat that as the band worth designing an experiment for, not as a result you can promise: it depends on how structured the corpus already was, and a corpus of clean short prose has far less to gain than a pile of PDFs with tables. The number you report should be the one your golden set produced.
The four strategies side by side:
| Strategy | Recall lift | Overhead | Best for |
|---|---|---|---|
| Fixed-size | Baseline | Low | A measurable starting point |
| Structure-aware | Higher | Near-free on HTML/Markdown; layout parsing on PDFs | Headings, sections, list items; tables |
| Semantic | Higher | Embedding similarity pass between sentences | Unstructured prose |
| Parent-child | Often the biggest practical win | Embed small chunks, return parent section | Reconciling retrieval precision with generation context |
What interviewers probe next
"What about tables in PDFs?" Extract them separately, serialize as Markdown or row-wise sentences, keep the table caption attached. "Slack threads?" The thread, not the message, is the semantic unit; chunk by conversation with speaker labels. "Does overlap actually help?" Modestly; it insures against boundary splits, and parent-child solves the same problem more cleanly. "Embedding model context limit?" Chunks must fit it; long-context embedding models relax but don't remove the precision argument for small chunks.
Common mistakes
Giving one number ("512 tokens") with no rationale and no evaluation plan. Ignoring document structure entirely: slicing contracts mid-clause is the #1 self-inflicted recall killer. Forgetting metadata, so retrieved chunks are correct but uncitable. And optimizing chunking by vibes: if you can't say "recall@10 went from 62% to 78% on our golden set," you're guessing, and interviewers at eval-obsessed companies will say so.
Key takeaways
- A chunk is both the retrieval unit and the reading unit; size it to the answer, not to a round number.
- Parent-child retrieval is the highest-leverage upgrade: match on small chunks, generate on parents.
- Every chunking choice needs a recall@k delta on a golden set, or it's a guess.