Late Chunking: Contextual Chunk Embeddings Using Long-Context Embedding Models

Retrieval systems split documents into chunks before embedding them, because one vector per short passage retrieves better than one vector per long document. The cost is context. In a Wikipedia article about Berlin, the sentence "Its more than 3.85 million inhabitants make it the European Union's most populous city" never says "Berlin". Embedded on its own, that chunk barely relates to a query about Berlin.

The fix most teams reach for is overlap, or asking an LLM to write a context sentence for every chunk. Jina AI proposes something cheaper. Modern embedding models accept long inputs, 8,192 tokens for the models tested, roughly ten pages. So run the whole document through the model first. Every token's vector now reflects the whole text, the way "its" in context reflects Berlin. Then apply the chunk boundaries afterwards: average the token vectors inside each chunk to get that chunk's embedding. The chunking happens late, after the transformer, which gives the method its name.

It needs no training and works with any chunking rule. Across three embedding models, four retrieval datasets and three chunking methods, late chunking beat naive chunking almost every time, by about 1.5 to 1.9 nDCG points on average. In the Berlin example, the similarity of "Its more than 3.85 million inhabitants..." to "Berlin" rose from 0.71 to 0.82. The authors also extend it to documents longer than the model's window and propose a fine-tuning method that adds a little more.

Enterprise documents are full of chunks that only make sense in context: "the Supplier shall...", "this policy applies to...", "the above figures exclude...". Chunked naively, these are exactly the passages retrieval loses. Late chunking is one of the cheapest fixes available, with clear limits.

It is cheap where it applies. No LLM call per chunk, no extra storage, the same number of vectors. The alternative, asking an LLM to write a context prefix for each chunk, costs one model call per chunk at indexing time, which adds up across millions of chunks.

It needs token-level access. Late chunking pools the model's token vectors. Most hosted embedding APIs return one vector per input, not per token, so you need a self-hosted model or a provider that exposes the option. Check this before proposing it.

It is not always better. The gains were modest on average and largest for small chunks. On "needle" tasks, where a short fact is hidden inside unrelated text, naive chunking with large chunks did better, because the surrounding text added noise rather than meaning. Late chunking also pulls every chunk toward the document's overall topic, which narrows the gap between relevant and irrelevant chunks. Test it on the customer's documents rather than assuming.