Context is Gold to find the Gold Passage: Evaluating and Training Contextual Document Embeddings

A passage can be meaningful to a reader while looking incomplete to an embedding model. A pronoun may refer to an entity named earlier. A short policy clause may depend on a heading. If the system embeds each passage independently, it discards those relationships before retrieval begins.

This paper introduces ConTEB, a benchmark designed to expose that problem, and InSeNT, a post-training method for contextual embeddings. InSeNT combines late chunking with two contrastive objectives. One helps identify the correct document; the other helps distinguish the correct passage from nearby passages in that document.

Late chunking first encodes a document together, then pools the token representations belonging to each chunk. Context can therefore influence a chunk vector without being copied into its displayed text. InSeNT adds training pressure to preserve both that contextual information and the chunk's specific meaning.

The June 2025 revision reports substantial contextual-retrieval gains. It also contains regressions on some individual tasks and on a non-contextual benchmark. The official project provides the benchmark, models and training artifacts. Its strongest practical lesson is to evaluate contextual and self-contained queries separately rather than treating one embedding score as complete evidence.

Many enterprise corpora repeat a template across customers, years or products. A clause saying “coverage lasts twelve months” may be useful only when paired with the correct contract header. Another clause in that same contract may discuss a completely different obligation.

These are two distinct errors: confusing similar documents and confusing passages within the right document. Better context propagation may help the first while making the second harder if all chunk vectors become too similar. The paper gives a training objective and benchmark design for examining both.

Use chunking strategies to define meaningful boundaries and RAG evaluation to label the actual answer-bearing passages. Contextual embeddings become worth testing when a reviewed failure set shows that missing document context, rather than absent source data or incorrect permissions, is the retrieval bottleneck.