Nemotron ColEmbed V2: Top-Performing Late Interaction Embedding Models for Visual Document Retrieval

Most enterprise retrieval runs on PDFs and slide decks. The usual pipeline extracts text with a parser or OCR, chunks it and embeds it, and loses tables, charts and layout along the way. Visual document retrieval skips extraction: a vision-language model embeds each page image directly, and text queries are matched against it.

Nemotron ColEmbed V2 is a family of three open-weights models for this: a 3B built on NVIDIA's Eagle 2 with a Llama 3.2 3B backbone, and 4B and 8B built on Qwen3-VL. All use late interaction, the ColBERT scheme: a page is stored as one vector per visual token (773 on average for the Qwen3-VL models on ViDoRe V3 pages, 2,304 for Eagle 2), each query token finds its best-matching page vector, and the page score is the sum of those maxima.

Training changes the decoder's causal attention to bidirectional and uses 500,000 samples from public datasets. Hard negatives are mined with an internal embedding model and kept only if their similarity is below 95% of the positive's, to avoid false negatives. Training pages are clustered (14 clusters) and sampled evenly, queries are translated into other languages with Qwen3-235B, and 4 to 8 separately trained models are averaged into one.

The 8B model ranked first on the ViDoRe V3 leaderboard with 63.42 nDCG@10. The paper then turns to cost: the 8B model's vectors for a million pages take 5,897.5 GB in fp16, and projecting them to 128 dimensions keeps 95.4% of the accuracy at 3% of the storage.

Customers with slide-heavy or table-heavy corpora (financial filings, engineering drawings, pharma reports, HR policy decks) lose answers in text extraction. A chart that shows the trend, a table cell with the number, a diagram label: parsers drop or scramble them. Retrieving the page image sidesteps the parser, which also removes a brittle stage from the indexing pipeline.

This paper is useful less for the leaderboard rank and more for its deployment section, which most retrieval papers skip. It puts a number on what late interaction costs. One million pages with the 8B model is 5,897.5 GB of fp16 vectors; with the 3B model it is 13,183.6 GB, because Eagle 2 produces three times as many tokens per page. A single-vector model needs 3.8 GB. Search also needs a vector store that supports the MaxSim operation over multi-vector documents.

It then gives two ways out. A model trained to emit 128-dimensional vectors keeps 95.4% of the accuracy at 3% of the storage. Or skip late interaction: a single-vector 1B model plus a 1B reranker on the top 50 reached 54.41, close to a late-interaction 1B model's 55.48 with a tiny index. For a customer with millions of pages, those two options are the conversation, not the top score.