ColPali: Efficient Document Retrieval with Vision Language Models

Most enterprise knowledge lives in PDFs: reports with charts, slide decks, scanned forms, tables of figures. A standard retrieval system cannot read any of it directly. First a parser extracts text, a layout model finds paragraphs, titles and tables, OCR reads scanned or image regions, an optional captioning model describes figures in words, and only then are chunks embedded. The authors found that improving this ingestion pipeline mattered more than improving the embedding model, and that the pipeline was slow.

ColPali skips it. It takes a picture of each page and runs it through PaliGemma-3B, a vision-language model. Instead of averaging the result into one vector, it keeps one small vector (128 numbers) for each of the 1,024 image patches plus a few text tokens: 1,030 vectors per page. A query is encoded the same way into one vector per token. The score is computed by late interaction, the ColBERT method: each query token finds its best-matching patch on the page, and those best matches are summed. A query about "Q3 revenue by region" can match the small table that contains it, even when the rest of the page is about something else.

To measure this, the authors built ViDoRe, a benchmark of page retrieval over figures, infographics, tables and text-heavy documents in several domains and two languages. ColPali averaged 81.3 nDCG@5 against 67.0 for the best text pipeline with captioning, and indexed pages in 0.39 seconds each against 7.22 for parsing. The cost is storage: about 257 KB per page.

On most document-heavy projects, ingestion is where the time goes: parsers that mangle tables, OCR on scanned pages, slides where meaning sits in a chart. This paper offers a real alternative and puts numbers on its trade-offs.

It removes the most fragile part of the pipeline. No layout detection, no OCR, no captioning step to maintain. A page goes in as an image. For corpora full of charts, infographics and tables, the gains were largest exactly where text pipelines struggle.

It changes what you store. A single-vector index stores one vector per chunk. ColPali stores 1,030 per page, about 257 KB at 16-bit precision, roughly 250 GB per million pages before compression. The paper shows pooling similar patches can cut this by two-thirds while keeping about 98% of retrieval quality. Price the storage and the vector database's support for multi-vector search before proposing it.

Retrieval is half the system. ColPali returns pages. A RAG system still has to answer from them, which means either a vision-capable model reading the page images or a text extraction step for the retrieved pages only. The authors list end-to-end visual RAG as future work.