RAGPerf: An End-to-End Benchmarking Framework for Retrieval-Augmented Generation Systems

Most RAG benchmarks ask whether answers are right. Few ask where the time goes, how much memory each stage needs, or what happens to latency when documents are being updated while people query them. Those are the questions that decide whether a pilot survives production.

RAGPerf, from the University of Illinois and IBM Research, is a benchmarking framework that answers them. It splits a RAG system into stages (embedding, indexing, retrieval, reranking, generation), lets you swap components at each stage, generates workloads with configurable mixes of queries and document inserts, updates and deletes, and records performance (latency, throughput, CPU, GPU, memory, disk) and quality (context recall, accuracy, factual consistency) at once. It supports text, PDF and audio pipelines and five vector databases: LanceDB, Milvus, Qdrant, Chroma and Elasticsearch.

The measurements are the useful part. For text queries, generation took 75% to 91% of latency, so the choice of vector database barely changed response time. For PDFs, OCR and format conversion took 98.2% of indexing time while the GPU sat around 10% busy. For page-image retrieval in the style of ColPali, reranking took 28% to 87% of each iteration. Host memory, not CPU, decided throughput, and keeping new documents searchable between index rebuilds cost latency that grew until the next rebuild.

The most common performance mistake on RAG projects is optimizing the stage that is easiest to change rather than the one that is slow. This paper gives you measured shares to anchor that conversation.

Generation is usually the bottleneck for queries. With generation at 75% to 91% of latency, even a tenfold faster retriever improves end-to-end response time by at most about a third. Shorter prompts, smaller or faster models, better batching and caching move the number; swapping vector databases mostly does not.

Ingestion has its own bottleneck. For scanned or complex PDFs, OCR dominated indexing so completely that everything else was noise. Budget the ingestion pipeline as a separate system with its own throughput target, and consider image-based retrieval or faster conversion when the backlog is large.

Freshness is a design decision with a measured cost. Many vector indexes do not make new documents searchable until they are rebuilt. A side buffer (a flat index scanned on every query) makes updates visible at once but slows queries as it grows. Customers who say "the bot must know about documents within minutes" are asking for that trade-off, and you should show them its price.

Memory sizes the deployment. Host memory for indexes and GPU memory for models set the ceiling. CPU cores barely mattered.