RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval

Retrieval-augmented generation usually splits documents into short chunks and retrieves the few most similar to the question. That works for "what year did X happen". It fails for "how did Cinderella reach her happy ending", where the answer is spread across the whole story and no chunk contains it. Parth Sarthi, Christopher Manning and colleagues at Stanford built RAPTOR (Recursive Abstractive Processing for Tree-Organized Retrieval) to retrieve at several levels of abstraction.

Building the tree. The document is cut into chunks of about 100 tokens (never splitting a sentence) and embedded with SBERT. The chunk embeddings are reduced with UMAP and clustered with a Gaussian mixture model; the number of clusters is chosen by the Bayesian information criterion, and a chunk may belong to more than one cluster. An LLM (gpt-3.5-turbo) summarizes each cluster. The summaries are embedded and clustered again, and the process repeats until clustering is no longer possible. The result is a tree: original chunks at the leaves, summaries of summaries toward the root.

Querying it. The paper tested two methods. Tree traversal starts at the root and descends, keeping the top-k nodes at each level. The collapsed tree puts every node from every layer into one pool, ranks them all by cosine similarity to the question, and adds nodes until a token budget (2,000 tokens, about 20 nodes) is filled. The collapsed tree did better and is used throughout. A broad question tends to match summaries; a detail question tends to match leaves.

On NarrativeQA (books and movie scripts), QASPER (NLP papers) and QuALITY (multiple choice over passages of about 5,000 tokens), RAPTOR beat retrieval without the tree with each of three retrievers, and with GPT-4 set new best results on QASPER and QuALITY.

Customers ask summary questions of long documents: what are the main risks in this contract, what changed across these meeting notes, what is this report's conclusion. Flat chunk retrieval answers these badly, and RAPTOR's pattern is now common in RAG frameworks.

Match the question's altitude. Indexing summaries alongside chunks lets one search return a section-level summary for a broad question and a precise passage for a narrow one, without routing logic.

The gain depends on the question mix. With the same retriever and reader, the tree added 0.5 to 1.7 points on the paper's datasets. If the customer's questions are mostly factual lookups, expect little; if they are mostly thematic, test it.

Building costs LLM calls, once per document. About one summarization call per seven nodes, and summarizer input roughly 1.2 times the document's length. Updates to a document mean rebuilding its tree.

Summaries can hallucinate. The paper found minor hallucinations in 4% of sampled summaries. In regulated domains, keep citations pointing at leaves, not summaries.