LEANN for Local RAG: Storage, Recall and Tradeoffs

LEANN is a storage-efficient approximate nearest-neighbor index. Its central trade is to retain a compact search structure and recompute selected document embeddings when a query needs them. That makes it worth evaluating when storing every dense vector is the constraint and the machine can afford additional inference during retrieval.

A personal knowledge base queried a few times an hour has different constraints from a shared service. Start with the customer’s disk ceiling, query rate and latency requirement. Use the vector database fundamentals to separate the index from the surrounding retrieval service.

Sources: LEANN project and implementation.

How the index and query paths fit together

The MLSys 2026 paper describes a build path that computes embeddings, constructs a proximity graph, preserves useful connections around high-degree nodes, and retains compressed vector information. Query processing uses cheap approximate distances to select promising candidates, then regenerates exact embeddings from source content. Exact and approximate scoring interact during traversal; this is more than a final reranking pass. Batching improves the utilization of the embedding hardware.

Follow the source-content box below: recomputation still needs the original chunks and a reproducible embedding model. Keep both with the index.

The figure explains the paper’s two-level design. The repository offers HNSW and DiskANN backends with different paths and settings. For example, its configuration guide describes DiskANN with recompute enabled as PQ traversal followed by final reranking, unlike the paper’s interleaved traversal. Check the selected backend before assuming that every paper optimization is active in your installed package.

Sources: MLSys 2026 paper, sections 3–6, Backend and recomputation configuration.

Read the 97% claim with its denominator

The project README reports a 60-million-chunk Wikipedia comparison: 201 GB for its conventional-vector baseline and 6 GB for LEANN. The arithmetic is (201 − 6) / 201 = 97.0% less index storage, or a 33.5-fold ratio. These are the project’s reported results, not a benchmark run by FDE Interviews.

That is not a 6 GB total application footprint. Source documents, parsed chunks, model weights and caches remain separate items. Measure peak build disk and memory too; creating the index can require more headroom than serving it.

Compare quality at the same k, embedding configuration and dataset. Nearest-neighbor recall, passage relevance and generated-answer quality are different measurements. “Without accuracy loss” on one benchmark is not a guarantee for a customer corpus.

Sources: Project-reported storage comparisons, Paper evaluation and experimental conditions.

Work a complete storage budget

Use the calculator to distinguish index savings from application savings. Its starting example is deliberately synthetic: one million chunks, 768-dimensional float32 vectors and 32 four-byte neighbor IDs per chunk. That gives 3.072 GB of dense vectors plus 0.128 GB of adjacency data before additional implementation overhead.

The example also reserves 1,000 bytes of source text per chunk and 0.5 GB for model files. Enter a measured candidate-index size instead of assuming the README’s percentage applies. At the initial 100 MB candidate size, the simplified index subtotal falls from 3.2 GB to 0.1 GB; the application subtotal falls from 4.7 GB to 1.6 GB. Those are 96.9% and 66.0% savings respectively. The different percentages are both correct because their denominators differ.

The calculator excludes metadata, filesystem overhead, runtime memory, build scratch space and caches. It predicts neither LEANN’s size nor its latency. Practice the same arithmetic in vector-index capacity planning.

A first evaluation that can change your decision

Start with a bounded, approved corpus and a frozen set of queries. For a hypothetical support-manual pilot, include ordinary paraphrases, exact part numbers, ambiguous product names and questions with no answer in the corpus. Label which source passages support each answer before comparing retrievers. Keep a holdout set for the final decision.

Build a dense baseline and the candidate index using the same chunk boundaries, embedding model and similarity convention. Record exact dependency versions and backend settings. An ANN recall test can compare returned IDs with exhaustive nearest neighbors on a manageable subset; a relevance test compares results with human-labeled useful passages. Report these as separate measurements.

Measure retrieval latency independently of generation. Run a cold process, a warmed process and concurrent queries. Record p50 and p95 alongside peak memory, disk footprint, build duration and query-time embedding work. Publish the machine specification and whether the corpus and model were already cached. A one-user warm-cache demonstration does not establish the p95 of the shared customer service.

Choose acceptance criteria before tuning: passage relevance, disk ceiling and p95 retrieval deadline. Set thresholds from the application. If latency fails, investigate query-time embedding work and contention before changing the generation model.

Local storage is only one privacy boundary

LEANN supports local and remote model providers. Selecting a cloud embedding provider can send source text out during indexing and query-time recomputation. Selecting a cloud generation provider can send the question and retrieved passages out even when the index itself stays on disk. Audit both paths.

For an offline deployment, provision models first, then test with outbound access disabled. Use a synthetic corpus while inspecting traffic. Installation downloads are a separate setup phase.

Access control belongs around the retrieval workflow. A local index is not proof that every user is entitled to every document. For an initial multi-user pilot, separate corpora by authorization boundary or verify that filtering prevents unauthorized source text from reaching embedding calls, returned snippets, logs and generation. Retrieved text remains untrusted input even when it came from a trusted directory.

Sources: Generation providers and local/remote configuration, Embedding-provider settings.

Where the trade can fail

Recomputation moves work into the query path. A high query rate, an expensive encoder or a saturated GPU can turn a disk win into a latency problem. Repeated popular queries may benefit from caching, but count the cache in the storage budget and test eviction behavior rather than reporting only its best case.

Preserve the model revision, tokenizer, normalization and document/query templates. A successful file load does not establish compatibility with a changed encoder. See embedding versions and drift before changing any of these inputs.

Treat updates and deletions as release tests. Add a document, change a passage, remove a document and revoke its permission. Verify the resulting search results and any saved snippets. Package support for incremental operations depends on the backend and version; the paper’s update design is not a promise that every interface exposes the same behavior.

For exact identifiers, retain an exact or lexical retrieval path and evaluate the combined system. Semantic proximity alone is a weak contract for a serial number. If the customer needs replication, transactional metadata changes or a particular service-level objective, evaluate those capabilities explicitly instead of assuming an ANN library supplies the surrounding database service.

Sources: Backend and tuning notes, Project troubleshooting notes.

How to explain the decision in an interview

A defensible answer starts with the constraint: “I would evaluate LEANN for a storage-constrained corpus with modest query traffic. I would keep the source text and embedding configuration reproducible, then compare retrieval relevance, cold and warm p95, and peak build resources against a dense baseline.” This is a proposed approach, not a reported company interview answer.

Practice this decision with the RAG and agent questions. For the 97% claim, identify the index-storage denominator. For privacy, trace both embedding and generation requests. For a rollout, record the approved corpus, model/backend revisions, acceptance thresholds and rollback trigger. Keep the existing retrieval path until the candidate passes the customer workload.