IN THIS BREAKDOWN · 10 SECTIONS
You can read 17% of this breakdown.
The paper in one minute
A question like "a researcher notices night-shift workers struggle to sleep during the day; what is the primary factor?" is best served by a document on circadian rhythms, which shares no key words with it. Retrievers are trained on queries such as "Where did Yellowstone National Park get its name?", whose answer passage repeats the question. NQ and MS MARCO queries average 20 and 21 tokens; BRIGHT, a benchmark of reasoning-intensive retrieval, averages 194.
ReasonIR-8B is a bi-encoder trained for the second kind. The authors build ReasonIR-Synthesizer, which makes two kinds of synthetic data with Llama 3.1 70B. Varied-length (VL): the LLM writes a long query, 300 to 2,000 words, and a positive document for it. Hard queries (HQ): the LLM reads a reasoning-worthy seed document, brainstorms background knowledge and realistic scenarios, and writes a self-contained hard question that the document helps answer. For both, a hard negative is written in a second turn, conditioned on the query and positive, because asking for it in the same turn produced short, easy negatives.
They fine-tune Llama 3.1 8B, switched to bidirectional attention, with a contrastive loss on 1,383,877 public pairs, 244,970 VL pairs and 100,521 HQ pairs, for 1,000 steps at batch size 2,048.
On BRIGHT, ReasonIR-8B scores 24.4 nDCG@10 with the original queries against 20.9 for GritLM-7B, and 29.9 when GPT-4 first rewrites each query into a long reasoning query, slightly above Rank1-32B, an LLM reranker (29.4). Adding BM25 reaches 32.0, and a zero-shot Qwen 2.5 32B reranker on top reaches 36.9. The paper also introduces that reranker; its main trick is breaking ties.
Research notes: the problem, approach and results
BRIGHT nDCG@10 with original queries, against 20.9 for GritLM-7B and 14.8 for BM25
the zero-shot reranker's BRIGHT score before and after blending its 0-5 scores with the retriever's to break ties
share of the synthetic hard-query pairs where BM25 ranks the hard negative above the positive; 25.1% for public data
Retrievers are trained on short factual queries whose answers share words or meaning with the question. Reasoning-heavy queries, such as a long problem description whose useful document explains the underlying principle, share neither, and retrievers trained on public data do poorly on them.
Generate training data with an LLM: long varied-length queries with positives, and hard queries brainstormed from reasoning-worthy documents, each with a hard negative written in a separate turn. Fine-tune Llama 3.1 8B as a bidirectional bi-encoder on that plus 1.38M public pairs.
ReasonIR-8B scores 24.4 nDCG@10 on BRIGHT with original queries (GritLM-7B 20.9), 29.9 with GPT-4 rewritten queries, 32.0 combined with BM25 and 36.9 with a zero-shot Qwen 32B reranker. As a RAG retriever it lifts GPQA from 31.3 to 38.4 and MMLU from 71.1 to 75.6.
Why it matters to an FDE
Customer queries in support, engineering, legal and research tools are often long problem descriptions, not lookups. An engineer pastes a stack trace and three paragraphs of context; the useful document is a design note on the retry policy. Standard embedders score that pair low because the words and topics differ. If the assistant answers from the wrong documents, nothing downstream recovers.
This paper puts numbers on four levers you can pull, in order of cost.
Swap the retriever. ReasonIR-8B is released with code and data. On BRIGHT it beats the best similar-size embedder by 3.5 points with no other change.
Add BM25. A 0.5 blend with BM25 adds 2.1 points on rewritten queries; the two retrievers share only 28.2% of their top 100 documents.
Rewrite the query. Having an LLM expand the query into a long reasoning query helps on BRIGHT, but in the RAG tests a small reader rewriting its own queries helped MMLU and hurt GPQA for every dense retriever. Test before turning it on.
Rerank with an LLM, and break ties. A zero-shot 32B model scoring 0 to 5 reached 30.3; blending its score with the retriever's reached 36.9. An LLM reranker asked for a small integer score is a common setup, and its ties are easy to miss.
Understand the mechanism. Try it on a real problem.
Unlock this breakdown and 149 more papers.
Premium opens every paper in full, plus every answer, concept, course and guide on the site. One payment for six months of access.
6 months · One payment · No auto-renewal
- ◇1 architecture diagram
- </>1 runnable code experiment
- ▦2 results tables
- ?5 interview questions with answers
- 03Key takeaways
What made the data hard, which mix worked, the reranker fix, compute claims and three misreadings
Free account - 04How it works
The generation prompts, hard negatives, training setup and the tie-breaking reranker
Premium - 05The numbers that matter
BRIGHT by setting, RAG results, data ablations, recall and overlap, and the FLOPs table
Premium - 06Try it yourself
Recheck the relative gains and FLOPs, then measure what ties cost an LLM reranker
Premium - 07Where it breaks
Seed documents from the benchmark's pool, small margins, rewrite costs and reranker mismatch
Premium - 08Using it on a deployment
Checking if a customer's queries are reasoning-heavy, then each lever in cost order
Premium - 09Interview angles
Five questions this paper prepares you for, with answer hints
Premium - 10Read next
Where to go from here on this site
Premium