Deep Research Agents Need a Reviewer: What a Jev Review Layer Gets Right, and the Gaps We Found Testing One
A deep research agent with a separate review layer is the right design: the model that writes the report should not decide it is done. We read and tested an open-source Nemotron, Tavily and Jev build. It fails open, retries without feedback and judges grounding against snippets. Here is the fix.
BY MEI LIN · FDEINTERVIEWS EDITORIAL · UPDATED OCTOBER 21, 2026 · 9 MIN READ
PRACTICE THIS:What is Jev, TypeSafe's System One model ·System One decision models, the concept ·RAG and agent design questions ·The must-know FDE questions
A deep research agent should not grade its own homework. Putting a separate review layer between the draft and the user is the right design, and an open-source build that uses Nemotron to research, Tavily to search and Jev to review shows the pattern cleanly. We read every file and tested the review path. The idea holds up. The implementation has three gaps that most research agents share. It fails open when the reviewer breaks. It retries without telling the agent what was wrong. And it checks grounding against search snippets instead of sources. Each gap is fixable in under a hundred lines, and fixing them is what turns a reviewer from decoration into a gate. Below: how the build works, what we found, the design we would ship to a customer, and when this pattern is worth its cost.
The build, and why the design is right
The project is deep_research_agent_with_review in Sumanth P's Hands-On-AI-Engineering repository on GitHub, built during the Nebius AI Builder Program; the README discloses the program's credits. It is a Streamlit app with about 700 lines of Python, tests included. A LangChain Deep Agent running nvidia/Nemotron-3-Ultra-550b-a55b on Nebius Token Factory plans the work and calls a search_web tool backed by Tavily (five results per query, deduplicated by URL, numbered as sources). It writes a Markdown report with numbered citations. A small function checks those citations. Then the report, the question, the sources and any citation issues go to Jev, TypeSafe's decision model, with three structured questions:
evidence_coverage(yes or no): is the evidence enough to answer the question without major gaps?claim_grounding(yes or no): are the report's material claims supported by its numbered sources?action(a choice ofapproveorresearch_again).
A probability of 0.70 or higher counts as a pass. If Jev asks for more research, the agent runs once more and Jev reviews the new report. LangSmith traces each step.
The README states the principle in one line worth keeping: Jev "does not write the report". That separation is the point. The model that produced the work is the worst-placed judge of whether it is finished, because it shares the blind spots that produced the gaps. This is the maker-checker rule from banking and from our agentic software factory guide, and it applies to research agents for the same reason it applies to code. Using a fast decision model as the checker is also a good choice: as we covered in what Jev is, it returns calibrated probabilities for structured questions in about a tenth of a second, and at the input price we recorded in September 2026 ($0.042 per million tokens, output free), a review of a 5,000-token state costs about two hundredths of a cent.
How it actually flows
Source: the project’s workflow and review implementation. This is the observed flow, including its release behavior after the second review.
Read the edge labels. They are where the three gaps live.
What we found when we tested the review path
The project's three unit tests pass. They cover response parsing and citation validation. We then called the same functions with the inputs a production system eventually sees. Running the full app needs Nebius, Tavily and Jev keys, so we tested the decision logic directly; nothing below depends on model behavior.
| Situation | What the code does | Why it matters |
|---|---|---|
Jev returns research_again | Triggers one more research run | Correct |
| Grounding probability 0.69 | Triggers a retry | Correct |
| Grounding probability exactly 0.70 | Passes | A hard line at 0.70 with no logging of near misses |
| Jev times out or no key is set | Report shown with a warning that it was not reviewed | Fails open, but at least says so |
| Jev's response has an unexpected shape | Report shown; the review panel reads Decision "Unavailable" next to the explanation "JEV approved the evidence review." | Fails open and contradicts itself |
Second review still says research_again | Report shown anyway | The cap is right; the missing escalation is not |
| Citation check finds problems | Problems sent to Jev as context; never blocks on their own | A deterministic failure becomes a suggestion |
The fifth row is the one to fix first. When the review response parses to nothing (a renamed field, a gateway error page returned with a 200, a schema change), the parser marks the review "completed", finds no rejection, and writes the default explanation, which is approval. A reviewer that fails open is a progress bar. One that reports approval when it read nothing is worse than no reviewer, because it removes the user's own caution.
We also probed the citation checker with common formats:
| Report text | Result |
|---|---|
Latency fell 40% [1]. Costs rose [2]. | Passes |
Latency fell 40% [1, 2]. Costs rose [3, 9]. (source 9 does not exist) | "No numbered citations" |
Three studies agree [1-3]. | "No numbered citations" |
Overview [1]. Revenue doubled in 2026. The CEO resigned. | Passes |
Revenue doubled [7]. (only five sources) | Flags source 7 |
The regular expression only matches a single number in brackets, so grouped citations look like none at all, and the invented source 9 in the second row is never flagged. The fourth row is the bigger issue: one real citation anywhere lets every uncited claim through. That is a reasonable first version of a check, and it shows why citation checks should run per sentence, not per report.
The three gaps, and what each one costs
1. Failing open. Covered above. In a demo this is harmless. In a customer deployment, the day the reviewer's API key rotates is the day every report ships unreviewed, and nobody notices because the UI still says "approved". Decide the failure policy explicitly: block, or show with a prominent unreviewed label, and alert someone either way.
2. Retrying without the reviewer's reasons. When Jev asks for more research, the second run starts from scratch: a new search collector, a new agent, the original question and one generic sentence saying evidence was weak. It does not receive the first report, the first sources or which check failed. The code asks Jev only for two booleans and a choice, so there are no reasons to pass along in any case. Research again without the reviewer's reasons is a re-roll, not a revision. You get a different report with different problems, and the second review grades a new draw from the same distribution.
3. Grounding against snippets. The search tool requests results with include_raw_content off, so each source is a title, a URL and a short snippet, and the review payload trims snippets to 1,200 characters. When Jev answers "are the claims supported by the numbered sources?", the sources it sees are teasers. A claim can match the snippet and contradict the page. Grounded in a snippet is not grounded in a source. The fetch layer is part of the evidence: if the agent never read the page, nobody in the pipeline did.
None of these are exotic. We see all three in research agents built by well-funded teams, which is why this small project is a good teaching case.
The design we would ship
This is our proposed design. Repair returns to evidence collection and drafting before repeating citation, grounding and coverage checks.
The changes, in order of value:
- Fail closed by policy. Any reviewer error, missing key, timeout or unparseable response produces an explicit unreviewed state. Never default to approval.
- Make deterministic checks gates. Parse
[1, 2],[1-3]and footnote styles. A sentence with a factual claim and no citation is removed or marked before any model sees the report. - Ground per claim, against passages. Split the report into claims and ask the decision model one yes-or-no question per claim against its cited passage. At about a tenth of a second and fractions of a cent per call, Jev is cheap enough to run twenty times per report. Per-claim results also tell the retry exactly what to fix.
- Retry with reasons. Pass the failed claims, the unanswered sub-questions and the sources already gathered. Ask for a targeted repair, not a new report.
- Log every probability. Keep the coverage and grounding scores for approved reports too. A gate whose scores always sit at 0.95 is not gating anything, and you only see that in the distribution.
- Escalate, don't loop. One targeted retry, then show the report with its open gaps listed and route it to a person if the stakes warrant it.
That design separates three different jobs that the original merges into one call: code checks what code can check, a fast decision model answers narrow factual questions, and a person handles what is left. The same split works for an LLM judge in place of Jev; a decision model is simply cheaper and faster for the narrow questions, and an LLM judge is better when you need a written critique.
Pros, cons and when to use the pattern
| Strengths | Weaknesses as built |
|---|---|
| Writer and reviewer are separate models with separate jobs | Fails open; a malformed review reads as approval |
| Structured review questions with probabilities and a stated threshold | Retry runs blind and re-researches from scratch |
| Bounded: at most one automatic revision, so cost and latency are predictable | Grounding judged against search snippets |
| Each part (model, search, reviewer, tracing) is swappable | Citation check is advisory and misses grouped citations |
| Readable code, tests for parsing, traced with LangSmith | No license file in the repository, so reuse terms are unclear |
Use a reviewed research agent when a wrong answer is costly but a slow one is fine: competitive and market briefs, vendor due diligence, literature reviews, account research before a customer meeting, policy and regulatory summaries. For a forward deployed engineer, the pre-engagement brief on a customer's industry is a good first use: low risk, high time saved, and easy to spot-check.
Do not use it when the answer must be current to the minute, when sources sit behind logins the agent cannot reach, or when the output goes straight to a customer with no human reading it. A review layer lowers the error rate. It does not make an unread report safe to send.
On cost, the review is the cheap part. A research run with a 550-billion-parameter model and a dozen searches costs far more than twenty Jev calls, so per-claim grounding adds very little to the bill. The expensive mistake is the opposite: saving a fraction of a cent on review and shipping a confident report built on a snippet.
A note on reusing this code
The repository has no license file at the time of writing (21 October 2026). Public code without a license is not open source in the legal sense; by default its author keeps all rights. Read it, learn from it, and ask the author before copying it into a client's codebase. The design ideas carry no such restriction.
If you are preparing for applied AI or FDE interviews, "design a research agent that doesn't hallucinate citations" is now a standard prompt. The answer interviewers want separates the writer from the checker, names the failure policy, grounds per claim against fetched text, and caps retries with an escalation path. Practice it with our RAG and agent questions, and read up on hallucination and RAG evaluation for the measurement side.
Turn it into offers. Work the real questions and concepts this maps to:
FAQ
A review layer is a separate component that decides whether a research report is good enough to show, instead of letting the model that wrote it decide. It typically checks whether the evidence covers the question and whether each claim is supported by a cited source, then approves the report or sends the agent back to research more. Keeping the writer and the reviewer separate is the maker-checker principle applied to agents.
Jev is a decision model: it answers structured questions (yes or no, a choice, a score) with probabilities in roughly a tenth of a second and costs a fraction of a cent per review. That suits a gate that runs on every report. An LLM judge can explain its reasoning and handle open-ended critique better, so a good design uses the fast decision model as the gate and a stronger model only when it says the report needs work.
Failing open. If the reviewer times out, is missing a key or returns something unexpected, many implementations show the report anyway, sometimes labeled as approved. A reviewer that does not block on failure is a progress bar. Decide in advance what happens when the reviewer is unavailable, and make that path visible to the user.
Give the retry the reviewer's reasons, the failed claims and the sources already gathered, and ask it to fix those specifically. Re-running the whole research task with a generic instruction to do better is a re-roll, not a revision, and may produce a different report with different problems. Cap retries and escalate to a human when the cap is hit.
No. Checking that cited source numbers exist only proves the citations are well formed. You also need claim-level grounding: split the report into claims and check each one against the cited passage, ideally the full page rather than a search snippet. Uncited claims should be removed or flagged, not tolerated because one citation exists elsewhere.
Discussion (4)
The snippet point is the one teams miss in production. Search APIs return a few hundred characters per result, and a reviewer reading those can approve a claim the full page contradicts two paragraphs later. If grounding matters, fetch and store the page and let the reviewer read the passage, not the teaser.
Reader question: one retry or several?
One targeted retry, then a human. In our experience the second automatic retry mostly rewrites prose and rarely finds new evidence. If the first retry did not close the gap, the question probably needs a source the search tool cannot reach, and a person should know that.
Worth adding: log the reviewer's probabilities for every report, approved or not. After a month you can plot approval rate against the 0.7 threshold and see whether the threshold is doing anything, or whether every report sits at 0.95 and the gate never closes.