GO TO A SECTION

↑ ↓ to navigate · Enter to openExplore all sections →
FDEInterviews logo

NVIDIA Molt Review: An Agentic RL Framework You Can Read in an Afternoon, and What It Leaves to You

NVIDIA's open-source Molt trains agents with reinforcement learning on Ray, vLLM and NeMo AutoModel, with any Python function as the reward. We traced its code, checked its size claim, compared its choices with ResolveAI's SRE-1 training report, and wrote a reward that closes known shortcuts.

BY ARJUN MEHTA · FDEINTERVIEWS EDITORIAL · UPDATED NOVEMBER 19, 2026 · 8 MIN READ

PRACTICE THIS:How to train tool-using AI agents ·PPO and GRPO, the concept ·The SRE-1 training report, broken down ·LLM and GenAI questions

NVIDIA Molt is an open-source framework for training language-model agents with reinforcement learning, small enough to read end to end. Ray places the processes, vLLM generates the rollouts, NeMo AutoModel with FSDP2 trains one actor, and the reward is whatever Python you write next to your agent: a grader, a multi-turn tool loop or an LLM judge. The same entry point that trains a 4B model is documented to train GLM-5.2 (753B) with expert parallelism. We could not run training, which needs CUDA 13 GPUs, so we tested what we could: we traced the RL code path to check the size claim, compared Molt's design choices with ResolveAI's SRE-1 training report published the same week, and wrote a reward function that closes the shortcuts SRE-1's report says a policy learned. Our verdict: a good, honest research framework. The hard part of agentic RL, though, is the environment and the reward, and no framework writes those for you.

What Molt is

Molt sits in the family of OpenRLHF, verl and slime, and it says so. Its README compares itself with all three and credits OpenRLHF, whose package layout it keeps. What it optimizes for is legibility at scale: one trainable actor, an optional KL reference, every mechanism (advantage, loss, token capture, training loop) in one file, configured by command-line flags only.

One loop, three components, one trainable actorAgent workers: Your Env.step or ChatAgent.run. Tools, graders, judges. Returns the reward.. vLLM engines: Generate tokens and record each token id and log-prob.. tokens in. tokens out. trajectories + reward. Ray queue: Buffers trajectories; its size bounds how stale a sample can be.. batches. Trainer: one actor: NeMo AutoModel + FSDP2. Advantage, IS correction, loss. TP, EP, CP flags.. weight sync. Token-first contract: Token ids, log-probs, action ranges and rewards stay aligned from rollout to loss. Nothing is re-tokenized from text.. Our trace of the RL entry point: 53 files, 10,125 non-blank, non-comment lines (8,560 without docstrings). The README says about 9,200, measured in July. One loop, three components, one trainable actor Molt at commit ec99ae4, from its README and guide. Asynchronous mode overlaps all of it. RAY: PLACEMENT AND QUEUES Agent workers Your Env.step or ChatAgent.run. Tools, graders, judges. Returns the reward. vLLM engines Generate tokens and record each token id and log-prob. tokens in tokens out trajectories + reward Ray queue Buffers trajectories; its size bounds how stale a sample can be. batches Trainer: one actor NeMo AutoModel + FSDP2. Advantage, IS correction, loss. TP, EP, CP flags. weight sync Token-first contract Token ids, log-probs, action ranges and rewards stay aligned from rollout to loss. Nothing is re-tokenized from text. Our trace of the RL entry point: 53 files, 10,125 non-blank, non-comment lines (8,560 without docstrings). The README says about 9,200, measured in July.

Two agent interfaces cover most work:

EnvChatAgent
Who runs the model loopMoltYour code, through the OpenAI or Anthropic SDK
You implementstep(state), returning a reward and an optional next observationrun(ctx), returning a reward
Good forGrading a turn, short tool loopsAn existing agent or harness you do not want to rewrite

The ChatAgent path is the interesting one for applied teams. Your agent points its SDK client at a session URL that Molt runs on loopback. The server behind it records every token with its log-probability, so the agent code contains nothing about training. A harness you already have can, in principle, become an RL environment without changes.

Supported algorithms include REINFORCE, RLOO, GRPO, Dr. GRPO, GAE (PPO) and on-policy distillation. The November release notes add REINFORCE Zero, which the README says kept asynchronous RL stable through 20,000 policy updates.

Checking the size claim

The README's headline number is "~9.2K lines" of RL code, with a careful footnote: every Python file the RL path imports, traced from the entry point, excluding SFT, tests, examples, blank lines and comment lines, measured on 20 July 2026. We traced the import graph from molt.cli.train_rl_ray at commit ec99ae4 (17 November 2026):

What we countedLines
Files reached from the RL entry point53
Non-blank, non-comment lines in those files10,125
The same, with docstrings removed8,560
Every .py file in the molt package, physical lines15,133

The claim holds within the range set by whether you count docstrings, and the code has grown since July. One file, the vLLM worker extension, is loaded by name rather than imported, so a static trace misses it and the true RL path is slightly larger than our count. The broader point survives: the part of Molt that touches your gradients is about one order of magnitude smaller than verl's, which the same README puts at about 62,000 lines.

Read against a production run: what SRE-1 needed

ResolveAI's SRE-1 technical report, published on 18 November, is unusually candid about what it took to train a 27B agent with RL for production incident investigation. It did not use Molt; it used a modified fork of the open-source prime-rl. But its list of problems is a good checklist for any agentic RL framework, and it shows which problems are the framework's and which are yours.

What SRE-1 found it neededDoes Molt give it to you?
Train on the exact token ids the sampler produced; re-encoding changed ids in 94% of rolloutsYes. Its "token-first contract" keeps token ids, log-probs and action ranges aligned end to end
Overlap sampling and training, with a staleness boundYes. Asynchronous queues, partial rollout and per-token importance-sampling correction
A mask on extreme trainer-to-sampler probability ratiosYes, several: token or sequence level, mask, clip or truncate
Keep in-flight rollouts across a weight updateA different choice. Molt keeps the prefix but drops gradient on tokens generated under old weights
Correct numerics: fp32 gradient accumulation and master weightsMixed-precision handling is inside the training stack; check it for your model before trusting a long run
Environments whose tools see only data from before the alertNo. That is your environment
An answer key for incidents nobody wrote up, and leakage checksNo. That is your data pipeline
A reward the policy cannot gamePartly. The reward is your Python, which is the right place for it

The mask comparison is worth a closer look, because the two projects made opposite-looking choices. SRE-1 masked tokens whose ratio left a wide band of 0.2 to 5, and reports that narrower bands it tried (0.8 to 1.25 and 0.92 to 1.08) dropped more tokens and lowered its validation score. Molt's recipes use a tight band of 0.99 to 1.01, but by default applied to a sequence's geometric-mean ratio, not to each token, alongside work to make trainer and inference kernels agree. These are not contradictory. The right band depends on what you apply it to and on how far your trainer and sampler disagree, so measure the mismatch first; Molt logs it as vllm_kl and mismatch/k3_kl.

The part you write: a reward that resists shortcuts

SRE-1's report lists the shortcuts its early policy learned against an LLM judge: restating the alert as the first causal link, listing ruled-out causes nobody had proposed, reversing causal order, and citing a README it never opened. Some of those need the judge. Others are cheap deterministic checks that belong in the reward, before the judge's score is used. Here is a reward of that shape, as plain Python you can unit-test on a laptop:

python
def rca_reward(answer, alert_text, evidence, judge_score):
    """answer = {"chain": [{"claim": str, "cites": [str]}], "candidates": int}
    evidence = ids of the files the agent actually opened this episode."""
    chain = answer.get("chain", [])
    if not chain:
        return 0.0, "no chain"
    first = chain[0]["claim"].lower()
    if first and first in alert_text.lower():                 # restated the alert
        return 0.0, "first link restates the alert"
    cited = [c for link in chain for c in link.get("cites", [])]
    if not cited or any(c not in evidence for c in cited):    # cites a file it never opened
        return 0.0, "citation not in gathered evidence"
    if answer.get("candidates", 1) > 1:                        # hedged across several causes
        return min(judge_score, 0.5), "several candidate causes: capped"
    return judge_score, "ok"

We ran it on five answers to one illustrative alert ("p99 latency on checkout-api above 2s for 10 minutes"), with the judge score supplied by hand:

AnswerJudgeRewardWhy
Three linked causes, each citing a file it opened0.860.86ok
First link restates the alert0.900.00first link restates the alert
Cites a README it never opened0.900.00citation not in gathered evidence
Same good chain, hedged across three candidate causes0.860.50capped
Empty chain0.700.00no chain

Two rows got a high judge score and a zero reward. That gap is the reason deterministic checks sit in front of a judge. The checks are also easy to evade in other ways: a paraphrased alert passes the substring test. SRE-1's answer to that was to fix exploits in the judge as well, and to draw each group's judge at random from 13 models across providers, because judge errors were correlated within a provider. In Molt this function would sit inside an Env.step or at the end of a ChatAgent.run. Molt calls it; deciding what it rewards is your job.

Cheap checks first, then the judgeRollout ends with an answer: A causal chain with a cited file for each link.. Deterministic checks: Empty chain? First link restates the alert? A citation the agent never opened? Several candidate causes?. passes. fails. LLM judge score: One judge per group, drawn at random from a cross-provider panel.. Reward 0: or capped at 0.5 when hedged. Grounded chain: judge 0.86, reward 0.86.. Restates the alert: judge 0.9, reward 0.. Cites unopened README: judge 0.9, reward 0.. Hedges across three causes: judge 0.86, reward 0.5.. Empty chain: judge 0.7, reward 0. Cheap checks first, then the judge The reward from this post, applied to five answers to one illustrative alert. Judge scores were supplied by hand. Rollout ends with an answer A causal chain with a cited file for each link. Deterministic checks Empty chain? First link restates the alert? A citation the agent never opened? Several candidate causes? passes fails LLM judge score One judge per group, drawn at random from a cross-provider panel. Reward 0 or capped at 0.5 when hedged JUDGE SCORE VS. REWARD Grounded chain 0.86 → 0.86 Restates the alert 0.90 → 0.00 Cites unopened README 0.90 → 0.00 Hedges across three causes 0.86 → 0.50 Empty chain 0.70 → 0.00 gray: judge score, color: reward

Should your team use it?

Training your own agent model is the right call far less often than it is proposed. The SRE-1 report is a useful reality check: it took environments built from 1,549 unique incidents, a multi-agent pipeline to write answer keys, leakage checks that excluded 63 of 1,663 candidate incidents for leakage alone, a 13-judge panel, and 7.4 days on up to 512 GPUs, after months of preparation. The payoff was real: on their test set, an ensemble of five SRE-1 runs matched the strongest frontier model they tested at 37% lower cost.

Use Molt whenUse something else when
You are doing agentic RL research and want to read every gradientYou need a managed training service with support
Your agent already speaks the OpenAI or Anthropic APIYour task can be solved with prompting, routing or distillation from a frontier model
You have GPUs that meet the CUDA 13 requirementYou have no reliable reward or answer key yet
You want one entry point from a small model to a large MoEYou need SFT-only training at small scale; simpler tools exist

Pros and cons

ProsCons
Small, readable RL path; the size claim holds upCUDA 13 and recent GPUs required; no CPU path for training
Reward is ordinary Python, testable before the clusterEnvironments, answer keys and leakage checks are entirely yours
Token-first contract avoids re-tokenization driftPartial-rollout tokens from old weights get no gradient
OpenAI and Anthropic SDK agents can be trained unchangedmolt-rl on PyPI lags the pinned dependencies and is not a supported install
One script from 4B to 753B MoEResearch-oriented: fewer guardrails than production platforms

What to say about it in an interview

If an interviewer asks how you would improve an agent with reinforcement learning, start with the reward and the environment, not the framework. "I would first prove I can grade the task: a held-out set, answer keys I trust, and checks that the environment cannot leak the answer. I would write the reward as testable code with deterministic checks in front of any judge. Then a framework like Molt is the cheap part." That answer shows you know where RL projects actually fail.

PRACTICE THIS

Turn it into offers. Work the real questions and concepts this maps to:

FAQ

Molt is an Apache-2.0 reinforcement learning framework from NVIDIA for training language-model agents. It uses Ray to place processes and carry queues, vLLM to generate rollouts and NeMo AutoModel with FSDP2 to train a single actor. The agent is a Python module you write, and its reward can be any Python: a grader, a multi-turn tool loop, a vision environment or an LLM judge.

Discussion (4)

Lukas HoffmannEditor

Molt's best design decision is that the reward is just Python next to the agent. That makes the reward reviewable like any other code, which matters because reward bugs are where RL projects go wrong. If you cannot unit-test your reward on a laptop, you are not ready for the cluster.

Mei LinEditor

Reader question: if the hard part is environments, why would anyone care which RL framework they use?

Arjun MehtaEditor

Because the framework decides how expensive it is to find out your environment is wrong. A small framework you can read lets one engineer trace a bad update to a token, a mask or a stale sample in an afternoon. A large one turns the same question into a week. Molt's pitch is legibility, and for research that is a fair thing to optimize.

Adam ReyesEditor

The token-first contract deserves more attention than it gets. ResolveAI found re-encoding text changed token ids in 94% of rollouts. Any stack that hands text back and forth between the sampler and the trainer is training on tokens the model never produced.