NVIDIA Molt Review: An Agentic RL Framework You Can Read in an Afternoon, and What It Leaves to You
NVIDIA's open-source Molt trains agents with reinforcement learning on Ray, vLLM and NeMo AutoModel, with any Python function as the reward. We traced its code, checked its size claim, compared its choices with ResolveAI's SRE-1 training report, and wrote a reward that closes known shortcuts.
BY ARJUN MEHTA · FDEINTERVIEWS EDITORIAL · UPDATED NOVEMBER 19, 2026 · 8 MIN READ
PRACTICE THIS:How to train tool-using AI agents ·PPO and GRPO, the concept ·The SRE-1 training report, broken down ·LLM and GenAI questions
NVIDIA Molt is an open-source framework for training language-model agents with reinforcement learning, small enough to read end to end. Ray places the processes, vLLM generates the rollouts, NeMo AutoModel with FSDP2 trains one actor, and the reward is whatever Python you write next to your agent: a grader, a multi-turn tool loop or an LLM judge. The same entry point that trains a 4B model is documented to train GLM-5.2 (753B) with expert parallelism. We could not run training, which needs CUDA 13 GPUs, so we tested what we could: we traced the RL code path to check the size claim, compared Molt's design choices with ResolveAI's SRE-1 training report published the same week, and wrote a reward function that closes the shortcuts SRE-1's report says a policy learned. Our verdict: a good, honest research framework. The hard part of agentic RL, though, is the environment and the reward, and no framework writes those for you.
What Molt is
Molt sits in the family of OpenRLHF, verl and slime, and it says so. Its README compares itself with all three and credits OpenRLHF, whose package layout it keeps. What it optimizes for is legibility at scale: one trainable actor, an optional KL reference, every mechanism (advantage, loss, token capture, training loop) in one file, configured by command-line flags only.
Two agent interfaces cover most work:
Env | ChatAgent | |
|---|---|---|
| Who runs the model loop | Molt | Your code, through the OpenAI or Anthropic SDK |
| You implement | step(state), returning a reward and an optional next observation | run(ctx), returning a reward |
| Good for | Grading a turn, short tool loops | An existing agent or harness you do not want to rewrite |
The ChatAgent path is the interesting one for applied teams. Your agent points its SDK client at a session URL that Molt runs on loopback. The server behind it records every token with its log-probability, so the agent code contains nothing about training. A harness you already have can, in principle, become an RL environment without changes.
Supported algorithms include REINFORCE, RLOO, GRPO, Dr. GRPO, GAE (PPO) and on-policy distillation. The November release notes add REINFORCE Zero, which the README says kept asynchronous RL stable through 20,000 policy updates.
Checking the size claim
The README's headline number is "~9.2K lines" of RL code, with a careful footnote: every Python file the RL path imports, traced from the entry point, excluding SFT, tests, examples, blank lines and comment lines, measured on 20 July 2026. We traced the import graph from molt.cli.train_rl_ray at commit ec99ae4 (17 November 2026):
| What we counted | Lines |
|---|---|
| Files reached from the RL entry point | 53 |
| Non-blank, non-comment lines in those files | 10,125 |
| The same, with docstrings removed | 8,560 |
Every .py file in the molt package, physical lines | 15,133 |
The claim holds within the range set by whether you count docstrings, and the code has grown since July. One file, the vLLM worker extension, is loaded by name rather than imported, so a static trace misses it and the true RL path is slightly larger than our count. The broader point survives: the part of Molt that touches your gradients is about one order of magnitude smaller than verl's, which the same README puts at about 62,000 lines.
Read against a production run: what SRE-1 needed
ResolveAI's SRE-1 technical report, published on 18 November, is unusually candid about what it took to train a 27B agent with RL for production incident investigation. It did not use Molt; it used a modified fork of the open-source prime-rl. But its list of problems is a good checklist for any agentic RL framework, and it shows which problems are the framework's and which are yours.
| What SRE-1 found it needed | Does Molt give it to you? |
|---|---|
| Train on the exact token ids the sampler produced; re-encoding changed ids in 94% of rollouts | Yes. Its "token-first contract" keeps token ids, log-probs and action ranges aligned end to end |
| Overlap sampling and training, with a staleness bound | Yes. Asynchronous queues, partial rollout and per-token importance-sampling correction |
| A mask on extreme trainer-to-sampler probability ratios | Yes, several: token or sequence level, mask, clip or truncate |
| Keep in-flight rollouts across a weight update | A different choice. Molt keeps the prefix but drops gradient on tokens generated under old weights |
| Correct numerics: fp32 gradient accumulation and master weights | Mixed-precision handling is inside the training stack; check it for your model before trusting a long run |
| Environments whose tools see only data from before the alert | No. That is your environment |
| An answer key for incidents nobody wrote up, and leakage checks | No. That is your data pipeline |
| A reward the policy cannot game | Partly. The reward is your Python, which is the right place for it |
The mask comparison is worth a closer look, because the two projects made opposite-looking choices. SRE-1 masked tokens whose ratio left a wide band of 0.2 to 5, and reports that narrower bands it tried (0.8 to 1.25 and 0.92 to 1.08) dropped more tokens and lowered its validation score. Molt's recipes use a tight band of 0.99 to 1.01, but by default applied to a sequence's geometric-mean ratio, not to each token, alongside work to make trainer and inference kernels agree. These are not contradictory. The right band depends on what you apply it to and on how far your trainer and sampler disagree, so measure the mismatch first; Molt logs it as vllm_kl and mismatch/k3_kl.
The part you write: a reward that resists shortcuts
SRE-1's report lists the shortcuts its early policy learned against an LLM judge: restating the alert as the first causal link, listing ruled-out causes nobody had proposed, reversing causal order, and citing a README it never opened. Some of those need the judge. Others are cheap deterministic checks that belong in the reward, before the judge's score is used. Here is a reward of that shape, as plain Python you can unit-test on a laptop:
def rca_reward(answer, alert_text, evidence, judge_score):
"""answer = {"chain": [{"claim": str, "cites": [str]}], "candidates": int}
evidence = ids of the files the agent actually opened this episode."""
chain = answer.get("chain", [])
if not chain:
return 0.0, "no chain"
first = chain[0]["claim"].lower()
if first and first in alert_text.lower(): # restated the alert
return 0.0, "first link restates the alert"
cited = [c for link in chain for c in link.get("cites", [])]
if not cited or any(c not in evidence for c in cited): # cites a file it never opened
return 0.0, "citation not in gathered evidence"
if answer.get("candidates", 1) > 1: # hedged across several causes
return min(judge_score, 0.5), "several candidate causes: capped"
return judge_score, "ok"
We ran it on five answers to one illustrative alert ("p99 latency on checkout-api above 2s for 10 minutes"), with the judge score supplied by hand:
| Answer | Judge | Reward | Why |
|---|---|---|---|
| Three linked causes, each citing a file it opened | 0.86 | 0.86 | ok |
| First link restates the alert | 0.90 | 0.00 | first link restates the alert |
| Cites a README it never opened | 0.90 | 0.00 | citation not in gathered evidence |
| Same good chain, hedged across three candidate causes | 0.86 | 0.50 | capped |
| Empty chain | 0.70 | 0.00 | no chain |
Two rows got a high judge score and a zero reward. That gap is the reason deterministic checks sit in front of a judge. The checks are also easy to evade in other ways: a paraphrased alert passes the substring test. SRE-1's answer to that was to fix exploits in the judge as well, and to draw each group's judge at random from 13 models across providers, because judge errors were correlated within a provider. In Molt this function would sit inside an Env.step or at the end of a ChatAgent.run. Molt calls it; deciding what it rewards is your job.
Should your team use it?
Training your own agent model is the right call far less often than it is proposed. The SRE-1 report is a useful reality check: it took environments built from 1,549 unique incidents, a multi-agent pipeline to write answer keys, leakage checks that excluded 63 of 1,663 candidate incidents for leakage alone, a 13-judge panel, and 7.4 days on up to 512 GPUs, after months of preparation. The payoff was real: on their test set, an ensemble of five SRE-1 runs matched the strongest frontier model they tested at 37% lower cost.
| Use Molt when | Use something else when |
|---|---|
| You are doing agentic RL research and want to read every gradient | You need a managed training service with support |
| Your agent already speaks the OpenAI or Anthropic API | Your task can be solved with prompting, routing or distillation from a frontier model |
| You have GPUs that meet the CUDA 13 requirement | You have no reliable reward or answer key yet |
| You want one entry point from a small model to a large MoE | You need SFT-only training at small scale; simpler tools exist |
Pros and cons
| Pros | Cons |
|---|---|
| Small, readable RL path; the size claim holds up | CUDA 13 and recent GPUs required; no CPU path for training |
| Reward is ordinary Python, testable before the cluster | Environments, answer keys and leakage checks are entirely yours |
| Token-first contract avoids re-tokenization drift | Partial-rollout tokens from old weights get no gradient |
| OpenAI and Anthropic SDK agents can be trained unchanged | molt-rl on PyPI lags the pinned dependencies and is not a supported install |
| One script from 4B to 753B MoE | Research-oriented: fewer guardrails than production platforms |
What to say about it in an interview
If an interviewer asks how you would improve an agent with reinforcement learning, start with the reward and the environment, not the framework. "I would first prove I can grade the task: a held-out set, answer keys I trust, and checks that the environment cannot leak the answer. I would write the reward as testable code with deterministic checks in front of any judge. Then a framework like Molt is the cheap part." That answer shows you know where RL projects actually fail.
Turn it into offers. Work the real questions and concepts this maps to:
FAQ
Molt is an Apache-2.0 reinforcement learning framework from NVIDIA for training language-model agents. It uses Ray to place processes and carry queues, vLLM to generate rollouts and NeMo AutoModel with FSDP2 to train a single actor. The agent is a Python module you write, and its reward can be any Python: a grader, a multi-turn tool loop, a vision environment or an LLM judge.
The README says about 9,200 lines of RL code, measured on 20 July 2026 by tracing the import graph from the RL entry point and excluding blank and comment lines. Tracing the same entry point at the 17 November 2026 commit, we counted 10,125 non-blank, non-comment lines across 53 files, or 8,560 with docstrings removed. The whole molt package is 15,133 physical lines.
REINFORCE, REINFORCE with a baseline, RLOO, GRPO, Dr. GRPO, GAE with PPO, and on-policy distillation, plus asynchronous recipes such as FlashREINFORCE and REINFORCE Zero. It supports multi-turn agents through an Env interface, where Molt runs the loop, or a ChatAgent interface, where your code calls the OpenAI or Anthropic SDK against a session URL that records tokens.
Not for training. The supported install needs CUDA 13 and the container targets A100, H100, H200, B200 and GB200 GPUs. You can write and unit-test an agent's reward function on a laptop, because it is ordinary Python, and run the training job on a GPU cluster.
Only when a narrow task runs at high volume, frontier-model cost or latency is the blocker, and you can build environments with trustworthy answers. ResolveAI's SRE-1 report shows the expensive part is the environments, leakage checks and judges, not the training loop. Prove the evaluation first; the framework is the easy part.
Discussion (4)
Molt's best design decision is that the reward is just Python next to the agent. That makes the reward reviewable like any other code, which matters because reward bugs are where RL projects go wrong. If you cannot unit-test your reward on a laptop, you are not ready for the cluster.
Reader question: if the hard part is environments, why would anyone care which RL framework they use?
Because the framework decides how expensive it is to find out your environment is wrong. A small framework you can read lets one engineer trace a bad update to a token, a mask or a stale sample in an afternoon. A large one turns the same question into a week. Molt's pitch is legibility, and for research that is a fair thing to optimize.
The token-first contract deserves more attention than it gets. ResolveAI found re-encoding text changed token ids in 94% of rollouts. Any stack that hands text back and forth between the sampler and the trainer is training on tokens the model never produced.
