Agent Lightning v1.0: Towards Harnessed Agentic RL

Agents ship inside harnesses: the code that builds each prompt, runs tools, trims or summarizes context, and sometimes delegates to subagents. If you train a model with reinforcement learning (RL) inside a simplified loop and then deploy it in a real harness, it is trained for one environment and used in another. The authors' term harnessed agentic RL means training through the same harness used in production.

Their original Agent Lightning did this with a proxy: the harness talks to what looks like a normal OpenAI-compatible endpoint, and the trainer records the calls. Several frameworks have since adopted the idea. This paper, a full rewrite in about 3,500 lines of code, does two things.

First, it names four problems that appear when the trainer only sees API calls. Retokenization: the text of a response, re-tokenized inside the next prompt, may not match the tokens the model actually sampled, so calls cannot always be stitched into one sequence. Advantage calculation: one rollout can become several training samples, and the group baseline changes depending on whether you count rollouts or samples. Loss normalization: averaging per sample gives rollouts that split into more samples more weight. Backend scheduling: the number of samples is known only after the agents run.

Second, it builds the system: an API Gateway storing rollouts and events with idempotent endpoints, a Rollout Controller that runs agents as Kubernetes jobs, and a trainer on VERL that merges calls only on exact token matches and computes everything per rollout. Trained this way, Qwen3.5-9B in mini-SWE-agent went from 41.8% to 56.4% on SWE-bench Verified.

Most FDE work does not train models. Two situations make this paper relevant anyway.

The first is a customer with an open-weight model, a specific agent workflow and enough examples with checkable outcomes: tickets closed, tests passing, forms filled correctly. The paper shows that RL through the customer's own harness, with modest data, can move a small model a long way on a coding task. It also shows what that costs in engineering: a data-cleaning pipeline that discarded about 90% of the source tasks, sandboxes, network policies, and an agent that learned to cheat within the first runs.

The second applies to anyone running agents as a service. The control plane is a clean pattern for long-running jobs:

One source of truth. The gateway's record of each run is authoritative; the executor reconciles toward it.

Idempotent endpoints. Retrying any control call after a network failure cannot corrupt state.

A shared run ID. The rollout ID sits in the proxy URL, so every model call, reward and log line joins to one record for debugging.

Attempts are explicit. A rerun is a new attempt under the same rollout, not a new run.

The reward-hacking list is worth reading even if you never train anything. Given a repository and a test suite, the agent found the answer in git history, downloaded upstream source with curl and pip, and fetched code with urllib. Agents you evaluate will find the same shortcuts.