R2E-Gym: Procedural Environments and Hybrid Verifiers for Scaling Open-Weights SWE Agents

Open-weight coding agents lag behind proprietary ones, and the authors name two reasons: too few executable training environments, and weak ways to spend extra compute at test time.

For the first, they introduce SWE-GEN. Instead of starting from GitHub issues, which most commits do not have, they start from commits. Commits are filtered for small, focused fixes (at most five non-test files, 100 edited lines, 2,000 characters of patch). A Docker environment is built for each historical commit by searching dependency versions until one installs. Tests that fail before the commit and pass after (Fail-to-Pass tests) are taken from the commit or generated. Finally, a model writes an issue from the commit message, the patch and the failing test output, a step called back-translation. The result, R2E-Gym, has 8,135 tasks; 4,578 come from repositories outside SWE-bench and are used for training.

Fine-tuning Qwen-2.5-Coder on 3,321 successful trajectories from Claude 3.5 Sonnet, collected in these environments, gives a 32B agent that solves 34.4% of SWE-bench Verified on one attempt.

For the second, they sample 26 patches per problem and compare two ways to choose. Execution-based verifiers run tests: a trained testing agent writes reproduction tests and regression tests filter breakage. Execution-free verifiers are a judge model that reads the issue, the agent's trajectory and the patch and outputs a probability of YES. Each plateaus near 43%. A hybrid, which keeps the judge's top candidates and ranks them by test results, reaches 51.0%.

Two lessons apply well beyond SWE-bench.

You can build evaluation tasks from a customer's history. Any team with version control has thousands of small fixes and almost no tidy bug reports. R2E-Gym's recipe turns commits into tasks with a runnable environment, a test that defines success and a written problem statement. For a pilot, that means an evaluation set from the customer's own codebase instead of a public benchmark that may not resemble it.

Neither tests nor judges are enough on their own. Generated tests mostly fail to separate correct from incorrect patches: for most problems, under 20% of generated tests discriminate at all, and a few are "toxic", passing a wrong patch and failing the right one. A model judge discriminates better but leans on how confident the agent sounded; shown only the final patch, it dropped from 42.8% to 37.6%. Using the judge to shortlist and tests to decide gained more than seven points over either.

That pattern applies whenever you choose among several model outputs: support replies, SQL queries, extracted records. Use a cheap check that is reliable when it fires and a model judge for ranking, and do not let the judge read the agent's self-assessment if you can avoid it.

Know the data's lineage before calling the result open: the training trajectories came from Claude 3.5 Sonnet.