IN THIS BREAKDOWN · 10 SECTIONS
You can read 19% of this breakdown.
The paper in one minute
By mid-2024 many teams were building agents that write code, run commands and browse the web, inspired by Cognition's Devin. Each had its own way of representing actions, executing them, keeping history and sandboxing. That made agents hard to compare and hard to build on. Xingyao Wang, Graham Neubig and a large group of collaborators from universities and companies built OpenHands (formerly OpenDevin) as a shared platform.
It has three core pieces. An agent is a class with a step method: it receives the current state and returns one action. The state includes an event stream, an ordered log of every action and observation, from the agent, the user and the environment, plus bookkeeping such as accumulated model cost. The runtime executes actions inside a Docker container per session: a bash shell, a Jupyter IPython kernel for Python, and a Playwright-driven Chromium browser using BrowserGym's action set. The agent never touches the host; the runtime turns each action into an observation and appends it to the stream.
Around these sit a library of reusable tools (AgentSkills, Python functions preloaded in the sandbox), a delegation action that lets one agent hand a subtask to another, support for running on any user-provided Docker image, a chat interface that shows what the agent is doing, and an evaluation harness covering 15 benchmarks. The default generalist, CodeActAgent, acts mainly by writing code. With one interface, it came close to specialist systems on software engineering, trailed them on web navigation, and beat prior results on graduate-level science questions.
Research notes: the problem, approach and results
benchmarks run through the same agent and runtime interface
SWE-bench Lite resolved by the generalist CodeActAgent with Claude 3.5 Sonnet, at $1.10 per issue
contributors and 2.1K contributions to the open-source project at writing
Every research agent shipped its own loop, tools and sandbox, so agents could not be compared, reused or safely run against real environments. Building a software-engineering agent meant rebuilding the plumbing each time.
Define an agent as a step function from state to action, record every action and observation in an event stream, execute actions in a Docker sandbox with a shell, a Python kernel and a browser, and support delegation, shared tools and benchmark harnesses on top.
An MIT-licensed platform with over ten agents and 15 benchmarks wired in. Its generalist CodeAct agent reached 26% on SWE-bench Lite, 15% on WebArena and 53% on GPQA, close to or ahead of task-specific systems, with one interface for all.
Why it matters to an FDE
Whether you adopt OpenHands or build your own, its architecture is a good reference for the plumbing every customer agent needs. Most production problems with agents are plumbing problems: where code runs, what was done, and how to debug it.
Keep a complete, ordered event log. Every action and observation, with who issued it. It is your audit trail, your debugging tool and your evaluation data, and it lets you replay or resume a session.
Separate decision from execution. The agent proposes actions; a runtime executes them under its own rules. That is where you enforce sandboxing, permissions and resource limits, independent of which model or agent design you use.
Sandbox per session. A fresh container with only the customer's workspace mounted limits the blast radius of a bad command or an injected instruction.
Make agents swappable. A common interface lets you test a new model or agent design against the same tasks without touching the runtime.
Delegate to specialists. A generalist that hands web tasks to a browsing agent, or reviews to a checking agent, keeps each component simpler.
Understand the mechanism. Try it on a real problem.
Unlock this breakdown and 99 more papers.
Premium opens every paper in full, plus every answer, concept, course and guide on the site. One payment for six months of access.
6 months · One payment · No auto-renewal
- ◇1 architecture diagram
- </>1 runnable code experiment
- ▦2 results tables
- ?5 interview questions with answers
- 03Key takeaways
The agent abstraction, runtime, tools, delegation, evaluation, and misreadings to avoid
Free account - 04How it works
The control loop, the runtime API, the browser, tools, delegation and the evaluation harness
Premium - 05The numbers that matter
Results across coding, web and assistance tasks, costs, and how to read them
Premium - 06Try it yourself
Build a miniature event-stream agent platform with a sandboxed runtime, delegation and replay
Premium - 07Where it breaks
The limits of the platform and of its evaluation
Premium - 08Using it on a deployment
Running developer-style agents for a customer with the right boundaries
Premium - 09Interview angles
Five questions this paper prepares you for, with answer hints
Premium - 10Read next
Where to go from here on this site
Premium
