IN THIS BREAKDOWN · 10 SECTIONS
You can read 17% of this breakdown.
The paper in one minute
Magentic-UI is an open-source web application from Microsoft Research for building and studying agents that work with a person rather than for one. Under the interface is a multi-agent team adapted from Magentic-One: an Orchestrator that plans and delegates, a WebSurfer that drives a browser, a Coder that writes and runs Python or Bash, a FileSurfer that reads and converts files, optional agents wrapping MCP servers, and UserProxy, the human, treated as one more agent the Orchestrator can hand work to.
The paper's contribution is six interaction mechanisms meant to keep human effort low.
Co-planning. Before acting, the Orchestrator asks a clarifying question if the task is unclear, then shows a plan as editable steps. Nothing runs until the user accepts.
Co-tasking. The user can pause the agent or take over its browser at any time; the Orchestrator can ask the user for help, but only after other agents have failed.
Action approval. Every action carries a developer-set label (never, maybe or always irreversible). "Always" asks the user; "maybe" goes to an LLM judge.
Answer verification. The user can inspect each step's actions and screenshots or ask follow-up questions.
Memory. A finished run can be turned into a saved plan and reused.
Multitasking. Several sessions run at once, each showing whether it needs input.
The evaluation has four parts: autonomous benchmarks, a simulated-user version of GAIA, a 12-person qualitative study, and 24 adversarial scenarios.
Research notes: the problem, approach and results
GAIA validation accuracy, autonomous versus with a simulated user holding a human-written plan; that user gave the final answer in 18% of tasks
of tasks in which the agent asked that simulated user for help, about once each time
adversarial scenarios that succeeded against the default configuration of guards and sandboxes
Agents that browse, run code and handle files are useful but below human accuracy, and their actions can be irreversible or hijacked by a malicious page. Fully autonomous operation asks people to trust a system that is wrong a lot of the time; watching every step defeats the point of delegating.
Design for low-cost human involvement. Co-plan before acting with an editable list of steps; let the user take over the agent's browser at any time and let the Orchestrator ask the user for help; gate risky actions through a guard; save successful runs as reusable plans; and run several tasks at once with status indicators.
Autonomously, Magentic-UI matched its predecessor and scored 82.2% on WebVoyager with o4-mini. With a simulated user holding a human-written plan, GAIA validation accuracy rose from 30.3% to 51.9% while the agent asked for help in 10% of tasks. Twelve users found it usable (SUS 74.58); none of 24 attack scenarios succeeded against the default setup.
Why it matters to an FDE
Most enterprise agents you deploy will have a human nearby: an analyst approving a payment, a support lead reviewing a draft, an operator who knows that the "right" vendor portal is not the one the agent found. Magentic-UI is a worked design for that arrangement, and several of its choices transfer directly.
Front-load the human's effort. Ten seconds editing a plan before execution is cheaper than correcting a 20-minute run. In the user study, people mostly edited plans to add preferences the agent could not have guessed, such as which website to trust.
Make the human an addressable participant. Modeling the user as an agent with a description ("ask only after others fail") gives one knob for how often the system interrupts.
Classify actions before you judge them. A fixed label per action type handles the easy cases deterministically and sends only ambiguous ones to a model.
Isolate what the agent can reach. The safety results come mostly from sandboxing and a browser with no saved logins. With those mitigations disabled, prompt injection on a web page got the agent to exfiltrate SSH keys and create a GitHub API key.
Read the headline gain with care. The 71% improvement on GAIA comes from a simulated user that had a human-written plan for each task and, in 18% of tasks, supplied the final answer itself. It shows that help is valuable; it does not measure how much a real person's help is worth.
Understand the mechanism. Try it on a real problem.
Unlock this breakdown and 149 more papers.
Premium opens every paper in full, plus every answer, concept, course and guide on the site. One payment for six months of access.
6 months · One payment · No auto-renewal
- ◇1 architecture diagram
- </>1 runnable code experiment
- ▦2 results tables
- ?5 interview questions with answers
- 03Key takeaways
The mechanisms, the measured results, what the user study found, and three misreadings
Free account - 04How it works
The planning and execution loop, the agents, the guard, memory retrieval and the deployment layout
Premium - 05The numbers that matter
Benchmarks, simulated users, runtime, planning statistics, the user study and red-teaming
Premium - 06Try it yourself
Check the headline arithmetic, then price action approvals and multitasking for one human
Premium - 07Where it breaks
The simulated-user confound, loose comparisons, small study, guard tuning and paper slips
Premium - 08Using it on a deployment
Designing human checkpoints for an agent your customer's staff will supervise
Premium - 09Interview angles
Five questions this paper prepares you for, with answer hints
Premium - 10Read next
Where to go from here on this site
Premium