Mobile-Agent-v3: Fundamental Agents for GUI Automation

The paper introduces two things. GUI-Owl is a vision-language model trained to operate graphical interfaces on Android, Ubuntu, macOS and Windows: locate elements, answer questions about screens, plan and act. It is trained on data produced in a cloud fleet of virtual devices, where the model itself generates trajectories, a judging pipeline keeps the correct ones, and the improved model generates the next round. A final stage uses online reinforcement learning with a single advantage per whole trajectory.

Mobile-Agent-v3 is a framework that uses GUI-Owl in four roles.

• The Manager turns the user's instruction into an ordered list of subgoals, using retrieved knowledge, and updates the list after every step: moving finished subgoals, re-prioritizing, or inserting corrective ones after a failure.
• The Worker looks at the current screen, picks the most relevant subgoal it can act on, and performs one action, recording its reasoning and what it expected to happen.
• The Reflector compares the screen before and after with the Worker's intent and returns SUCCESS or FAILURE with a reason, such as "login failed: 'Invalid credentials' is visible".
• The Notetaker, triggered only after successful steps, saves information later screens will need, such as a verification code.

The loop stops when no subgoals remain or the Worker finds nothing it can execute.

On live benchmarks, GUI-Owl-7B acting alone scored 66.4 on AndroidWorld and 29.4 on OSWorld-Verified. Mobile-Agent-v3 with GUI-Owl-32B scored 73.3 and 37.7.

Many enterprise workflows run through applications with no usable API: an insurer's claims portal, a supplier's ordering site, a legacy desktop ERP screen. GUI agents are the fallback, and this paper is a clear reference design for making them more reliable.

Separate acting from checking. The Reflector's job is to ask whether the action did what the Worker intended, using the screen before and after. Silent failures (a tap that did not register, a dropdown that did not open) are the main way GUI agents go wrong, and an explicit check catches many of them.

Keep transient facts outside the screen history. A code shown once, an order number on a confirmation page: the Notetaker turns them into notes that persist after the screen changes.

Plans should change after failures. The Manager inserts corrective subgoals when the Reflector reports a failure instead of retrying blindly.

Two cautions. First, every role is another model call: roughly three per step instead of one, which multiplies latency and cost on long tasks; the paper reports neither. Second, the Reflector judges success from images. When the application exposes state (an accessibility tree, a DOM, a database row, an API you can read even if you cannot write), check that instead. The authors also found their Reflector leaned heavily toward saying SUCCESS, and had to rebalance its training data.