Executable Code Actions Elicit Better LLM Agents

Most agents act by producing a tool call in a fixed format: a JSON object with a function name and arguments, or a line of text the harness parses. One call per turn, then the result comes back, then the next call. For a task like "find the cheapest of these four products and give the price in euros", that is five or six turns, each one re-reading every earlier call and result. The format also cannot say "do this for each item" or "if the first lookup fails, try the other site".

Xingyao Wang, Heng Ji and colleagues at the University of Illinois with Apple propose CodeAct: let the action be Python code, executed in an interpreter attached to the agent. A single action can loop over items, call several tools, store results in variables, use any library already installed, and print just what the model needs to see. When code fails, the interpreter's traceback comes back as the observation, and the model fixes its code in the next turn, as a programmer would.

The paper has two halves. First, an evaluation of 17 models comparing code, JSON and text actions on two benchmarks: API-Bank, where each action is a single API call, and M3ToolEval, a new set of 82 tasks that need several tools across several turns. Code actions were roughly even on single calls and clearly better on multi-tool tasks for most capable models. Second, CodeActInstruct, about 7,000 multi-turn trajectories, used to fine-tune Llama 2 and Mistral 7B into CodeActAgent, an open model that acts through code.

The action format is one of the first decisions in an agent build, and it is often made by default: whatever the framework does. This paper is the evidence for choosing deliberately.

Use code actions for compositional work. Data analysis, reconciliation across systems, batch operations over many records: anything where the natural plan contains a loop or a condition. That is where code actions won by large margins.

Keep structured calls for single, high-stakes actions. On single API calls, JSON was as good or better for the strongest models. A refund, a record update or a message send is one call with exact arguments; a schema validates it more simply than reviewing code.

Expect cheaper long tasks. Collapsing many turns into one program avoids re-reading a growing history on every step. The paper reports up to 30% fewer actions; for tasks with dozens of tool calls the token difference can be much larger.

Budget for the interpreter as a security boundary. Executing model-written code needs a sandbox: no network unless allowed, limited files, resource limits, and policy checks on the tools the code can call. The paper focuses on capability; deployments must add containment.

Small open models need training for it. Untuned 7B models scored near zero on the multi-tool benchmark. If the customer must run a small model, plan for fine-tuning or use structured calls.