Vending-Bench: A Benchmark for Long-Term Coherence of Autonomous Agents

Andon Labs built a benchmark where each step is easy and the difficulty is staying on track for a long time. An LLM agent owns a vending machine with 12 slots and starts with $500. Every day costs $2. To make money it must find products, email wholesalers to order them, wait for delivery, have them stocked, set prices, and collect the cash.

The agent works in a loop that keeps the last 30,000 tokens of history. It can write to a scratchpad, a key-value store and a vector database, read and send email, search the web, and check inventory and money. Physical actions, such as stocking the machine, collecting cash and setting prices, go through a sub-agent it instructs. Supplier replies are generated by GPT-4o from real information about the wholesaler the agent contacted. Customer demand comes from a model of price elasticity per product, with weekday, month and weather effects. A run lasts 2,000 messages, around 25 million tokens and 5 to 10 hours, or ends if the agent cannot pay the fee for 10 days in a row. Each model ran five times. The score is net worth: cash, cash in the machine, and unsold stock at cost.

Claude 3.5 Sonnet averaged $2,217.93 and o3-mini $906.86, ahead of a human who played once for five hours and ended at $844.05. Every model also had runs that sold nothing. The common failure is mundane: the agent assumes an order has arrived on the expected date, the sub-agent reports nothing to stock, and the agent concludes something is broken. From there runs went off course: Sonnet tried to report the $2 fee to the FBI, Claude 3.5 Haiku sent escalating legal threats to a supplier, and o3-mini typed out tool calls as text for about 1,300 messages. The authors find no clear relation between collapse and the memory window filling.

Plan for variance across runs, not just average quality. Sonnet's best runs made thousands of dollars and its worst lost money. A customer deploying an agent for an ongoing process experiences individual runs, so report the worst case and the share of runs that derail, not only the mean.

Give the agent a way to check state rather than assume it. Every collapse in the paper began with a belief about the world (the order arrived) that the agent did not verify. Design tools that return ground truth on demand (open orders, delivery status) and prompts that require checking before acting on an assumption.

Detect derailment and stop. The derailed runs kept consuming tokens for hundreds of messages. Production agents need monitors for loops, repeated failures and off-task actions, with a path to a human.

Compare with doing nothing. An agent that never acted would end each run with $500 less $2 a day. Some models' means were close to that. For any agent that manages money, inventory or tickets, compute the do-nothing outcome before claiming value.

Memory tools are not used just because they exist. Sonnet wrote a detailed daily summary to its scratchpad and never read it back. If the design depends on the agent consulting its notes, make reading them part of the loop.