Voyager: An Open-Ended Embodied Agent with Large Language Models

Minecraft has no fixed goal. Players explore, gather resources and climb a tech tree from wooden tools to stone, iron and diamond. Guanzhi Wang, Linxi Fan, Anima Anandkumar and colleagues at NVIDIA, Caltech and elsewhere built Voyager, an agent that plays it with GPT-4, without fine-tuning, and keeps getting better by accumulating skills.

It has three parts. An automatic curriculum asks GPT-4 to propose the next task, given the goal "discover as many diverse things as possible", the agent's inventory and surroundings, and the tasks it has completed and failed. A skill library stores every program that worked, indexed by an embedding of its description, so later tasks can retrieve and call it: craftStoneShovel(), combatZombieWithSword(). An iterative prompting loop writes each new skill as JavaScript against high-level Mineflayer control functions, runs it, and feeds back three kinds of signal: messages from the game ("I cannot make stone_shovel because I need: 2 more stick"), execution errors from the interpreter, and a verdict from a second GPT-4 call acting as a critic that checks the agent's state and says whether the task is done and, if not, how to finish it. After four failed rounds the curriculum moves on.

Using code as the action space matters. A program can loop, wait, and call other programs, so skills compose: the iron pickaxe skill calls the furnace and smelting skills, which call earlier ones. Because skills are stored, not retrained, old ones are not forgotten.

Over 160 prompting iterations, Voyager discovered 63 unique items, 3.3 times as many as baselines adapted from ReAct, Reflexion and AutoGPT, travelled 2.3 times farther, and unlocked tool tiers much faster. It was the only method to craft diamond tools. Given a new world and four unseen tasks, it solved all of them using its library.

Voyager is a game agent, but its memory design applies directly to agents that do repeated work for customers: a library of verified, reusable procedures that the agent builds as it goes.

Store what worked, as code. A skill that is a tested function can be reviewed, versioned and reused. Prose notes about "how I did it last time" cannot be executed or checked the same way.

Verify before storing. The self-verification step was the most important feedback in the ablations. In a customer setting, the verifier should be a real check where possible: a query result, a test, a state comparison. A library that stores failures poisons everything built on it.

Order the work. A random curriculum cut discoveries by 93%. Agents learning a new environment, and people onboarding agents into one, should start with tasks whose prerequisites are already in place.

Know what the model already knows. GPT-4 knew Minecraft from the internet: recipes, item names, strategies. A customer's internal systems are not in the model's training data. Expect the curriculum and the code to need far more help there.