MemGPT: Towards LLMs as Operating Systems

A language model only knows what is in its prompt. For a companion chatbot that talks to someone for months, or an analyst reading a document longer than the window, that is a hard limit. Charles Packer, Sarah Wooders, Ion Stoica, Joseph Gonzalez and colleagues at UC Berkeley borrowed a fix from operating systems. Programs see a large virtual memory while the OS moves pages between RAM and disk. MemGPT gives a model the same illusion of unbounded context.

The prompt, called main context, has three parts. System instructions are read-only and describe the memory system and the functions available. Working context is a small block of text the model can edit through function calls, used for key facts about the user and the persona it plays. A FIFO queue holds recent messages, function results and system alerts, with a recursive summary of older messages at its head.

Outside the prompt, external context has two stores. Recall storage keeps every message ever exchanged. Archival storage holds arbitrary text, such as documents, with vector search. The model moves data between tiers itself: it can append to or replace its working context, search recall or archival storage page by page, and insert results back into the prompt. A queue manager warns it when the prompt reaches about 70% of the window, so it can save what matters, and at 100% evicts about half the queue into recall storage and updates the summary. A request_heartbeat flag lets the model chain several function calls before replying, and events such as user messages, uploads or timers trigger it to run.

On a multi-session chat benchmark, MemGPT answered questions about earlier sessions far more accurately than the same models given a summary, and wrote conversation openers that drew more on what it knew about the user. On documents, it could page through retrieval results beyond what fits in the window, and it solved multi-hop key-value lookups that defeated plain models.

Customers want assistants that remember: a support agent that knows the last three tickets, an analyst that keeps track of a long investigation, a coach that recalls goals from last month. MemGPT is the reference design for that, and the project behind it continues as open source.

Separate always-present facts from searchable history. A short working context (who the user is, their preferences, the current goal) belongs in every prompt. Everything else should live in storage the model can search. That split keeps prompts small and costs predictable.

Let the model see the budget. Warnings at a fill threshold give the model a chance to save important facts before eviction, instead of losing them silently.

Memory writes are product decisions. When the model decides what to store about a user, you need rules for what may be stored, how it is corrected, and how it is deleted on request. The paper does not address these; your deployment must.

Function-calling quality decides everything. With GPT-3.5, MemGPT lost much of its advantage because the model handled the memory functions poorly. Test the model you will deploy on the memory operations themselves.