LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory

An assistant that remembers its user has to do more than find a sentence. It has to recall details the user mentioned in passing, combine facts from several conversations, work out when something happened, notice that a fact has changed, and say "I don't know" when the user never said something. Di Wu, Kai-Wei Chang, Dong Yu and colleagues at UCLA, Tencent AI Lab and UC San Diego built LongMemEval to test exactly those five abilities: information extraction, multi-session reasoning, temporal reasoning, knowledge updates and abstention. It appeared at ICLR 2025.

The 500 questions were written and filtered by people from model-proposed drafts, each with evidence statements. Each statement is then placed in a simulated task-oriented chat in which the user reveals it indirectly: instead of "I bought a new car last month", the user asks for help with car insurance. People edited every evidence session. At test time, the evidence sessions are mixed with unrelated sessions, from simulated chats and public datasets such as ShareGPT and UltraChat, to make a history of any length. Two standard settings: LongMemEval S, about 115,000 tokens and about 50 sessions per question, and LongMemEval M, 500 sessions and about 1.5 million tokens. Answers are judged by GPT-4o, which agreed with human experts more than 97% of the time.

The pilot results were sobering. Long-context models that scored well when given only the evidence sessions dropped 30% to 55% when given the full S history. Two commercial assistants with memory features, ChatGPT and Coze, tested by hand on much shorter histories, fell far below GPT-4o simply reading those histories.

The second half of the paper uses the benchmark to study memory design as three stages, indexing, retrieval and reading, with four control points: what is stored (value), what it is indexed by (key), how the query is formed (query), and how the model reads what comes back (reading strategy).

When a customer wants an assistant that remembers, LongMemEval is a template for the evaluation you should build, and its findings are a starting point for the design.

Test the five abilities separately. A memory that finds facts may still fail on updates (the user moved), time (what did you suggest last week) or abstention (the user never said that). Each fails differently, and averages hide it.

Expect indirect mentions. Users rarely announce facts. They reveal them while asking for something else. Memory that only captures explicit statements misses most of what matters, which is how Coze failed in the pilot.

Do not rely on a long context alone. Even GPT-4o lost 30% of its accuracy reading a 115k-token history instead of just the evidence. Retrieval with good indexing beat stuffing the context.

Index rounds, add facts to keys, filter by time. Store individual exchanges rather than whole sessions, index each by its text plus extracted user facts, and use a strong model to turn time references into a date filter.

Reading needs help too. Even with perfect retrieval, how the model reads the retrieved items changed accuracy by up to 10 points.