Hindsight is 20/20: Building Agent Memory that Retains, Recalls, and Reflects

Most memory layers for agents extract snippets from conversations, store them in a vector or graph store, and paste the top few into the prompt. Hindsight argues that this blurs three things an assistant should keep apart: what it observed, what it did, and what it concluded.

Its memory bank has four networks. The world network holds objective facts ("Alice works at Google"). The experience network holds the agent's own actions, in the first person ("I recommended Yosemite to Alice"). The opinion network holds judgments with a confidence score between 0 and 1. The observation network holds neutral summaries of entities, rebuilt in the background when their facts change.

Three operations work on the bank. Retain turns conversations into two to five narrative facts each, self-contained paragraphs rather than single sentences, with time ranges and entities, and links facts that share an entity, are close in time, are semantically similar, or are causally related. Recall runs four retrieval channels in parallel (vector similarity, BM25 keyword search, spreading activation through the link graph, and a temporal filter), fuses them with Reciprocal Rank Fusion, reranks with a cross-encoder, and returns as many facts as fit a token budget. Reflect answers with a configured disposition (skepticism, literalism, empathy, each 1 to 5) and can form new opinions or adjust the confidence of existing ones when evidence arrives.

On LongMemEval, the same 20B open model scored 39.0% with the whole history in context and 83.6% with Hindsight.

When an enterprise assistant says "this customer prefers email" or "this supplier is unreliable", someone eventually asks why. If memory mixes the customer's own words with the assistant's summary and a judgment it made last month, the answer cannot be traced. Hindsight's main contribution is a structure that keeps those apart.

Store facts, actions and judgments separately. A fact has a source; an action has a time and an actor; a judgment has a confidence and can change. Retrieval for a factual question should not return judgments as if they were facts.

Retrieve through several routes. Similarity search misses the fact about the same person from three months ago, or the event that caused the current problem. Keyword search catches exact names and IDs; graph links catch shared entities and causes; a time filter catches "what happened in June". Rank fusion combines them without tuning score scales.

Prefer narrative facts. Storing one paragraph per exchange ("Alice and Bob considered three names and chose Beach Beats because...") keeps the reasoning that five separate one-line facts lose.

Be careful with the numbers. The comparison systems' scores were copied from other reports that used a GPT-4o judge, while Hindsight was judged by a different model. The token budget for retrieval, the setting that most affects cost, appears in the paper as a literal placeholder, "<add> tokens". There are no ablations showing which of the four networks or channels matters, and no ingestion cost.