When Large Language Models Meet Vector Databases: A Survey

A support assistant needs to find a refund policy, remember a customer's preferred language and avoid answering the same general question repeatedly. Vector search can help with all of these. The records it retrieves, and the conditions for using them, differ substantially.

This survey organizes the relationship between language models and vector databases around three roles: an external knowledge base, a semantic response cache and a store of longer-term memory. Embeddings represent records as vectors so a query can find nearby representations even when the wording differs. The application then retrieves the associated records and decides what to do with them.

For a knowledge base, the record is evidence used to compose an answer. For a semantic cache, it includes a previous answer that might be reused. For memory, it might be a preference or an event from an earlier interaction. A similar refund question from another customer is unsuitable for answer reuse if the answer depends on that customer's order. A remembered preference can also become wrong after the customer changes it.

The paper is a survey, with a useful taxonomy and discussion of limitations. It offers no controlled benchmark proving that one database makes all these tasks cheaper or more accurate. This explanation uses its June 2025 revision and separates the survey's categories from the application checks an FDE needs to build around them.

“We need a vector database” leaves most of a customer's requirements unresolved. Who may see a retrieved record? How quickly must a policy correction reach the assistant? Can an answer be reused across users? What should happen when a memory contradicts an authoritative source?

Answer those questions before comparing index speeds. A fast nearest-neighbor query can return perfectly similar records that the application cannot use. Conversely, an exact product code may be better served by a structured lookup even when the rest of the workflow uses semantic retrieval.

The survey provides a starting map for those decisions. Read it alongside the vector database concept, then choose one concrete customer operation to evaluate. For a policy assistant, test whether it retrieves current, permitted evidence. For a cache, measure incorrect reuse as well as hit rate. For memory, test a correction and a deletion all the way through to the next response. These are different acceptance tests, even if one storage engine serves every role.