Building an AI Agent with Dual-Memory Architecture
The problem: agent amnesia
A stateless agent forgets everything after each turn. It can’t hold a coherent multi-turn conversation or learn from past interactions, because nothing carries over. Fixing this — agent amnesia — means giving the agent a memory that mimics human recall: a fast, immediate working memory and a slower, durable long-term store.
Two kinds of memory
- Short-term memory (STM) keeps the current conversation coherent. Scope: this session.
- Long-term memory (LTM) persists key facts across all past conversations — user preferences, resolved issues, important context. Scope: everything the agent has ever learned.
Keeping them separate is the point: STM stays cheap and immediate, LTM stays searchable and durable.
Short-term memory: the rolling summary
The constraint on STM is the context window. Replaying the full transcript every turn is wasteful and eventually breaks. The fix is to summarize progressively rather than store verbatim:
- STM starts empty.
- After each turn, take the existing summary, the latest user query, and the latest response.
- Send all three to an LLM: “Concisely summarize this conversation, incorporating the new turn.”
- The updated summary replaces the old one in the agent’s state.
This rolling summary preserves the thread of a long conversation without spending tokens on a growing transcript.
Long-term memory: vector search
LTM lets the agent recall something from days or months ago. A vector database is the natural fit — see Vector Databases for how they work.
Storing: at the end of a meaningful interaction, the agent extracts memorable items (an LLM can identify these), converts each to an embedding, and writes the embedding plus its text to the database.
Recalling: on a new query, the agent embeds the query, runs a similarity search, and retrieves the most relevant past memories.
Because retrieval is by meaning, not keyword, a question about “latest project deadlines” can surface a stored memory about “timelines for the Q3 report.”
The integrated reasoning loop
The architecture earns its keep when both memories feed one loop:
- Query arrives.
- Retrieve from LTM — similarity search for relevant history.
- Assemble context — combine the retrieved memories with the current STM summary.
- Generate — the LLM answers using that combined context plus the new query.
- Update memory — refresh the STM summary, and decide whether anything new is worth embedding and storing in LTM.
Each pass makes the agent slightly more informed and more personalized. That’s the compounding loop dual memory is built to create.
Keep going
Memory is what separates a coherent assistant from a goldfish. Split it in two — a rolling summary for now, a vector store for always — and the agent gets to keep what it learns.
