How to Build a Custom LLM Memory Layer

Every LLM call starts from zero. Unless you feed it information from earlier sessions, the model has no continuity — a hard limit for anything that needs to feel personalized. This guide builds a persistent memory layer from scratch, following the extract-embed-retrieve-maintain pattern popularized by memory frameworks like Mem0.

Two implementations, two data models. This guide uses DSPy plus a vector database — the right fit when you want semantic recall across a large, unstructured memory store. For a fuller, fully runnable version built on SQLite with scene consolidation — better when you want auditable, structured memory and lexical search — see Build a Self-Organizing Agent Memory System.

Memory is a context-engineering problem

Context engineering means filling the model’s window with exactly the information a task needs. Memory is its hardest case, because it chains several techniques together:

1. Extracting structured facts from raw text.
2. Summarization.
3. Vector storage and search.
4. Query generation and similarity retrieval.
5. Agentic tool-calling to decide when to do all of the above.

Architecture

A working memory system does four things — extract, embed, retrieve, maintain:

• Extract atomic memories from user-and-assistant turns.
• Embed those facts and store them in a vector database.
• Retrieve similar memories when the agent needs them.
• Maintain the store — add, update, or delete — as new information arrives.

Every step is optional at runtime. The agent should touch memory only when a query actually calls for it, not on every turn.

1. Extract memories with DSPy

The first job is turning transcripts into atomic factoids: short, self-contained facts that embed and retrieve cleanly. In DSPy, a signature declares the task; its docstring becomes the system prompt.

import dspy

class MemoryExtract(dspy.Signature):
    """
    Extract relevant information from the conversation. 
    Memories are atomic independent factoids that we must learn about the user.
    If transcript does not contain any information worth extracting, return empty list.
    """
    transcript: str = dspy.InputField()
    memories: list[str] = dspy.OutputField()

memory_extractor = dspy.Predict(MemoryExtract)

Call the predictor with conversation history to get a list of factoids back.

2. Embed and store

Embed each factoid with an efficient embedding model and store it in a vector database. Wrap the store in helper functions for insert, update, delete, and search, and filter every operation by user_id so one user’s memories never surface for another.

3. Retrieve via tool-calling

Rather than searching on every turn, give a tool-calling agent a fetch_similar_memories tool and let it decide when to reach for it. A dspy.ReAct agent fits well: it observes the conversation, reasons about the next step, then either answers directly or calls the tool to pull memories first. The same agent also flags when the latest turn contains something worth saving.

4. Maintain the store

Memory is not an append-only log. When something new should be saved, a separate maintenance agent decides how to integrate it — implemented as four tools:

• add_memory(text) — insert a new fact.
• update_memory(id, updated_text) — correct or refine an existing one.
• delete_memories(ids) — remove obsolete or contradicted facts.
• no_op() — do nothing when the information is irrelevant or already stored.

This loop keeps the store accurate as it grows, so the primary agent’s answers get more personalized over time instead of drifting.

Where to take it next

• Graph memory — store facts as triples in a graph database to capture relationships between them.
• Metadata filtering — tag memories by category (food, hobbies, work) for more targeted retrieval.
• System-prompt injection — automatically inject a user’s critical facts into the system prompt at the start of every session.