LLM Knowledge Base Architecture: The Karpathy Pattern
A way to give an LLM durable, auditable access to a body of knowledge without a vector database — by making the model maintain a structured Markdown wiki instead.
Andrej Karpathy — former Director of AI at Tesla, an OpenAI co-founder, now an independent researcher — described this approach in a widely shared post, calling it an “LLM Knowledge Base.” The idea inverts the usual retrieval setup: rather than embedding documents and searching them, the LLM works as a full-time research librarian that compiles, lints, and interlinks a library of Markdown (.md) files. Karpathy’s framing: “You rarely ever write or edit the wiki manually; it’s the domain of the LLM.”
The result is a “Second Brain” that is self-healing, auditable, and entirely human-readable — because tokens go into curating knowledge rather than shuffling boilerplate.
The three-stage pipeline
Standard RAG chunks documents, embeds them into vectors, stores them in a specialized database, and retrieves by similarity at query time. The Karpathy pattern drops that machinery for mid-sized corpora and leans on the LLM’s growing ability to reason over structured text directly. It runs in three stages.
1. Ingest. Raw materials — papers, repositories, datasets, web articles — land in a raw/ directory. Karpathy uses tools like the Obsidian Web Clipper to convert web pages into Markdown, storing images locally so the model can reference them through its vision capabilities.
2. Compile. This is the core move. Instead of indexing the raw files, the LLM compiles them: it reads the source material and writes a structured wiki — summaries, identified key concepts, encyclopedia-style articles, and, critically, backlinks between related ideas. Here the model acts as a librarian, not a search engine, and its output is readable, navigable, and editable.
3. Maintain (lint). The wiki isn’t frozen. Periodic health-check or “linting” passes scan for inconsistencies, gaps, and new connections to draw. One practitioner described the effect as a knowledge base that heals itself.
Because the Markdown files are the source of truth, there is no black box: every claim traces back to a specific .md file a human can read, edit, or delete. That connects directly to dual-readability principles.
Three pillars
Markdown as source of truth. Markdown is compact, LLM-friendly, and vendor-neutral. If the surrounding app disappears, the files still open in any text editor — the “file-over-app” philosophy behind tools like Obsidian.
The LLM as maintainer. Traditional knowledge management has humans curate and software index. This pattern flips that: the human owns the data, but the LLM is the wiki’s primary author and maintainer.
Structure over similarity. RAG retrieves by semantic similarity — nearest neighbors in vector space. This pattern relies on explicit structure — backlinks, indices, curated summaries — producing a knowledge graph you navigate rather than a store you query. See Agentic Markdown Programming for Markdown as an instruction layer.
Extending to multiple agents
Community architects have extended the pattern to multi-agent systems, sometimes called a “Swarm Knowledge Base.” The danger there is compounding: one agent’s hallucination can infect the collective memory.
The proposed guard is a dedicated quality gate — an independent supervisor model (Nous Research’s Hermes has been cited as an example of a model tuned for structured evaluation) that scores and validates each draft article before it is promoted to the live wiki. That forms a loop:
- Agents dump raw outputs into a working area.
- The compiler organizes them into structured drafts.
- The quality gate validates the drafts.
- Verified briefings feed back to the agents at the start of each session.
No agent “wakes up blank” — each starts from a filtered, high-integrity summary of what the group has already learned.
Keeping the record clean
Steph Ango, co-creator of Obsidian, added a useful refinement: keep the trusted personal vault clean and let agents experiment in a separate “messy vault,” promoting only the distilled, useful artifacts into the canonical store. This contamination mitigation draws a hard line between agent scratch space and the human-trusted record.
Where it scales — and where it doesn’t
The common objection to non-vector approaches is scale. Karpathy notes that at roughly 100 articles and 400,000 words, an LLM can navigate the wiki through summaries and index files perfectly well — and that traditional RAG can add more latency and retrieval noise than it removes at that size. For a departmental wiki, a research project, or a focused domain, structure can beat vector retrieval on both speed and traceability.
The pattern also supports an “ephemeral wiki”: a temporary mini-knowledge-base spun up for one task and dissolved when the work is done — pointing toward spawning task-specific research environments rather than just chatting with a model.
Vector RAG vs. Markdown wiki
| Vector DB / RAG | Markdown wiki | |
|---|---|---|
| Data format | Opaque vectors | Human-readable Markdown |
| Retrieval logic | Semantic similarity | Explicit backlinks and indices |
| Auditability | Low (black box) | High (direct traceability) |
| Upkeep | Static; needs re-indexing | Active; self-heals through linting |
| Ideal scale | Millions of documents | 100 – 10,000 high-signal documents |
A useful mental image: the vector store is a vast warehouse with a very fast forklift — you can find anything, but not why it’s there or how it relates to the next pallet. The Markdown wiki is a curated library with a head librarian constantly writing new books to explain the old ones.
When to use it
Well-suited to:
- Mid-sized, high-signal corpora (roughly 100 to 10,000 documents).
- Research and writing where traceability matters.
- Personal knowledge management and “Second Brain” systems.
- Departmental or domain wikis with clear ownership.
- Multi-agent systems where a quality gate can vet outputs before they enter shared memory.
Less suited to:
- Massive collections (millions of records).
- Real-time streams that can’t be compiled into stable artifacts.
- Purely unstructured retrieval where similarity is the only signal that matters.
At the boundary, a hybrid — Markdown as source of truth with an optional vector layer for breadth — can combine both. See RAG for the complementary pattern.
The end state: synthetic data and fine-tuning
As the wiki grows and continuous linting makes the data “purer,” it becomes a strong training set. Rather than reading the wiki into context at query time, you could fine-tune a smaller model on the wiki itself, so the knowledge lives in the model’s weights — turning a research project into a private, custom intelligence. This connects to the broader fine-tuning discussion.
Why it matters
For an individual, the pattern ends the “forgotten bookmark”: knowledge compounds instead of scattering. For an organization, it marks the shift from a raw data lake to a compiled knowledge asset. Most teams drown in unsynthesized material — chat logs, stale wikis, PDFs nobody has time to read. An LLM-maintained layer wouldn’t just search those; it would actively author a living organizational record that stays current.
The deeper claim is about the model’s role — moving from one-shot conversational partner to autonomous archivist. The user owns the data; the AI is a sophisticated editor that visits the files to do the work. It is the beginning of LLM-maintained memory.
