Vector Databases

A vector database stores, manages, and searches high-dimensional vectors — the embeddings that machine-learning models produce. It’s the retrieval layer behind anything that needs semantic matching: Retrieval-Augmented Generation (RAG), recommendation engines, and semantic search.

Why ordinary databases fail

Traditional indexes like B-trees are built for exact matches and range queries on low-dimensional data (userID = 123, price < 50). They can’t help with similarity search over embeddings, because:

• They have no way to index the geometric relationships between vectors.
• Brute force — comparing the query vector against every stored vector — doesn’t scale. A million 1,536-dimensional vectors is on the order of a billion floating-point operations per query.

The core idea: approximate nearest neighbor

Vector databases trade perfect accuracy for large speed gains using Approximate Nearest Neighbor (ANN) search. The insight: finding the almost-nearest neighbors is nearly as useful as finding the exact ones, and can be orders of magnitude faster.

Three indexing algorithms dominate:

HNSW (Hierarchical Navigable Small World). Builds a multi-layer graph; search starts at a sparse top layer and descends, greedily moving toward the query vector.
– Pros: very fast, high recall.
– Cons: memory-hungry — the whole graph lives in RAM.

IVF (Inverted File Index). Partitions the vector space into clusters (e.g., via k-means); a query searches only the most relevant clusters (nprobe).
– Pros: lower memory than HNSW; handles datasets larger than RAM.
– Cons: lower recall than HNSW at the same speed.

PQ (Product Quantization). Splits vectors into sub-vectors and quantizes them, shrinking the memory footprint and speeding distance math.
– Pros: large memory savings, faster scans.
– Cons: compression costs accuracy. Often paired with IVF (IVF-PQ) for a strong hybrid.

The recall vs. latency trade-off

In production you tune two quantities against each other:

• Recall — the share of true nearest neighbors the search actually returns.
• Latency — how long a query takes.

Widening the search scope (ef_search in HNSW, nprobe in IVF) raises recall but also latency. For most applications, recall in the low-to-mid 90s is the sweet spot; chasing the last few percentage points can multiply query time for a gain users never notice.

Do you actually need one?

• Yes — millions of vectors and a need for low-latency search in production (RAG, semantic search).
• No — under ~100k vectors, a brute-force search with NumPy or FAISS is usually fast enough and avoids the operational overhead.

The options

• Managed: Pinecone
• Open-source / self-hosted: Weaviate, Chroma, Qdrant, Milvus
• Extensions to existing databases: pgvector for Postgres; native HNSW indexes in Elasticsearch and OpenSearch.

Keep going

• Building an AI Agent with Dual Memory
• Architecting a Private Multi-User RAG Agent
• Introduction to AI Agents

A vector database is where an agent’s long-term memory and a RAG system’s knowledge actually live. Whenever a design mentions embeddings, similarity search, or “the retrieval part of RAG,” this is the component doing the work.