Embeddings and Vectorization: Turning Meaning into Math

What embeddings are

An embedding is a list of numbers — a high-dimensional vector — that captures the meaning of a piece of content. The process that produces it is vectorization. Convert enough text this way and something useful happens: concepts that mean similar things land close together in vector space, whether or not they share any words.

That is the whole trick. Keyword search asks “does this string appear?” Embeddings ask “is this about the same thing?” A model that has learned good embeddings places king nearer to queen than to cabbage — a distinction plain text matching cannot make. It is the foundation of semantic search, RAG, and recommendation systems.

	Keyword search	Semantic search (embeddings)
Logic	Does the document contain the exact string “CEO”?	Are “CEO,” “chief executive,” and “company leader” close in meaning?
Query behavior	“CEO role description” matches only that phrasing.	Finds “responsibilities of a chief executive” even with “CEO” absent.
Mechanism	String comparison	Distance between vectors (e.g. cosine similarity)

The pipeline

Building a searchable knowledge base for RAG follows four steps:

1. Chunk — split large documents into smaller, self-contained segments so each embedding stays focused.
2. Embed — pass each chunk through an embedding model to produce its vector.
3. Store — save the vectors, their source text, and metadata in a vector database.
4. Retrieve — vectorize the incoming query and run a similarity search to pull the chunks whose vectors sit closest.

Embedding models

The model does the vectorization, and its choice drives both quality and cost.

Provider	Example models	Notes
OpenAI	text-embedding-3-small, text-embedding-3-large	Widely used, strong performance, simple API; supports variable dimensions.
Cohere	embed-english-v3.0	Strong multilingual coverage; supports compressed embeddings.
Open source (Hugging Face)	Sentence-Transformers, BGE	Free, self-hostable, fine-tunable — at the cost of running your own infrastructure.
Google	text-embedding-004 (Vertex AI)	Tuned for the Google Cloud stack; benchmarks well.

An embedding’s length is its dimensionality (for example, 1536 dimensions for OpenAI’s ada-002). More dimensions can hold more nuance but cost more to store and compute.

Vector databases

These are built to index and query high-dimensional vectors at speed.

Database	Type	Strengths
Pinecone	Managed	High-performance, scalable, API-first.
Weaviate	Open source / managed	Hybrid (keyword + vector) search; built-in GraphQL.
Chroma	Open source	Lightweight, in-memory — ideal for development and small workloads.
Supabase (pgvector)	Postgres extension	Adds vectors to a relational database, keeping data types unified.

They rely on approximate-nearest-neighbor indexes such as HNSW (Hierarchical Navigable Small Worlds) to search millions of vectors far faster than a brute-force scan.

How similarity is measured

Retrieval hinges on measuring how close two vectors are, and the usual measure is cosine similarity — the cosine of the angle between them:

• 1 — same direction, effectively identical meaning.
• 0 — orthogonal, unrelated.
• -1 — opposite direction, opposite meaning.

Ask “What are the duties of a CEO?” and the system surfaces a chunk beginning “The responsibilities of a chief executive include…” because the angle between those two vectors is small.

Where embeddings show up

Application	Role of embeddings
RAG	Powers the retrieval step — finds the most relevant context to feed an LLM. The dominant use case.
Semantic search	Search that reads intent, not just keywords.
Recommendations	Surfaces items whose vectors resemble what a user already liked.
Clustering	Groups similar documents or images with no predefined labels.
Anomaly detection	Flags points that sit far from every cluster as unusual or suspect.

Best practices

Output quality tracks embedding and retrieval quality closely.

Practice	Why it matters
Chunk deliberately	Small chunks are precise but context-poor; large ones carry context but add noise. Semantic chunking (by paragraph or section) usually beats fixed-size splits.
Match the model to the domain	Pick for domain and budget; specialized fields (legal, medical) may warrant fine-tuning an open-source model.
Store rich metadata	Keep source, date, and author beside each vector to enable pre-filtering (search within a category) and post-filtering. Structured fields such as a semantic summary and key concepts measurably lift retrieval accuracy.
Add hybrid search	Pair vector search with keyword search to catch exact identifiers — product names, error codes — that pure vector search can miss.
Preprocess consistently	Clean and normalize text (strip HTML, stray characters) before embedding.
Evaluate retrieval	Track hit rate and Mean Reciprocal Rank (MRR) so you know retrieval is actually finding the right chunks.

Writing text for embedding

What you feed the embedding model matters. Text written to read well — with introductions, transitions, and calls to action — embeds worse than dense, factual text, because the filler dilutes the signal. The prompt below produces a clean representation optimized for retrieval rather than for a human reader.

Objective: Generate a dense, semantic representation of the following content for vector embedding. Capture core meaning, concepts, and relationships for machine understanding.

Instructions:
1. Identify the primary topic and purpose of the content.
2. Extract all key concepts, technical terms, named entities (products, people, organizations), and specific data points.
3. Summarize the core arguments, processes, or conclusions factually and concisely.
4. Omit conversational filler, introductions, conclusions, calls to action, and formatting artifacts (tables, code blocks).
5. Output only the resulting clean, semantically rich text — no preamble, no closing remarks.

Key takeaways

1. Embeddings convert content into vectors that encode meaning, letting systems match on concept rather than keyword.
2. Vectorization is a pipeline: chunk, embed, store, retrieve.
3. The payoff is similarity search — finding conceptually related information instead of exact strings.
4. Embeddings are the retrieval half of RAG, grounding an LLM in factual, current, or private data.
5. Results live or die on chunking, model choice, and metadata discipline.