Most of us building AI applications spend the bulk of their time on the LLM - picking the right model, tuning the prompt, evaluating output quality. The embedding model gets a passing thought.
We use what’s easily available like OpenAI’s text-embedding-3-small. Then, three months into production, retrieval starts silently returning the wrong context, users complain the system “misses stuff,” and the team realizes the retrieval layer was never really designed at all.
Embedding models are the floor on which every RAG pipeline, semantic search system, and recommendation engine stands. A weak embedding layer poisons everything downstream. Getting embeddings right — the model, the chunking, the evaluation — is arguably the best leverage.
This article covers all of it: how embeddings work, how to pick and evaluate a model, how chunking interacts with retrieval quality, how to wire everything together in a production RAG pipeline, and what breaks when you ship to real users.
What Embeddings Actually Are
An embedding model takes a piece of text and produces a fixed-length vector of floating-point numbers. A 1536-dimensional vector from OpenAI’s text-embedding-3-small is just an array of 1536 floats. The insight is in the geometry of that space: texts that mean similar things end up close together, and texts that mean different things end up far apart, as measured by cosine similarity or dot product.
This is fundamentally different from keyword matching. A BM25 index counts token overlap. An embedding model encodes meaning. The query “how do I cancel my subscription” will retrieve “steps to terminate your account” even though they share no words, because both project to nearby regions of the vector space.
Modern embedding models are transformer encoders trained on large corpora with contrastive objectives. The training signal is roughly: given a query, bring its embedding closer to its positive document and push it away from negative documents. The result is a model that has learned to compress semantic meaning into a dense vector.
The Model Landscape in 2026
You have three tiers: proprietary APIs, managed specialist APIs, and open-source self-hosted models.
OpenAI: The Safe Default
OpenAI’s text-embedding-3-small and text-embedding-3-large are the easiest entry point. The small model produces 1536-dimensional vectors, costs 0.13 per million tokens and scores higher on MTEB benchmarks.
Both support Matryoshka Representation Learning (MRL). This means you can truncate the full vector to a smaller dimension (512, 256) and still get semantically useful embeddings, with a graceful quality degradation. This is a significant operational lever: truncating from 3072 to 512 dimensions reduces vector storage by 6x with modest quality loss, which can matter when you are indexing hundreds of millions of documents.
The honest assessment: OpenAI embeddings are reliable, well-documented, and the path of least resistance for prototypes and mid-scale applications under 15M tokens per month. They are not the best retrieval accuracy available. Switching later is expensive because every stored vector must be re-embedded.
Cohere Embed v4
Cohere’s embed-v4 occupies a different niche. It handles text and images in the same vector space, which eliminates OCR pipelines for PDF-heavy workloads. It supports a 128K token context window — dramatically longer than the 8K window on OpenAI models — which matters when you are embedding long technical documents without aggressive chunking.
Cohere also supports Matryoshka dimensions and offers tight integration with its rerank-v3 model, giving you an end-to-end retrieval stack from a single vendor with matching fine-tuning data contracts.
The tradeoff is cost. Cohere embed-v4 is the most expensive commercial option for pure-text, high-volume workloads. It makes the most sense when you are processing mixed-media documents, need multilingual coverage across many languages, or have governance requirements that benefit from Cohere’s enterprise data handling agreements.
Open Source: BGE-M3, Sentence Transformers, and the Gap Closing
The open-source embedding landscape has closed the quality gap with commercial APIs dramatically. Qwen3-Embedding-8B (Apache 2.0) scores 70.6 on MTEB, surpassing OpenAI’s text-embedding-3-large at 64.6 and Cohere embed-v4 at 65.2. BGE-M3 from BAAI supports 100+ languages, uses 568M parameters, and supports hybrid dense and sparse retrieval natively, which is unusual for an open model.
all-MiniLM-L6-v2 from the sentence-transformers library remains a reliable baseline for prototypes. It is small (80MB), runs on CPU in milliseconds, and is the right choice when you need to iterate fast without API costs. Do not ship it to production for a quality-sensitive use case.
The calculus for self-hosting looks like this: if you are processing more than 10M embeddings per month, have GPU infrastructure, or have data sovereignty requirements that prohibit sending text to an external API, open-source models are now the correct default. Quantized E5-large-v2 runs in 10ms on CPU, faster than any embedding API round-trip.
Here is a reference table:
| Model | MTEB Score | Dimensions | Cost / 1M tokens | Context | Best For |
|---|---|---|---|---|---|
| OpenAI text-3-small | ~62 | 1536 | $0.02 | 8K | Prototypes, low-volume apps |
| OpenAI text-3-large | 64.6 | 3072 | $0.13 | 8K | English RAG, easy integration |
| Cohere embed-v4 | 65.2 | 1024 | Higher | 128K | Multimodal, multilingual, enterprise |
| BGE-M3 | 63.0 | 1024 | Self-hosted | 8K | Multilingual, data sovereignty |
| Qwen3-Embedding-8B | 70.6 | Configurable | Self-hosted | 32K | High accuracy, GPU infra |
MTEB scores above are approximate and change with new releases. Always verify against the MTEB Leaderboard before making a final decision, and always benchmark on your own data.
Picking a Model Is Not The Hard Part
The choice of embedding model is less important than the decisions you make around it. In practice, chunking strategy has equal or greater impact on retrieval quality than model selection. Teams spend weeks evaluating OpenAI versus Cohere, then split documents every 512 tokens and wonder why recall is mediocre.
The reason is straightforward: your embedding model cannot fix a bad chunk. If a sentence is split mid-thought across two chunks, neither chunk will retrieve correctly for a query about that fact. If a table is flattened to text without its headers, the embedding of that text is semantically meaningless noise.
Before you finalize a model, you need a chunking strategy.
Chunking: The Hidden Variable
Chunking is the process of splitting source documents into the pieces you embed and store. The size and shape of those pieces directly determines what the retrieval system can surface.
The Core Tension
Small chunks produce precise embeddings that represent one idea clearly, which makes them easy to retrieve. But they lose surrounding context, so the LLM may not have enough information to answer from that chunk alone. Large chunks preserve context but produce averaged embeddings that represent many ideas at once, which makes them hard to retrieve with precision for any single idea.
The practical sweet spot is 256-512 tokens with 10-20% overlap. The overlap ensures that facts near chunk boundaries appear in at least one chunk with enough surrounding context. For most document types, recursive character splitting respects paragraph boundaries and performs well.
One important gotcha with LangChain: RecursiveCharacterTextSplitter counts characters by default, not tokens. A chunk_size=512 setting means 512 characters, which is roughly 128 tokens — far too small. Use the tiktoken encoder:
from langchain.text_splitter import RecursiveCharacterTextSplitter
splitter = RecursiveCharacterTextSplitter.from_tiktoken_encoder(
chunk_size=512, # now actual tokens
chunk_overlap=50,
)
chunks = splitter.split_documents(docs)Tables Are the Most Common Silent Failure
Tables are a two-dimensional structure in a one-dimensional representation. When you flatten a table to text naively, the row-column relationships disappear. A chunk containing "Product A EMEA 4.2M 12% Product B APAC 3.1M -3%" is semantically unintelligible without the column headers. The embedding model produces a vector for that text, but it is a vector of noise. Retrieval will fail silently: the embedding is stored, the chunk is indexed, but no meaningful query will surface it.
The fix is to serialize tables differently. Repeat the header on every row, or convert each table cell to a "column_name: value" pair. Better yet, consider a parent-child chunking strategy where the table itself is embedded at the parent level and individual rows at the child level.
Semantic Chunking
Semantic chunking uses embeddings to detect topic shifts and splits when the semantic distance between adjacent sentences exceeds a threshold. It outperforms fixed-size chunking on heterogeneous documents. The tradeoff is cost: it requires an embedding model call during ingestion rather than a simple token count, which adds latency and cost to the indexing pipeline. For documents with clear structural boundaries (code files, formal reports, legal contracts), structural chunking is usually sufficient and faster. Use semantic chunking when documents mix topics unpredictably, such as meeting transcripts or long-form research papers.
Building the RAG Pipeline
With a model chosen and chunking designed, here is how a production RAG pipeline fits together.
Indexing
from sentence_transformers import SentenceTransformer
import numpy as np
model = SentenceTransformer('BAAI/bge-m3')
def embed_chunks(chunks: list[str]) -> np.ndarray:
# Normalize for cosine similarity
embeddings = model.encode(
chunks,
batch_size=64,
normalize_embeddings=True,
show_progress_bar=True,
)
return embeddingsFor BGE models, prefix the query with "Represent this sentence for searching relevant passages: " at query time but not at indexing time. This asymmetry is intentional and documented — skipping it is a common mistake that degrades retrieval quality.
For OpenAI:
import openai
def embed_chunks_openai(chunks: list[str]) -> list[list[float]]:
response = openai.embeddings.create(
model="text-embedding-3-small",
input=chunks, # batch up to 2048 inputs
)
return [item.embedding for item in response.data]Always batch. Sending one chunk per API call introduces 50-100ms of network overhead per request. Batching 64-256 chunks per call reduces that overhead by orders of magnitude.
Hybrid Retrieval
Pure vector search misses documents with specific technical terms, product names, or codes that the model has not seen enough times to generalize over. Hybrid search combines dense vector retrieval with sparse BM25 retrieval and merges the results. The standard fusion technique is Reciprocal Rank Fusion (RRF).
def reciprocal_rank_fusion(
dense_results: list[str],
sparse_results: list[str],
k: int = 60,
) -> list[str]:
scores: dict[str, float] = {}
for rank, doc_id in enumerate(dense_results):
scores[doc_id] = scores.get(doc_id, 0) + 1 / (rank + k)
for rank, doc_id in enumerate(sparse_results):
scores[doc_id] = scores.get(doc_id, 0) + 1 / (rank + k)
return sorted(scores, key=lambda x: scores[x], reverse=True)Hybrid retrieval consistently improves recall, especially for domain-specific terminology. The improvement in context recall moving from dense-only to hybrid is often substantial and measurable with RAGAS in the first evaluation run.
Reranking
Once you have the top 50 candidates from hybrid retrieval, run them through a cross-encoder reranker. The reranker sees the query and each candidate together and produces a precise relevance score. It catches ranking errors that the bi-encoder makes because it sees the pair jointly rather than encoding each side independently.
from sentence_transformers import CrossEncoder
reranker = CrossEncoder('BAAI/bge-reranker-v2-m3')
def rerank(query: str, candidates: list[str], top_k: int = 5) -> list[str]:
pairs = [(query, doc) for doc in candidates]
scores = reranker.predict(pairs)
ranked = sorted(zip(candidates, scores), key=lambda x: x[1], reverse=True)
return [doc for doc, _ in ranked[:top_k]]A lightweight cross-encoder adds roughly 80-120ms to query latency when reranking 20 documents on CPU. That is a reasonable tradeoff for the precision improvement. Only rerank the top 20-50 candidates — the reranker cannot fix missing documents, it can only reorder what retrieval already found. If your recall is low, fix retrieval first before adding a reranker.
The Query-Document Asymmetry
This is one of the most important non-obvious facts about embedding models: queries and documents have different characteristics. A query is short, often incomplete, phrased as a question. A document chunk is longer, declarative, and uses domain-specific terminology. Models like Voyage AI and BGE-M3 are specifically designed to handle this asymmetry by training on query-document pairs rather than document-document pairs.
When the query and document style are very different (short user queries against long technical docs), this asymmetry matters more. One mitigation is HyDE: Hypothetical Document Embedding. You prompt the LLM to generate a hypothetical answer to the query, then embed the hypothetical answer and search with that embedding instead. Because the hypothetical answer looks more like a real document chunk, retrieval quality often improves on difficult queries.
def hypothetical_document_embed(query: str, llm_client) -> list[float]:
response = llm_client.messages.create(
model="claude-sonnet-4-20250514",
max_tokens=200,
messages=[{
"role": "user",
"content": f"Write a short passage that would answer: {query}"
}]
)
hypothetical_doc = response.content[0].text
return embed_chunks_openai([hypothetical_doc])[0]Evaluating Retrieval Quality
RAGAS is the standard framework for measuring RAG pipeline quality. It computes four metrics:
- Context Recall: what fraction of relevant information was retrieved
- Context Precision: what fraction of retrieved information was relevant
- Answer Faithfulness: does the generated answer stay grounded in the retrieved context
- Answer Relevancy: does the answer actually address the query
The key insight about RAGAS is what it cannot tell you. A Context Precision of 0.79 means 79% of what you passed to the LLM was relevant on average across your evaluation set. It does not mean 79% of users get correct answers. Use RAGAS as a before/after instrument every time you change the pipeline — when you added hybrid search, did Context Recall improve? When you added reranking, did Context Precision improve without destroying Recall? These comparisons are reliable. Absolute numbers are not directly comparable across different evaluation sets.
Build a golden set of 100-200 representative query/expected-answer pairs from your actual domain. Generic benchmarks like MTEB measure broad English retrieval quality. Your users have a specific domain, specific terminology, and specific query styles that generic benchmarks do not reflect. A model that scores 67 on MTEB may perform worse on your medical or legal corpus than a model scoring 63.
Production Gotchas
Model Stickiness
Every document in your vector index was embedded with a specific model. If you switch models, you must re-embed everything. There is no way to mix vectors from different models in the same index — cosine similarity between a Voyage vector and an OpenAI vector is meaningless because they live in different geometric spaces. This is the most underappreciated engineering risk in the embedding layer. Treat model selection with the seriousness of a database schema decision. Migrating is possible, but it is an offline batch job that can take hours to days for large corpora and zero-downtime migration requires running two indices in parallel during the transition.
Embedding Drift
Your documents change. If documents are updated, deleted, or added, the corresponding embeddings must be updated. Stale embeddings for updated documents are worse than no embedding at all: retrieval returns the old version, the LLM answers from outdated context, and the user sees a confident wrong answer. Build incremental re-indexing into your ingestion pipeline from day one. Track a last_embedded_at timestamp on every document record.
Context Window Mismatch
Most production embedding models have an 8K token context window (OpenAI, BGE-M3). If a chunk exceeds the model’s context window, the model silently truncates the input and embeds only the beginning of the chunk. You get no error — you get a vector that represents half your document. Keep assembled context under 8K tokens for most queries, and validate that no chunk exceeds the model’s maximum input length at indexing time.
Dimensionality and Storage
A 1024-dimensional float32 vector takes 4KB. At 10 million documents, that is 40GB of vector storage. Doubling dimensions doubles that cost. If you are using OpenAI’s text-embedding-3-large at 3072 dimensions, you are using 3x the storage of a 1024-dim model for marginal retrieval improvement on most domains. Use Matryoshka truncation to 1024 or 512 dimensions and measure the recall impact on your evaluation set before paying the full storage and latency cost of 3072 dimensions.
Do Not Trust Generic Benchmarks Alone
MTEB is a useful prior, not a decision. The benchmark covers a broad range of English retrieval tasks. If your application is multilingual, domain-specific (legal, medical, code), or has a non-standard query format (voice transcripts, structured forms), MTEB scores are a starting point. Benchmark every shortlisted model on a sample of your actual data before committing. The cost of this evaluation is a few hours of engineering time. The cost of discovering your model choice was wrong at production scale is re-indexing the entire corpus.
The Decision Matrix
Here is how to apply all of the above in a concrete decision:
You are building a prototype or MVP with an English corpus under 5M documents: use text-embedding-3-small. It costs less than $1 to index 50M tokens, integrates everywhere, and the model can be replaced later when the stakes are higher.
You are in production with quality-sensitive English retrieval: benchmark Voyage 4 against your golden set. The retrieval quality improvement is real and measurable, and the cost is lower than OpenAI’s large model.
You have multilingual content or mixed document types (PDFs with tables and images): Cohere embed-v4 is the only production multimodal embedding model and the strongest multilingual managed option.
You have data sovereignty requirements, volume above 10M embeddings per month, or GPU infrastructure: BGE-M3 or Qwen3-Embedding-8B are the correct defaults. The open-source quality gap versus commercial APIs has largely closed.
You are optimizing an existing pipeline with mediocre retrieval: before switching models, measure retrieval separately from generation quality. Add RAGAS to your evaluation pipeline. If Context Recall is below 0.7, your chunking or retrieval breadth is the bottleneck, not the model. If Context Precision is low, add a reranker before you change anything else. Only switch models if your evaluation set shows the current model is systematically failing on queries your users actually ask.
Where This Is Going
The embedding landscape is moving in two directions simultaneously. The first is multimodality: the ability to embed text, images, PDFs, and video into the same vector space, which eliminates the need for preprocessing pipelines that extract text from non-text documents. The second is long context: models with 32K and 128K context windows that can embed entire documents as a single unit, reducing the sensitivity of the pipeline to chunking decisions.
Both trends reduce the number of moving parts in a retrieval pipeline. But they do not eliminate the fundamental engineering discipline of measuring retrieval quality, understanding where it fails, and building systems that degrade gracefully when retrieval is imperfect.
Embedding models convert text into dense vectors whose geometric proximity encodes semantic meaning, forming the retrieval backbone of every RAG pipeline and AI search system. Model choice matters but is less decisive than engineers assume: chunking strategy, hybrid search with BM25, and cross-encoder reranking each have equal or greater impact on retrieval quality. For most English applications, start with OpenAI’s text-embedding-3-small. Move to Voyage AI when retrieval precision drives product quality. Use BGE-M3 or Qwen3-Embedding-8B when data sovereignty or volume makes self-hosting necessary. Treat model selection as a sticky decision, measure retrieval with RAGAS on your own golden set, and fix recall before tuning precision.