The embedding model is the most consequential and most reversible-at-great-cost decision in a RAG stack. Consequential because it determines the quality of every similarity comparison your system will ever make. Expensive to reverse because changing it means re-embedding your entire corpus — the vectors from model A are meaningless to model B. So it's worth choosing deliberately rather than defaulting to whatever the first tutorial used. Here's the framework we'd apply, across the dimensions that actually differ.
The axes that matter
There's no "best" embedding model — there are trade-offs, and the right pick depends on which ones bind for you.
Retrieval quality. The headline: how well do the embeddings actually separate relevant from irrelevant for your content? Public benchmarks (like MTEB-style leaderboards) are a starting point, but they measure average performance across general tasks. Your corpus is not the average. A model that tops a leaderboard on web text may underperform on your legal contracts or code. Benchmarks narrow the field; they don't pick the winner.
Dimensions. Embedding size (e.g. 384, 768, 1024, 1536, 3072) trades quality against storage and speed. More dimensions can capture more nuance but cost more to store and compare — and the storage cost is per-vector across your whole corpus, so it compounds. Higher isn't automatically better; many tasks saturate well before the largest dimension. This choice ripples into your vector store (Kognita, for instance, routes vectors to dimension-specific tables), so it's not a knob you flip casually later.
Latency. Embeddings happen twice: in bulk at ingestion, and once per query at search time. Query-time latency is user-facing — a slow embedding model adds delay to every search. Ingestion latency matters for throughput when you're indexing large or frequently-changing corpora. Hosted APIs add network round-trips; self-hosted models trade that for infra you operate.
Cost. Priced per token (hosted) or per GPU-hour (self-hosted). At small scale it's negligible; at large scale, embedding cost — especially if you re-embed often for freshness — becomes a real line item. Factor in re-embedding cost, not just initial ingestion, because a fresh knowledge base re-embeds changed content continuously.
Hosted vs. self-hosted
A structural fork worth deciding early:
- Hosted API (managed embedding endpoints, including providers like DeepInfra that Kognita uses): no infra to run, easy to start, scales elastically, priced per token. You depend on the provider's availability and pricing, and your data transits their service.
- Self-hosted open model: full control, data stays in your environment, fixed infra cost that can be cheaper at high volume — but you operate the serving, GPUs, and scaling. Worth it at scale or under strict data-residency requirements.
For most teams, a hosted endpoint is the right default: the operational simplicity outweighs the per-token cost until you're at serious volume or have hard data-locality constraints.
Don't ignore domain fit
The most common mistake is choosing on general benchmarks and ignoring your domain. General-purpose embeddings are trained on broad text and can flatten distinctions that matter in specialized corpora — medical terminology, legal language, code, dense technical jargon. Two passages a domain expert considers opposite can look similar to a general model. If your content is specialized, test candidate models on your data with your queries before committing. A model that's mid-tier on public benchmarks but strong on your domain will beat a leaderboard-topper that treats your jargon as noise.
A practical selection process
- Shortlist on benchmarks + language/domain coverage. Use public leaderboards to eliminate weak options and ensure the model covers your languages. This gets you to a handful of candidates.
- Build a small evaluation set from your own corpus — representative queries with known-relevant passages. This is the step teams skip and the one that actually decides quality.
- Measure retrieval quality on your data (recall/precision against your golden passages), not just leaderboard scores. Let your content break the tie.
- Factor dimensions, latency, and cost for the candidates that pass the quality bar. Pick the smallest-dimension, cheapest, fastest model that meets your quality threshold — not the biggest model overall.
- Confirm the switch cost is acceptable. Remember you're committing your whole corpus to this model's vector space. Choose like you'll live with it.
The bottom line
Choose the embedding model like a foundation, because that's what it is — everything above it inherits its quality, and replacing it means rebuilding. Shortlist with benchmarks, but decide with an evaluation on your own content. Pick the smallest, cheapest, fastest model that clears your quality bar on your data, prefer a hosted endpoint until scale or compliance says otherwise, and treat domain fit as a first-class criterion rather than an afterthought. Get this right and the rest of the pipeline has a solid base to build on; get it wrong and you'll be re-embedding your corpus in six months to fix it.