Embedding API Selection for Semantic Search and RAG (2025)

The embedding decision is more expensive than it looks

Most teams building a RAG pipeline treat the embedding model as an afterthought — grab the first one that works and move on. That's a mistake you'll feel later in your AWS bill and in recall metrics that never quite get there.

Three things make this choice matter more than it appears:

• Storage costs scale with vector dimensions. A 3072-dimension model costs roughly 3x more to store per document than a 1024-dimension one.
• Language coverage directly affects recall quality. Models not specifically trained on your target language will miss semantically close matches that a native-optimized model catches.
• API pricing varies by over 10x across providers at high throughput.

To put numbers on it: a system handling 1 million daily queries against a 5 million-chunk document corpus can run anywhere from a few hundred to tens of thousands of dollars per month — at the embedding layer alone. This decision deserves deliberate attention.

---

Model comparison: what actually matters

Current embedding models fall into three camps: OpenAI, Cohere, and open-source/self-hosted. Here's how the main options stack up on the dimensions that matter in production:

Model	Dimensions	Max tokens	Price (per 1M tokens)	Multilingual	Best for
text-embedding-3-small	1536	8191	~$0.02	Fair	Cost-sensitive, English-primary
text-embedding-3-large	3072	8191	~$0.13	Good	High accuracy, mixed language
Cohere embed-v4	1024	512	~$0.10	Excellent	Enterprise search, multilingual
BGE-M3 (open source)	1024	8192	Self-hosted	Excellent	Cost control, non-English docs
Jina embeddings-v3	768–1024	8192	~$0.018	Good	Long documents, cost efficiency

Two details that get overlooked:

text-embedding-3-large supports Matryoshka Representation Learning — you can truncate vectors to a lower dimension (say, 256) without a significant accuracy penalty. If storage pressure is real, this is worth using:

from openai import OpenAI

client = OpenAI()
response = client.embeddings.create(
    input="Your document text here",
    model="text-embedding-3-large",
    dimensions=256  # truncate to 256 dimensions
)

BGE-M3 consistently leads on non-English MTEB benchmarks, including Chinese. If your documents are primarily non-English and you have GPU capacity, self-hosting BGE-M3 is the highest-value path.

---

Semantic search vs. RAG: different bottlenecks

Both use embeddings, but where they break down is different.

Semantic search lives and dies by top-k recall and latency. A user submits a query; your system needs the most relevant results back in milliseconds. Here, semantic alignment matters more than raw dimensionality. A 768-dimension model with strong language understanding will typically outperform a 3072-dimension model with weak semantic alignment on real-world recall benchmarks.

Practical guidance: - English-only: text-embedding-3-small is sufficient and cheap - Mixed or non-English: BGE-M3 self-hosted, or Cohere embed-v4 via API - Latency-sensitive (<50ms): prefer lower-dimension models — retrieval time scales with vector size

RAG has a more complex failure surface. Embedding quality matters, but chunk strategy often has a bigger impact on final answer quality than model choice. The common mistake is setting chunk size too small (128 tokens or fewer) under the assumption that finer granularity means better retrieval. In practice, chunks that short strip out the context an LLM needs to generate a useful answer.

What works in practice: 256–512 tokens per chunk, 10–15% overlap. Pair with text-embedding-3-large when output quality is the priority, or BGE-M3 when cost is the constraint.

---

Three dimensions to guide your embedding API selection

1. Language distribution

Audit your document corpus before picking a model. If more than 30% of your content is non-English, text-embedding-3-small will leave recall on the table — its semantic alignment outside English is noticeably weaker than purpose-built multilingual models. A quick sanity check: take 50 representative queries, run retrieval with two candidate models, and manually evaluate the top-5 results. The gap is usually obvious.

2. Throughput and cost budget

Quick back-of-napkin math:

Monthly cost ≈ (daily document updates + daily queries) × average token count × unit price × 30

At 100,000 daily queries, ~50 tokens each: - text-embedding-3-small: 100,000 × 50 × $0.00000002 × 30 ≈ $3/month - text-embedding-3-large: same conditions ≈ $19.50/month

Document indexing is a one-time cost; queries are recurring. Model those separately or you'll underestimate the ongoing spend.

3. Vector database lock-in

Switching embedding models means rebuilding your vector index from scratch. Moving from text-embedding-3-small (1536 dimensions) to BGE-M3 (1024 dimensions) requires reindexing your entire corpus. That migration cost is real — factor it into the initial selection, not after you've already indexed millions of documents.

---

A minimal working example

Here's a complete semantic search flow in Python using OpenAI embeddings and Qdrant:

from openai import OpenAI
from qdrant_client import QdrantClient
from qdrant_client.models import Distance, VectorParams, PointStruct

client = OpenAI()
qdrant = QdrantClient(":memory:")

# Collection dimensions must match your model
qdrant.create_collection(
    collection_name="docs",
    vectors_config=VectorParams(size=1536, distance=Distance.COSINE),
)

def embed(text: str) -> list[float]:
    resp = client.embeddings.create(
        model="text-embedding-3-small",
        input=text
    )
    return resp.data[0].embedding

# Index documents
docs = [
    "Semantic search is built on vector similarity",
    "RAG combines retrieval with generation"
]
points = [
    PointStruct(id=i, vector=embed(doc), payload={"text": doc})
    for i, doc in enumerate(docs)
]
qdrant.upsert(collection_name="docs", points=points)

# Query
query_vec = embed("how does vector retrieval work")
results = qdrant.search(
    collection_name="docs",
    query_vector=query_vec,
    limit=2
)
for r in results:
    print(r.score, r.payload["text"])

Swapping models means changing model and size — nothing else in the architecture changes.

---

What I'd actually do

If you're running A/B tests across multiple embedding providers — comparing OpenAI, BGE, and Cohere on the same dataset — managing separate API keys and format differences gets tedious fast. I use XycAi for exactly this: one OpenAI-compatible endpoint covering 200+ models, with GPT and Claude available from 14% of list price. Switching between models is a single base_url change, which makes iterative embedding API selection dramatically less painful.