RAG Implementation Tutorial: Chunking to Production in 4 Steps

Most people building their first RAG system hit the same wall: the pipeline runs, but the results are bad. Retrieved chunks miss the point, the model answers the wrong question, or everything falls apart under production load. This tutorial breaks a production-ready RAG pipeline into four concrete stages — with specific parameters and the reasoning behind each tradeoff. The goal is a system you can actually ship, not another demo.

Chunking: Get This Wrong and Nothing Else Matters

Document chunking is the most underestimated step in the entire pipeline. Chunks that are too large produce blurry embeddings. Chunks that are too small lose context, and the retrieved fragments don't give the model enough to work with.

Fixed-size chunking works well for long, loosely structured text. A chunk size of 512 tokens with 64–128 tokens of overlap is a solid starting point. The overlap prevents key information from getting cut exactly at a boundary between two chunks.

from langchain_text_splitters import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=512,
    chunk_overlap=128,
    length_function=len,
)
chunks = splitter.split_text(raw_text)

Semantic chunking is a better fit for technical documentation with clear heading hierarchies — split on structural boundaries so a single concept stays in one chunk. The tradeoff is implementation complexity. It pays off when your document quality is consistent and high.

The practical advice: start with fixed-size chunking to get a baseline, measure retrieval quality, and only add semantic chunking if the numbers justify it. Don't over-engineer before you have data.

---

Embeddings: Model Selection and Deployment

Embedding maps each chunk into a high-dimensional vector space. At query time, you compute cosine similarity between the query vector and every chunk vector.

Model	Dimensions	Best for	Access
text-embedding-3-large	3072	High-accuracy English / multilingual	OpenAI API
text-embedding-3-small	1536	Cost-sensitive workloads	OpenAI API
BAAI/bge-m3	1024	Strong on CJK, supports offline deployment	HuggingFace / self-hosted

For any corpus with significant non-English content, bge-m3 is usually more consistent than OpenAI's general-purpose embeddings, and self-hosting it sidesteps data residency concerns entirely.

For vector storage, Qdrant is the production recommendation — it supports payload filtering, persistence, and horizontal scaling. Weaviate is a reasonable alternative. For a quick prototype where you don't need real-time updates, FAISS with an in-memory index is fine.

from qdrant_client import QdrantClient
from qdrant_client.models import Distance, VectorParams

client = QdrantClient(url="http://localhost:6333")
client.create_collection(
    collection_name="docs",
    vectors_config=VectorParams(size=1024, distance=Distance.COSINE),
)

---

Retrieval + Reranking: Two Stages, Meaningfully Better Results

Vector search alone — pulling the top-k nearest neighbors — often isn't precise enough. Embedding space compresses semantics lossily, and for short queries or domain-specific terminology, the chunks ranked highest by cosine similarity aren't always the most relevant ones.

The standard fix is a two-stage pipeline:

1. Coarse retrieval: vector search returns the top 20–50 candidate chunks
2. Reranking: a Cross-Encoder model scores each (query, chunk) pair directly and returns the top 3–5 for the prompt

from sentence_transformers import CrossEncoder

reranker = CrossEncoder("BAAI/bge-reranker-v2-m3")
pairs = [(query, chunk) for chunk in candidates]
scores = reranker.predict(pairs)

ranked = sorted(zip(scores, candidates), reverse=True)[:5]
final_chunks = [chunk for _, chunk in ranked]

bge-reranker-v2-m3 is reliable and fast enough for production: roughly 50–100ms on GPU for a batch of 20 pairs, 300–500ms on CPU. If reranking becomes a latency bottleneck, reduce the candidate pool or pre-compute scores asynchronously.

Hybrid search is worth adding once the basics are stable: fuse vector search results with BM25 keyword search using Reciprocal Rank Fusion (RRF). Keyword search handles exact matches — product codes, names, code snippets — that embedding models sometimes miss. The two approaches are genuinely complementary.

---

Context Injection and Prompt Engineering

After retrieval, you're assembling the final chunks into a prompt. The details here have a real impact on generation quality.

A few things that actually matter:

• Position the most relevant chunk last, immediately before the question. Attention is not uniform across a long context — models tend to weight content near the end more heavily.
• Label each chunk with its source: [Source: filename, Section X]. This lets the model cite references in its answer and makes it easier for users (and you) to trace the output back to the original document.
• Explicitly tell the model to refuse: instruct it to say "I couldn't find relevant information in the provided documents" rather than speculating. Without this, you'll get hallucinations dressed up as confident answers.

system_prompt = """You are a document Q&A assistant.
Answer questions only using the documents provided below.
If the documents don't contain enough information to answer, say so directly — do not guess."""

context = "\n\n".join([
    f"[Source: {meta['source']}]\n{chunk}"
    for chunk, meta in final_chunks
])

user_prompt = f"Documents:\n{context}\n\nQuestion: {query}"

For the generation model, both GPT-4o and Claude Sonnet handle long context and instruction-following reliably in production. Set temperature to 0–0.2 for consistent, reproducible answers.

---

Pre-Launch Production Checklist

Once the pipeline works end-to-end, there are a few things to verify before going live.

Latency budget: target P95 end-to-end latency under 3 seconds. Vector search is typically under 50ms, reranking adds 100–500ms, and LLM generation is the dominant cost at 1–2 seconds. Profile before optimizing.

Index update strategy: use incremental upserts rather than full rebuilds when documents change. Qdrant supports per-document-ID vector updates. Pair that with document hash-based change detection and you can keep the index in sync within minutes of a document update.

Offline evaluation: before launch, run a round of evaluation against a labeled test set. The two metrics that matter most are Recall@5 (does the correct chunk appear in the top 5?) and Answer Faithfulness (does the generated answer stay grounded in the retrieved context?). RAGAS provides a straightforward framework for both.

Production monitoring: log retrieval result IDs and rerank scores for every query. When retrieval fails — and it will — you need this data to diagnose whether the problem is in chunking, embedding, or the reranker.

---

Running this pipeline at scale, the two cost centers that grow fastest are embedding writes (especially for large document corpora) and LLM generation. I use XycAi to access GPT and Claude models — pricing starts at 14% of OpenAI list price, the API is fully OpenAI-compatible, and switching is a one-line base_url change. For a multi-tenant RAG service processing thousands of documents a day, that cost difference adds up fast.