RAG (Retrieval-Augmented Generation) has become the go-to pattern for grounding LLM responses in real data without the cost of full fine-tuning. But most tutorials stop at the "happy path" — a single PDF, a localhost Chroma instance, and a GPT-4 call. Production is a different story.
Why RAG Fails in Production
The failures I've seen repeatedly fall into three categories: retrieval quality, context window mismanagement, and latency. If your retrieval returns the wrong chunks — even slightly — the LLM hallucinates confidently. This is worse than no system at all.
- Retrieval quality degrades with poor chunking strategies (fixed-size vs. semantic splits)
- Context windows fill up fast with large documents, cutting off critical information
- Cold-start latency from embedding and vector search can destroy UX at scale
- Embedding model drift when you switch providers breaks semantic similarity scores
Chunking Strategy Matters More Than Your Model
Fixed-size chunking (e.g., 512 tokens with 50-token overlap) is the default but rarely optimal. Semantic chunking — splitting at paragraph and section boundaries — dramatically improves retrieval precision. I've seen recall scores jump from ~58% to ~84% just by switching chunking strategies.
from langchain.text_splitter import RecursiveCharacterTextSplitter
splitter = RecursiveCharacterTextSplitter(
chunk_size=800,
chunk_overlap=100,
separators=["\n\n", "\n", ".", "!", "?", ",", " ", ""],
)
chunks = splitter.split_documents(docs)Choosing a Vector Store for Scale
For prototypes: Chroma or FAISS. For production: Qdrant, Weaviate, or Pinecone. The differentiators are filtered search (metadata pre-filtering before ANN), hybrid search (dense + sparse BM25), and managed hosting with SLA guarantees.
- Qdrant: best for self-hosted, payload filtering is excellent
- Pinecone: best for fully managed with zero ops overhead
- Weaviate: best when you need hybrid search out of the box
- pgvector: surprisingly viable for <1M vectors if you're already on Postgres
Latency Optimization
A naive RAG call is: embed query → vector search → LLM call. Each step adds latency. Strategies that work in production: cache embeddings for repeated queries, use async retrieval with streaming LLM responses, and implement a re-ranking step only when precision matters more than speed.
The goal is not the smartest RAG system — it's the fastest one that users trust. Accuracy at 2 seconds beats perfection at 10 seconds.
Evaluation: The Step Everyone Skips
You cannot improve what you don't measure. Set up automated evaluation using RAGAS (Retrieval-Augmented Generation Assessment) to track faithfulness, answer relevance, and context recall. Run this on every deployment and gate releases on a minimum score threshold.
