Building Production-Ready RAG Systems: Lessons from 10+ Deployments

Most RAG systems work beautifully in demos. They fall apart in production.

After building RAG systems across legal, semiconductor, manufacturing, and enterprise domains, we've accumulated a set of hard-won lessons that don't make it into the LangChain tutorials. This post covers the ones that matter most.

The Core Problem: Demo-to-Production Gap

A typical RAG demo uses:

• A handful of clean PDF documents
• One embedding model
• One vector store
• A single retrieval strategy

Production looks like:

• Thousands of documents with inconsistent formatting
• Scanned PDFs, spreadsheets, HTML exports, email threads
• Multiple document collections with different access permissions
• Users asking questions that require reasoning across 5+ retrieved chunks
• Latency requirements under 2 seconds

The gap is enormous. Here's how we bridge it.

Lesson 1: Chunking is 80% of your retrieval quality

The most common mistake is using default chunking. Splitting documents at 512 or 1024 tokens without considering structure destroys context.

What works:

• Semantic chunking — split at natural boundaries (paragraphs, sections) rather than token counts
• Hierarchical chunking — store both small precise chunks and larger context windows; retrieve small, expand to large
• Document-aware chunking — tables chunk differently than prose, which chunks differently than code

For legal documents, we chunk by clause. For technical manuals, by section. For emails, by message thread. One-size-fits-all fails.

Lesson 2: Hybrid retrieval outperforms pure vector search

Pure semantic search misses exact keyword matches. BM25 alone misses semantic similarity. The winner is hybrid:

final_score = α * bm25_score + (1 - α) * vector_score

Typical α values: 0.3–0.5. The right value depends on your domain — keyword-heavy technical docs lean toward BM25; conceptual questions lean toward vectors.

We also add a re-ranking step using a cross-encoder model after the initial retrieval. This adds ~200ms but measurably improves answer quality.

Lesson 3: The retrieval ceiling problem

If your retrieval doesn't surface the right chunks, no amount of prompt engineering will save the answer. Always evaluate retrieval independently from generation.

Our evaluation loop:

1. Create a golden dataset of 100+ question/answer pairs with known source documents
2. Run retrieval, check whether correct chunks appear in top-5
3. Measure Hit@1, Hit@3, Hit@5, MRR
4. Only move to generation evaluation after retrieval metrics are acceptable

A Hit@3 below 0.7 means your chunking or embedding needs work — not your prompt.

Lesson 4: Access control is not optional

Enterprise RAG systems contain sensitive data. Every user should only retrieve chunks from documents they're authorized to see.

Common approaches:

• Filter by metadata at query time — pass user's permission set as filter alongside the embedding query
• Separate indices per permission group — simpler but expensive to maintain
• Row-level security in the vector store — Pinecone, Weaviate, and Qdrant all support namespace/filter combinations

We've seen companies skip this and create serious data leakage bugs. Build it from day one.

Lesson 5: Latency compounds across the pipeline

The full RAG pipeline involves:

1. Query embedding (~50ms)
2. Vector retrieval (~50-200ms depending on index size)
3. Re-ranking (~200ms)
4. LLM generation (~500-2000ms)

Total: 800ms–2.5 seconds before streaming. For a chatbot, this feels slow.

Optimizations:

• Cache embeddings for repeated queries
• Stream generation output immediately (don't wait for full response)
• Use a smaller re-ranker model if latency is critical
• Consider hybrid sync/async: return a partial answer quickly, then refine

Lesson 6: Hallucination detection at the chunk level

Even when retrieval is correct, LLMs can generate claims that aren't grounded in the retrieved context. We add a grounding check:

After generating an answer, extract the key claims and verify each claim appears (semantically) in the retrieved chunks. Flag answers with a low grounding score rather than surfacing them as confident.

Users trust a system that says "I'm not sure" more than one that confidently hallucinates.

What Production RAG Looks Like

A production-grade RAG system we build typically includes:

• Document ingestion pipeline with format-specific parsers
• Semantic + hierarchical chunking per document type
• Hybrid BM25 + vector retrieval with re-ranking
• Access control at the retrieval layer
• Answer grounding verification
• Streaming response with source attribution
• Evaluation suite (retrieval metrics + answer quality)
• Monitoring dashboard for retrieval quality drift over time

It's significantly more than a LangChain quickstart. But it's also what clients actually need.

---

If you're building a RAG system and want to talk through the architecture, reach out — this is one of our most common project types.