RAG for internal documents: what actually works in production

Summary: Production RAG (retrieval-augmented generation) needs structured chunking, hybrid search (BM25 + vectors), citations, evaluation, access control, and a plan to keep the embedding index fresh.

Retrieval-Augmented Generation (RAG) is the part of "AI for your documents" that does the heavy lifting. It is also the part that, in most demos, is held together with hopes and prompt-engineering.

The retrieval layer: what actually works

Chunking strategy

Forget recursive splitters by character count. Use the document's structure: headings, paragraphs, list items. Keep table rows whole. Carry a path of headings as metadata on every chunk.

Hybrid search (BM25 + vectors)

Vector search alone misses too many keyword-heavy queries (acronyms, IDs, codes). Combine BM25 with vectors and rerank the top 50.

Source citations

Always render the source span the model used. Users trust an assistant that points back at the document; they distrust one that just talks.

Operations, security, and freshness

Evaluation before you ship

Production RAG fails quietly: answers look fluent while retrieval is wrong. Before launch, we maintain a small set of golden questions with expected sources — regression-tested on every deploy. We also log retrieval hits and user thumbs-up/down so we can see which document types and queries drift over time.

Access control and tenancy

Internal docs are not one big bucket. Every chunk carries tenant, team, and sensitivity metadata; retrieval filters before the LLM sees context. If your org chart and folder permissions are the source of truth, sync those rules into the index — otherwise you will eventually retrieve something a user should not see.

Keeping the index fresh

Stale RAG is worse than no RAG: it confidently cites last quarter's policy. We wire webhooks or scheduled jobs so that when a Confluence page, PDF, or contract changes, affected chunks are re-embedded or invalidated. For large corpora, incremental updates beat full re-indexing every night.