RAG in production 2026: the working reference

What RAG actually solves

RAG addresses three concrete problems:

• Stale or missing knowledge. The model's training data has a cutoff; RAG injects fresh or proprietary content at query time.
• Hallucinations on factual queries. When the model has evidence, hallucination rates drop near zero.
• Auditability. Citing source IDs in the answer lets downstream code (and humans) verify the model used the supplied evidence.

See the RAG glossary entry for the canonical definition and grounding for the broader pattern family.

Production RAG anatomy

A production RAG pipeline has five stages:

1. Ingestion — load documents, split into chunks, embed, store in a vector database.
2. Query understanding — rewrite the user question, expand with synonyms, classify intent.
3. Retrieval — hybrid (vector + BM25) search returning top-K candidates.
4. Re-ranking — a second model re-orders top-K by query-document relevance.
5. Generation — assemble the prompt with the top documents and generate a cited answer.

Optional but common in 2026: a verification pass where a separate model fact-checks the generated answer against the supplied documents and rejects unsupported claims.

Chunking strategy

Chunking is the highest-leverage knob in retrieval quality. Bad chunks make great embeddings look stupid.

• Chunk size: 200-500 tokens per chunk is the production sweet spot in 2026. Smaller fragments lose context; larger fragments dilute the embedding signal.
• Overlap: 10-20% overlap between adjacent chunks preserves continuity across boundaries.
• Semantic chunking: split on natural boundaries (paragraphs, sections, headings) instead of arbitrary character counts.
• Hierarchical chunks: store both small (sentence-level) and large (section-level) chunks; retrieve small, return large as context.
• Code, tables, and figures need bespoke chunking. Code by function. Tables by row group. Figures by caption block.

Picking an embedding model

The embedding model is the foundation; switch it later only at enormous cost (re-embed the corpus). Choose with care.

• OpenAI text-embedding-3-large: strong general-purpose default, 3072 dims, $0.13 / 1M tokens.
• Voyage AI voyage-3: tops MTEB benchmarks in 2026, available via API.
• Cohere embed-v3: strong multilingual, good ergonomics, 1024 dims.
• nomic-embed-text-v1.5 (open): excellent open-weight option, runs on CPU at modest scale.
• BGE-M3 (open): multilingual + hybrid (dense + sparse + multi-vector) in one model.

Evaluate on your own data before committing. Public benchmarks (MTEB) are signal, not truth — domain-specific recall matters more than overall rank. See the embeddings glossary entry.

Hybrid + re-rank retrieval

Pure vector search misses exact-match queries (product SKUs, error codes). Pure keyword search misses semantic intent. The default in 2026 is hybrid: dense vector + BM25 keyword, fused with reciprocal rank fusion (RRF).

1. Stage 1: retrieve broadly. Pull top 50-100 candidates from both vector and BM25 indexes.
2. Stage 2: fuse. RRF or weighted score combining both ranked lists.
3. Stage 3: re-rank. Pass top 20-30 through a cross-encoder (Cohere Rerank, Voyage Rerank, BGE-Reranker, or a small custom LLM) for query-document scoring.
4. Stage 4: select top K. Usually 3-8 documents go into the final prompt.

See semantic search and vector database.

Prompt design for grounded answers

The prompt is where you enforce grounding. Key patterns:

• Refuse if context is missing. "If the documents do not contain the answer, reply 'INSUFFICIENT.' Do not guess."
• Cite source IDs inline. "Cite each claim with [doc-id] referring to the supplied documents."
• Put the question last. Models attend best to the end of the prompt — put retrieved docs first, then the question.
• Format documents predictably. "Document <id>: <text>" with a clean separator.
• Don't paste raw HTML. Clean the retrieved text — boilerplate destroys answer quality.

Evaluating retrieval and generation

RAG evals decompose into two stages:

Retrieval evals

• Recall@K: for a labelled golden set of (query, ideal_doc_ids), what fraction of ideal docs appear in top-K?
• nDCG@K: rank-aware quality metric.
• Coverage: what fraction of queries have at least one relevant doc in top-K?

Generation evals

• Faithfulness: does the answer only state things supported by the retrieved docs?
• Answer relevance: does the answer address the user's actual question?
• Citation accuracy: are the cited document IDs correct?

Tools that help in 2026: Ragas, TruLens, Braintrust, Langfuse, Patronus. Even a simple LLM-as-judge over a golden set beats nothing.

Observability

You cannot debug what you cannot see. A production RAG system logs every retrieval + generation:

• Query (original + rewritten).
• Top-K candidates with scores from each retrieval stage.
• Re-rank scores.
• Documents actually injected into the prompt.
• Generated answer + cited doc IDs.
• User feedback (thumbs up/down, click-through on citations).

Tracing tools: Langfuse, Braintrust, Phoenix (Arize), LangSmith. The goal is to debug a bad answer by replaying every stage of the pipeline.

Scaling past 1M documents

Most RAG tutorials assume a corpus of a few thousand documents. Production corpora hit 1M-100M. The engineering changes:

• Index choice matters. HNSW for in-memory speed (Qdrant, Weaviate, pgvector). DiskANN or ScaNN for disk-resident huge indexes (Turbopuffer, Vespa).
• Filtering matters. Almost every query in 2026 has metadata filters (tenant, language, date range). Make sure your DB supports filtered ANN well.
• Sharding. Past ~50M docs, shard by tenant or domain. Aggregate top-K across shards.
• Cost ramp. Vector storage cost scales linearly with corpus size + dimensionality. Use Matryoshka embeddings (variable-dim) or quantisation to control.
• Freshness. Incremental ingestion with low-latency index updates is non-trivial at scale — design for it.

Antipatterns

• Pure vector search — misses exact-match terms.
• No re-ranker — first-stage retrieval is noisy; re-ranking is cheap quality.
• Chunks too large (over 1000 tokens) — dilutes the embedding signal.
• Chunks too small (under 100 tokens) — loses surrounding context.
• Letting the model invent citations — always inject documents with explicit IDs and require citing IDs from that set.
• Re-embedding the corpus on every prompt change — only the query side changes.
• Mixing embeddings from different models in the same index — distances aren't comparable.
• No evals — quality regressions go unnoticed.

FAQ