RAG in production: retrieval quality beyond embeddings

About this session

Most RAG implementations get the basics right — they embed documents, store vectors, and retrieve by cosine similarity — and then plateau at a quality level that’s useful but not reliable enough for production. Users learn to double-check AI answers, which defeats the purpose. The gap between “good demo” and “production-reliable” RAG is almost never about the embedding model. It’s about everything that happens before and after embedding.

This session goes deep on the factors that actually determine retrieval quality: document chunking strategy, hybrid retrieval (vector + BM25 keyword search), re-ranking with a cross-encoder, and how to measure retrieval precision independently of generation quality. All examples use ManyLayers Workspace’s knowledge base configuration directly.

What you will learn

• Why chunking strategy has more impact on retrieval quality than embedding model choice — and how to select chunk size and overlap for different content types
• How hybrid retrieval combines semantic vector search and BM25 keyword search to catch what pure vector search misses (technical terms, model names, version numbers, and other low-frequency tokens that embeddings don’t encode well)
• Cross-encoder re-ranking: what it is, when it’s worth the latency cost, and how to configure it in ManyLayers Workspace
• Connector-level filtering: how metadata filters on retrieved chunks reduce hallucination by scoping retrieval to relevant document subsets
• Building a retrieval eval pipeline in ManyLayers Workspace: how to measure recall@k and precision@k independently of the generation model
• Common failure patterns in production RAG systems and how to diagnose them from retrieval logs

Session agenda

0:00 – 12:00 — RAG quality decomposed: a framework for separating retrieval quality from generation quality. Why fixing the generator doesn’t fix retrieval failures, and how to isolate which component is causing user-facing errors.

12:00 – 24:00 — Chunking strategy in depth: live comparison of 256-token, 512-token, and 1,024-token chunks on a real 200-page policy document. Precision and recall curves at each chunk size. Structured chunking for FAQ and table content. The parent-document chunking pattern for long-context retrieval.

24:00 – 36:00 — Hybrid retrieval and re-ranking: configuring BM25 alongside vector search in ManyLayers Workspace. Tuning the alpha parameter that weights the two scores. Adding a cross-encoder re-ranker to the retrieval pipeline and measuring its impact on precision@3. When re-ranking is worth the added latency (typically 80–150ms for a 20-document re-rank pass).

36:00 – 48:00 — Eval pipeline setup: creating a retrieval eval dataset in Workspace, running recall@3 and precision@3 evaluations, and reading the results dashboard. How to set up continuous eval that runs against new connector syncs to catch retrieval regressions when source content changes.

48:00 – 55:00 — Production failure patterns: the five most common RAG failures we see in deployed systems, how to identify them from retrieval logs, and the configuration change that fixes each one.