Why Your RAG Demo Falls Apart on Real Production Data

The demo was perfect. Twenty documents, a handful of questions, and the answers came back crisp and cited. So you loaded the real corpus, a few thousand documents, and shipped it. Now the same assistant confidently answers questions using a chunk from the wrong contract, the wrong month, or the wrong customer. Nothing errored. The retrieval "worked." It just returned the wrong thing, dressed up as the right thing.

This is the most common way a RAG system fails in production, and it is badly served by the existing writing on the subject, which tends to hand you a menu of techniques (hybrid search, reranking, better chunking) without telling you the uncomfortable part: you cannot know whether any of them helped, because you have no way to measure retrieval quality. That is the real problem, and I am going to be honest that a chunk of it is genuinely unsolved.

Why retrieval gets worse as you add documents

Vector search does not find "relevant" chunks. It finds near chunks, where near means small cosine distance between the query embedding and the chunk embedding. In a small corpus, near and relevant are almost the same thing, because there simply are not many chunks to be confused with. The correct answer is the only thing in the neighbourhood.

As the corpus grows, the neighbourhood fills up. Every new document adds chunks, and some of them land close to queries they have nothing to do with, because they share vocabulary, tone, or structure. A question about "the refund policy for the March invoice" is now close, in embedding space, to April's invoice, to every other refund clause, and to a boilerplate paragraph that appears in fifty documents. Top-k still returns the k nearest chunks. It is just that k nearest is no longer k relevant.

The precision loss is structural, not a bug in your embedding model. More neighbours means a lower base rate of relevance among the near ones. And because the language model downstream will fluently synthesise an answer from whatever chunks it is handed, the failure is invisible unless you look at the retrieved chunks themselves. The confident-but-wrong output is not the model hallucinating from nothing; it is the model faithfully summarising the wrong source.

The mitigations, and their real limits

Metadata pre-filtering: shrink the haystack first

The single most effective move, and the most underused, is to not search the whole corpus. If a query is about one customer, one month, or one document type, filter the vector search to that subset before computing similarity. You are not making similarity smarter; you are removing the confusable chunks entirely so they cannot be returned. Most vector stores support a metadata filter alongside the query:

# pseudocode: the exact call differs per store, the idea does not
results = store.similarity_search(
    query="refund policy",
    k=5,
    filter={"customer_id": "C-1042", "doc_type": "contract"},
)

The limit: this only works when you can derive the filter from the query, which often means an extra step (an LLM or a rule) that extracts the customer or date first, and that step can be wrong. But when it applies, it beats every fancier technique, because a chunk that is filtered out cannot be a false positive.

Hybrid search: put keywords back in

Pure vector search throws away exact-match signal. "Invoice INV-2024-0312" embeds to roughly the same place as any invoice reference, so vector search cannot distinguish the exact ID. Classic keyword search (BM25) nails exact tokens and rare terms but misses paraphrase. Hybrid search runs both and fuses the rankings, commonly with Reciprocal Rank Fusion, so exact identifiers and specific jargon get their weight back while semantic matches still surface.

The limit: hybrid search helps most when queries contain distinctive tokens (IDs, names, codes). For purely conceptual questions it adds little, and the fusion weighting is one more thing you now have to tune without a way to measure it.

Reranking: retrieve wide, then judge

A reranker (a cross-encoder that scores query and chunk together, rather than comparing two independent embeddings) is far more accurate at judging relevance than cosine distance, but too slow to run over the whole corpus. The pattern is: retrieve a wide candidate set with cheap vector search (say the top 50), then rerank those down to the top 5 that actually go to the model. Because the reranker sees the query and chunk jointly, it catches the "shares vocabulary but is about the wrong thing" cases that fool cosine similarity.

The limit: reranking improves ordering within the candidate set, but it cannot rescue a relevant chunk that vector search never retrieved into the candidate set in the first place. Widen the initial k and you pay latency; narrow it and the reranker has nothing good to promote.

Smaller, domain-specific indexes

Sometimes the honest answer is that one giant index is the problem. Splitting into per-domain or per-tenant indexes is metadata pre-filtering taken to its structural conclusion, and it keeps each neighbourhood sparse. If you are building this over business data, the companion piece on RAG architecture over ERP data that survives production goes deeper on partitioning by tenant and document type.

What people get wrong: tuning blind

The widespread failure is not choosing the wrong technique. It is applying techniques with no measurement, then declaring victory because a handful of manual spot-checks looked better. This is how teams "improve" a system into being worse: you add a reranker, three cherry-picked queries look great, you ship, and precision quietly drops on the 90 percent of queries you did not check.

You cannot eyeball retrieval quality at scale, and you cannot trust the fluent final answer to reveal a retrieval problem, because the model will smooth over bad context. The only way out is a labelled set. There is no clever prompt, no self-grading trick, and no off-the-shelf metric that reliably substitutes for knowing, for a set of real queries, which chunks are actually relevant. Anyone selling you a fully label-free retrieval quality score is selling you a proxy, and you should treat its number as a hint, not a grade.

The fix: a minimal retrieval eval harness

Step 1: label 50 queries by hand

Collect 50 real queries (from logs if you have them, invented if you do not, but make them realistic). For each, run your current retrieval, read the returned chunks, and mark which document IDs are genuinely relevant. This is tedious and it is the entire point: a day of labelling buys you the ability to measure everything else. Store it as plain data:

labeled = [
    {"query": "refund policy for March invoice",
     "relevant_ids": ["doc_314#c2", "doc_314#c3"]},
    {"query": "who signed the ACME contract",
     "relevant_ids": ["doc_88#c0"]},
    # ... 48 more
]

Step 2: compute precision@k and recall@k

Run each query through your retriever and compare the returned IDs against the labelled relevant set. Precision@k is "of the k I returned, how many were relevant." Recall@k is "of the relevant ones that exist, how many did I return in the top k."

def evaluate(retriever, labeled, k=5):
    p_sum = r_sum = 0.0
    for row in labeled:
        got = [c.id for c in retriever(row["query"], k=k)]
        relevant = set(row["relevant_ids"])
        hits = sum(1 for g in got if g in relevant)
        p_sum += hits / k
        r_sum += hits / max(len(relevant), 1)
    n = len(labeled)
    return {"precision@%d" % k: p_sum / n,
            "recall@%d" % k: r_sum / n}

print(evaluate(current_retriever, labeled, k=5))

Step 3: change one thing, re-run, compare

Now every change is a hypothesis you can test. Add the reranker, re-run evaluate, and look at the numbers. If precision@5 went from 0.42 to 0.68, keep it. If it went down, you just avoided shipping a regression that spot-checks would have hidden. Change one variable at a time so you know what moved the number.

Technique	Best against	Hard limit
Metadata pre-filter	Queries scoped to a tenant, date or type	Needs a reliable way to derive the filter
Hybrid (BM25 + vector)	Exact IDs, names, rare jargon	Little gain on purely conceptual queries
Reranker	Vocabulary-overlap false positives	Cannot recover chunks not in the candidate set
Per-domain indexes	Naturally partitioned corpora	Routing a query to the wrong index

Verification and the honest unsolved part

You have verified an improvement only when precision@k and recall@k on your labelled set both hold or rise, and a fresh handful of unseen queries agree. If precision rose but recall fell, you tightened retrieval and are now missing relevant chunks; that trade may or may not be worth it depending on whether your answers need completeness.

Here is the part the content farms will not tell you: detecting this regression in production, on live traffic, without labels, is not solved. Your labelled set is a fixed snapshot; the day real queries drift away from it, your metrics stop reflecting reality and you will not get an alert. The current honest practice is to refresh the labelled set periodically, sample live queries into it, and watch for a rising rate of "answered from a chunk a human would call irrelevant" on manual review. If you are evaluating an LLM feature more broadly before it reaches customers, the wider evaluation playbook covers where retrieval eval fits into the rest.

When it is still broken

If precision is still low after pre-filtering, hybrid search and reranking, look upstream at chunking: chunks that are too large bury the relevant sentence among noise and dilute the embedding, while chunks that are too small lose the context that makes them findable. If recall is the problem (the right chunk is never retrieved at all), your embedding model may be a poor fit for your domain's vocabulary, and a domain-adapted or larger embedding model is worth testing, again measured against the same labelled set. And if the numbers look fine but users still complain, the failure may have moved downstream into how the model uses good context, which is a generation problem, not a retrieval one.