Why does my RAG system return wrong or irrelevant answers in production?

In the large majority of cases the model is fine and retrieval is failing — the right chunk was never fetched into the context window, so the model answers from priors or hedges. Debug retrieval first: log what was retrieved for a failing query, check whether the correct chunk even exists in the index, and measure recall (did we fetch the relevant chunk) separately from generation. Chunking, embedding mismatch, missing metadata filters and lack of reranking cause most production drops.

How do I know if it's a retrieval problem or a model problem?

Separate the two stages. For a failing query, inspect the retrieved chunks before generation. If the correct information is not in the retrieved set, it is a retrieval problem — no better model fixes it. If the correct chunk was retrieved but the answer is still wrong, it is a generation/prompting problem. This single split resolves most 'the LLM is dumb' complaints, which are usually 'the LLM never saw the answer'.

What are the biggest causes of poor RAG retrieval?

Chunking that splits or dilutes the answer; an embedding model that doesn't match the domain or query style (or a mismatch between the model used to index and the one used to query); missing metadata filtering so the right document is buried under near-duplicates; user queries that don't lexically resemble the source text; and no reranking step to reorder the top candidates before they hit the context window.

Does adding reranking fix RAG retrieval quality?

Reranking helps a lot when your vector search returns the right chunk somewhere in the top 20-50 but not in the top 3-5 you actually pass to the model. A cross-encoder reranker reorders candidates by true relevance to the query, lifting precision at the small k you can afford in the context window. It does not help if the correct chunk is not retrieved at all — that is a chunking, embedding or hybrid-search problem upstream of reranking.

Why RAG Retrieval Quality Drops in Production (and How to Fix It)

September 7, 2026

A retrieval-augmented generation system that answered perfectly in the demo and then returns vague, wrong or "I don't have that information" answers in production is following a script every AI engineer eventually learns: the model is almost never the problem — retrieval is. If the correct chunk never lands in the context window, no upgrade from one frontier model to another will save the answer. This guide is the retrieval debugging checklist, in the order that finds the cause fastest.

First, split the pipeline in two

RAG has two stages that fail for completely different reasons: retrieval (find the relevant chunks) and generation (write the answer from those chunks). Before touching anything, take a failing query and log what was actually retrieved:

The correct information is not in the retrieved set → retrieval problem. Stop blaming the model. Everything below applies.
The correct chunk was retrieved but the answer is still wrong → generation/prompting problem (context ordering, prompt, model refusing to trust context, hallucination — see the companion piece).

This one split resolves most "the LLM got dumber" complaints. They are almost always "the LLM never saw the answer." Measure recall (did we fetch the relevant chunk at all) separately from answer quality — they are different metrics with different fixes.

Cause 1 — Chunking that destroys the answer

Chunking is the highest-leverage and most-neglected knob. Failure modes:

Too small — the answer is split across two chunks, so no single retrieved chunk contains it, and the model gets half the story.
Too large — one chunk covers many topics; its embedding is an average that matches nothing sharply, so it loses to more focused chunks even when it holds the answer.
Structure-blind — fixed-character splitting cuts through tables, code blocks, or a heading and its paragraph, orphaning context.
No overlap — a fact that straddles a boundary is lost. Modest overlap (or semantic/structure-aware chunking) preserves it.

What I check first: pull the source document, find the passage that should answer the query, and look at how it was chunked. If the answer spans a boundary or sits in a giant mixed chunk, that is your bug — fix ingestion, not the model.

Cause 2 — Embedding mismatch

Embeddings decide what "similar" means. Common mismatches:

Index/query model mismatch — vectors were built with one embedding model and queries are embedded with a different one (or a different version). The spaces don't align and similarity is noise. Re-embed the whole corpus when you change the model.
Domain mismatch — a general-purpose model that has never seen your acronyms, part numbers or legal phrasing embeds them poorly. Domain-adapted or higher-quality embedding models help.
Asymmetry — short user questions and long document passages live in different regions of the space. Models trained for asymmetric retrieval (query vs passage) matter here; some need an instruction/prefix to work well.

Cause 3 — The query doesn't look like the source

Users ask "why is my invoice rejected?"; the document says "validation failure codes for AP submissions." Pure semantic similarity can bridge some of that gap, but not all. Two fixes:

Query transformation — rewrite or expand the user query (multi-query, HyDE-style hypothetical answer) before retrieval so it better matches source phrasing.
Hybrid search — combine dense vectors with keyword/BM25 so exact tokens (error codes, IDs, product names) are matched lexically while semantics handles paraphrase. This is often the single biggest production win — covered in depth in hybrid search vs vector search.

Cause 4 — Missing metadata filtering

In a demo corpus of 50 documents, everything is findable. In production with 500,000 chunks, the right passage is buried under near-duplicates from other tenants, other product versions, or archived material. Without metadata filters (tenant, product version, date, document type, access scope) applied before or during vector search, you retrieve the semantically-closest chunk from the wrong context. Filtered retrieval is frequently the difference between a toy and a product — and it is also a security control (never retrieve a chunk the user isn't entitled to see).

Cause 5 — No reranking

Vector search is fast but approximate; it gets the right chunk into the top 20–50 far more reliably than into the top 3–5. But you can only afford to put a few chunks in the context window. A cross-encoder reranker re-scores the top candidates by true query–passage relevance and reorders them, so the chunk that matters rises into the small k you actually pass to the model. Reranking lifts precision-at-small-k dramatically — when the correct chunk was retrieved at all. It cannot rescue a chunk that chunking/embedding/hybrid failures never fetched, so fix those first.

The production drift you didn't cause

Retrieval quality can degrade over time without a code change:

Corpus growth — more documents means more near-neighbours competing for the top slots; thresholds tuned on a small corpus stop holding.
Stale index — ingestion silently failing means new documents never get embedded; users ask about content that isn't indexed.
Query distribution shift — real users ask differently than your test set; the queries you never evaluated are the ones failing.

A retrieval triage table

Symptom	Most likely cause	First fix
Right doc exists, never retrieved	Chunking or embedding mismatch	Inspect chunk boundaries; verify index/query model match
Exact IDs/codes missed, paraphrase works	Pure vector, no lexical match	Add hybrid (BM25 + dense)
Right chunk in top-30 but not top-5	No reranking	Add a cross-encoder reranker
Wrong tenant/version chunk returned	No metadata filter	Filter by tenant/version/date/scope
Worked, degraded over weeks	Corpus growth / stale index	Re-tune k/threshold; verify ingestion

Measure it, don't guess

Build a small golden set of real queries with known-correct source passages and track retrieval recall@k and answer correctness separately on every change. Without this, every tweak is a vibe. Evaluation beyond raw accuracy — faithfulness, context precision/recall — is its own discipline; frameworks like RAGAS operationalise it.

Common wrong approaches

Swapping to a bigger LLM. If retrieval missed the chunk, a smarter model just hallucinates more confidently.
Cranking k to 50. Floods the context with noise, raises cost and latency, and can bury the right chunk in distractors.
Tuning the prompt to fix a retrieval miss. You cannot prompt your way to information that isn't in the context.
Re-embedding queries with a new model but not the corpus. Guarantees a broken vector space.

Related resources

Hybrid search vs vector search: a RAG retrieval decision guide.
LLM hallucination: a root-cause analysis framework — the generation-stage counterpart.
RAG explained: retrieval-augmented generation architecture and, for AWS stacks, Bedrock Knowledge Bases RAG troubleshooting.
RAG & Agentic AI job support guide and MLOps job support guide.

If a RAG system is misbehaving in a live project and you need a second senior opinion on whether it's retrieval or generation, real-time proxy job support can work the pipeline with you. And if you're being interviewed on production RAG, being able to reason through this split is exactly what RAG interview proxy support and GenAI interview proxy support prepare you for.

Last reviewed: September 2026.