Why does RAG give wrong answers? Most failures are in retrieval, not the model
The Ship AI Engineer team · Published 2026-08-31
A system that answers questions about internal documents goes live. In the first week an employee asks about the return window, and the system confidently says "30 days". The policy says 14. The team's first move is to change the prompt, then to switch to a more capable model. Two days later the same failure shows up on a different question.
Eventually someone looks at which document chunks were sent to the model for that question. None of them contains the return policy; with no answer in hand, the model said the most plausible number. The problem was never the model.
What is RAG?
A way of grounding the model's answer in your documents rather than its training data.
Retrieval-Augmented Generation has two steps: first the document chunks relevant to the question are found (retrieval), then those chunks are given to the model with the question, and the model produces an answer based on them (generation). That lets the model answer about documents written after it was trained, or ones it has never seen. We covered how retrieval works in our post on embeddings.
Why does a RAG system give wrong answers?
Most of the time because the chunk carrying the answer was never sent to the model.
A wrong answer has three possible sources: the right chunk was never retrieved, the right chunk was retrieved but lost among wrong ones, or the right chunk was retrieved and the model misread it. In practice the first two are far more common than the third. That's why changing the prompt or upgrading the model is the most expensive and least effective first reaction to a RAG failure.
How do you tell whether the failure is in retrieval or generation?
By looking at the chunks sent to the model for that question before looking at the answer.
Log the retrieved chunks on every request. When you hit a wrong answer, the first question is: is the information that answers it in these chunks? If not, the problem is retrieval: look at how documents are chunked, how the question searches and which kind of search you use. If it is, the problem is generation: look at the prompt, the order of the chunks and whether the model is following instructions. Any fix made without separating those two cases is down to luck.
The same split applies to measurement. Keep two separate numbers: how often the right chunk is among the top few results, and how often the answer stays faithful to the retrieved chunks. Only then can you see one improve while the other gets worse.
Why does the model say something that isn't in the documents?
Because its job is to produce an answer, and when there isn't one it produces the most likely-looking one.
A language model produces what is likely, not what is true. When the retrieved chunks don't contain the answer, the model builds a plausible one from its general knowledge or from similar information in the chunks. The ways to contain this are well known: tell the model plainly "answer only from the documents provided, and say so if the answer isn't there", give it a real "I don't know" exit, and check before sending whether the answer's claims actually appear in the chunks. A system that can say "these documents don't cover that" to an unanswerable question is far more trustworthy than one that answers everything.
Why does citing sources matter?
Because it makes the answer verifiable, for the user and for you.
Showing under each answer which piece of which document it rests on lets the user check the answer themselves. For you it's a measuring tool too: you can check automatically whether the cited source actually supports the answer. An answer that can't cite a source should be treated with suspicion before it reaches the user.
How many chunks should you retrieve?
As few as will contain the answer, and no more.
Sending more chunks looks safe at first: the right one is bound to be in there somewhere. But every chunk costs tokens, which means cost and latency. And models pay less attention to information buried in the middle of a very long input than to what's at its start and end. Sending twenty chunks can bury the right one at position seventeen where the model overlooks it. The better path is to improve retrieval and send fewer, more accurate chunks.
How should you evaluate a RAG system?
With questions phrased the way users actually ask them, including questions whose answer isn't in the documents.
The most common mistake is having the person who wrote the documents write the test questions. They ask using the documents' own words, and search looks perfect. Real users ask in their own words. A good test set includes questions taken from real usage logs, paraphrases and questions the documents can't answer. That last group is the only test that shows whether the system makes things up. You can follow a real RAG system breaking, step by step, in the failure cases.
Sources
- Lewis et al. (2020): Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks: the paper that introduced the RAG approach. (added 2026-08)
- Gao et al. (2023): Enabling Large Language Models to Generate Text with Citations: models answering with citations, and how that's measured. (added 2026-08)
- Anthropic: Reducing hallucinations: techniques for letting a model say "I don't know" and tie its answer to a source. (added 2026-08)
- LangChain: text splitters: different ways of chunking documents. (added 2026-08)
Keep reading
2026-10-06
Resources worth following as an AI engineer: what to read, and when
A new post, model or tool appears in this field every day, and trying to follow all of it is the fastest way to learn none of it. A handful of resources that genuinely help, and the stage of learning at which each one is worth reading.
2026-10-05
Being an engineer in the age of AI: code got cheap, structure didn't
Writing code has never been this fast. But code produced quickly amplifies whatever structure sits underneath it. Why engineering matters more today, not less, and where to go deeper.
2026-10-02
Shipping an AI application: spending caps, caching, monitoring and model changes
In an ordinary web app every request costs roughly the same. In an app with a language model inside, it doesn't. Capping cost when you ship, using caching safely, watching the right things, and being ready when the provider retires a model.
2026-09-30
An AI engineer portfolio: what a project that gets you hired looks like
A chatbot demo impresses nobody anymore. Six signs that make an AI project credible, five project ideas that each show a different skill, and a README someone can read in two minutes.
2026-09-27
Prompt injection and abuse: why the server has to be the single source of truth in an AI app
An application with a language model inside carries every security problem of an ordinary web app and adds new ones. Why prompt injection can't be fully prevented, how to limit the damage, protecting the endpoint, and why the model's output can't be trusted.
2026-09-23
Building your first test set for an LLM application: a guide to evaluation
You changed the prompt and the answers look better. Do they? Building a first test set, taking a baseline, seeing what the average hides, the limits of one model grading another, and watching the system once it's live.
2026-09-19
When should you fine-tune, and when is it a waste of time?
Training a model on your own data is an attractive idea. Most of the time it's the wrong tool. What fine-tuning teaches and what it doesn't, how it compares with prompting and RAG, data quality, leakage, LoRA, and licences on open models.
2026-09-14
7 questions to ask before you pick an AI engineering course
Seven questions worth asking before you spend money and months on an AI engineering course. A short answer to each, and the concrete sign you can check in the syllabus before you buy.
2026-09-09
AI with image and audio input: the costs and failures your tests don't show
Handing a model a photo or a recording isn't the same as sending text. Request size, unreadable images, sentences invented in silence, generation that takes seconds, and an image paid for again on every turn.
2026-09-04
When don't you need an agent? Building and stopping a tool-using system safely
"Agent" is today's most requested and most easily misbuilt thing. The difference between a fixed workflow and an agent, why tool descriptions are critical, how to stop the loop, and how to protect irreversible actions.
2026-08-26
How to become an AI engineer: a roadmap that builds products, not models
Most roadmaps in the search results describe a different job. What the AI engineers being hired today actually know, the order to learn it in, and what makes each stage "done".
2026-08-21
When do you actually need a vector database? A practical guide to embeddings
The first step people take after deciding to build semantic search is usually picking a vector database. Most of the time it's an early step. What an embedding is, what a similarity score doesn't tell you, and the real reasons to move to a database.
2026-08-17
Managing prompts like code: versions, templates, schemas and tests
A prompt that works in a chat window quietly breaks inside an application. Keeping prompts in the repository, separating user input from instructions, validating the answer against a schema, and shipping a change only after measuring it.
2026-08-12
Why LLM APIs break under real traffic: rate limits, retries and timeouts
Three errors you never saw in development arrive together in the first busy hour. Telling 429 from 5xx, retrying properly, setting timeouts, and keeping the product up when the provider goes down.
2026-08-07
Do you need maths to become an AI engineer?
The honest answers to the six questions that stop people before they start: maths, degrees, machine learning, Python, hardware and money. Short answers first, reasons after.
2026-08-07
An AI engineer is not a machine learning engineer
Search "how to become an AI engineer" and the first page teaches linear algebra, CNNs and training loops. That is a different job. Here is what the work actually is, and what you can skip.