- ·RAG is for questions whose answer lives in text you own: policies, contracts, resolved tickets. The answer itself is the deliverable, the corpus is read-only, and staleness is measured in months.
- ·MCP is for questions whose answer lives in a system that keeps changing: balances, stock, ticket state. The sentence the model writes is a receipt for an action, and every call reads current data under the caller's permissions.
- ·Freshness, permissions, and auditability decide this, not model quality. And before either, check whether one SQL query or one well-named endpoint answers the question with no AI retrieval layer at all.
Most of the RAG versus MCP arguments we get pulled into are not really about RAG or MCP. They are about a question nobody asked at the start: where does the answer live? If it lives in text your organisation wrote and stores, you have a retrieval problem. If it lives in a system that is changing while you read this, you have a tool problem. The two look almost identical in a design doc and behave nothing alike once real users arrive. Picking wrong is not fatal, but it is expensive: you rebuild the pipeline in week six, after the demo went well and the first real question landed somewhere the architecture could not reach. Here is how we make the call before any code gets written.
Where does the answer live
Ask it in the most literal way you can. If a competent employee had to answer this by hand, what would they open? If they would open a document, a policy PDF, a signed contract, a wiki page, a folder of old tickets, and read until they found the paragraph, that is retrieval. The deliverable is the answer itself, assembled out of text that already exists and does not move while they read it. If instead they would open a screen in an application, check a value, and then change something, that is a tool call. The deliverable is the change. The sentence the model writes afterwards is a receipt, not the product. Teams skip this question because both paths start the same way, with someone typing into a chat box. The chat box is the interface, not the architecture.
What RAG is actually good at
RAG earns its cost over a body of text that is large, mostly static, and written by humans for humans: staff handbooks, regulatory guidance, engineering runbooks, five years of resolved support threads, a contract archive. The defining property is not size. It is that the meaning lives in prose, so no query language can reach it. Nobody can write a WHERE clause for "the clause about early termination when the vendor is at fault". You need semantic similarity, which means chunking, embeddings, an index, and a reranker. It also means paying the tax that comes with it: an ingestion pipeline, a re-embedding job when documents change, an evaluation set, and a person who owns retrieval quality. If the corpus is forty pages, the tax outweighs the benefit and the right answer is to put the forty pages in the prompt. We covered the index half of this in vector databases, explained without the hype.
What MCP is actually good at
MCP hands a model a set of named, typed, permissioned actions against a live system: read this order, check stock at this warehouse right now, open the ticket, credit the line item, book the callback. The point is that the system stays the system of record. Nothing is copied, nothing is embedded, nothing goes stale, because every call reads the current value at the moment it is asked. It also gives you two things an index cannot: the caller's permissions get enforced per request by the system that already owns them, and every action leaves a log line with arguments and a result. If you have not built one before, the mechanics are in the piece on APIs, MCP, and giving a model hands. The real design question is never whether to expose tools. It is how coarse each tool should be, which is where most agent behaviour is won or lost, and it is the subject of designing tools an agent will not misuse.
Two ways teams get this backwards
The first: a team wants an assistant that answers stock questions, so they export the inventory table, chunk it, embed it, and ship. It demos well, because during the demo the numbers are minutes old. A week later the assistant is confidently telling a salesperson there are fourteen units in the Riyadh warehouse when there are none, and re-indexing every hour has become a cron job nobody watches until it fails quietly. Structured rows sitting in a vector index are the clearest sign the wrong tool got picked. Embeddings blur exact values by design: nearest-neighbour search over numbers cannot do greater-than, cannot aggregate, cannot join, and has no way to tell you a row was deleted. If the question has one correct answer that a query would produce deterministically, retrieval is a lossy and expensive impersonation of a database.
The second runs the other way. A team has ten thousand policy documents, decides MCP is the modern answer, and builds a single tool called search_documents that returns the top forty chunks. Now the agent is doing retrieval, badly, in the middle of a reasoning loop. It burns context on passages it did not ask for, has no reranking, no citations it can stand behind, and no way to separate a relevant paragraph from a keyword collision. Wrapping a search endpoint in a tool does not give you RAG. It gives you a search endpoint with worse ergonomics, because a model with no retrieval training is now the query planner. When the deliverable is an answer grounded in documents, build the retrieval pipeline properly and have it return few, well-chosen, cited passages. The tool interface on top is optional. The pipeline underneath is not.
The option nobody puts in the architecture diagram
Before either one, ask whether this needs an AI retrieval layer at all. A fair share of what arrives specced as RAG is one SQL query with a date filter, and several of the MCP servers we get asked to build are three endpoints a scheduled job could call directly. If the question is always the same shape, "open invoices over thirty days for this customer", the honest implementation is a view or a named endpoint called by a small deterministic workflow. It is faster, it costs nothing per call, it cannot hallucinate, and you can write a test for it. Bring in a model at the point where the input is genuinely open ended: free-text questions, messy documents, a judgement call about which of several actions applies. Our notes on choosing storage for these systems cover the plain options that keep outliving the clever ones. A model is worth adding when the alternative is a form with forty fields.
The hybrid case, which is most of them
These two are not rivals, and the most useful systems we build run both in sequence. A support agent takes a ticket, retrieves the three paragraphs of refund policy that apply to that customer tier and region, then calls a tool to issue the credit and write the note back to the CRM. Retrieval supplies the reasoning, tools supply the effect. Two rules keep the pattern honest. First, retrieved text must never be the source for a value the system already knows: read the customer's tier from the API, not from a policy chunk that happens to mention tiers. Second, retrieval runs before the action, so the policy constrains the tool call instead of explaining it afterwards. Sequence it the other way and you have built something that finds reasons for whatever it already did, which is worse than having no policy step at all.
Freshness, permissions, and audit decide this
Model quality is close to the least useful input here. Every frontier model and most open-weight ones can do either job. Three properties settle it. Freshness: how wrong is a five-hour-old answer? For a policy clause, not wrong at all. For a balance, a stock count, or a shift roster, wrong enough to cost you something. Permissions: an index flattens access control, because once a paragraph becomes a chunk it has lost the row-level rules that governed the document it came from, and you end up partitioning indexes per role and hoping. A tool call runs as a caller with an identity, so the system of record enforces what it already enforces. Auditability: "the model saw these five passages" is a weaker record than "the model called credit_order with these arguments and got this result". In regulated work that difference tends to be the entire conversation.
A checklist you can run in an hour
- 1 Name what a person would open
Take ten real questions from real users and write down the single thing a competent employee would open to answer each one. Documents on most rows means retrieval. Application screens means tools. A mix means you are building both, and it is far cheaper to know that now than in week six. - 2 Write the shelf life next to each answer
For every question, note how long the answer stays correct: a quarter, a day, a minute. Anything that decays inside a day should be read live through a tool. Anything measured in months is a fair candidate for an index, and anything in between deserves an argument. - 3 Ask who is allowed to see it
If two users asking the same question must get different answers, the permission model belongs in the system of record rather than in a copy of it. That points to tools, or at minimum to retrieval that filters by identity at query time instead of at index time. - 4 Build the boring version first
Write the query or the endpoint call that answers the question with no model involved. If that takes twenty minutes and covers most of the traffic, ship it, and spend the AI budget on the tail of open-ended questions where it actually earns its cost. - 5 Decide what the log has to prove
Write the line you would want in the audit trail six months from now, when someone asks why the system did what it did. If that line needs arguments and a result, you need tools. If it needs sources, you need retrieval with citations. Most regulated workflows need both lines.
RAG and MCP answer different questions, and the one that separates them is short: is the answer in a document, or in a system of record? Documents mean retrieval, and the answer is the product. Systems mean tools, and the answer is a receipt for something that happened. When both are true, and often they are, retrieve first and act second. Before either, check whether a query and a well-named endpoint would have been enough, because more often than the diagrams suggest, they would.
