RAG vs long context: run the crossover maths
Take a corpus of 180,000 words — a product manual, three years of internal policy documents, a mid-sized legal agreement set. At the usual English ratio of about 1.33 tokens per word that is roughly 239,000 tokens, which fits inside a million-token context window with room to spare. So you have a genuine choice that did not exist three years ago: build a retrieval pipeline, or paste the entire corpus into every prompt and let the model sort it out.
The honest answer is that this is not a philosophical question about architecture. It is an arithmetic question about query volume, and a second question about how often the corpus changes. Both have computable answers, and most teams argue about the first without ever doing the multiplication.
So do the multiplication. Gemini 2.5 Flash charges $0.30 per million input tokens. Stuffing the corpus costs 239,000 ÷ 1,000,000 × $0.30 = $0.0717 per query, call it $0.072, in input alone. At 3,000 queries a month that is $215.10. Now retrieve instead: eight or ten relevant chunks, the user's question and your instructions come to about 4,000 tokens, which is 4,000 ÷ 1,000,000 × $0.30 = $0.0012 per query, or $3.60 a month. Retrieval is 59.75 times cheaper per query on input tokens.
Output tokens are the same either way — the model writes the same answer regardless of how it got the evidence — so they cancel out of the comparison entirely and can be ignored. That is why every honest version of this calculation is about input tokens only.
Sixty times cheaper sounds decisive. It is not, and the reason is that the absolute numbers are small. The monthly saving is $211.50. A retrieval pipeline is chunking, an embedding job, a vector store, a retrieval function, an evaluation harness to check that the right chunks come back, and something to keep the index in sync. Call it a week of one engineer's time at a fully loaded $600 a day: $3,000. At $211.50 a month, that pays back in fourteen months, by which point the model you built it for has been deprecated twice.
At 3,000 queries a month, stuffing wins. Say it plainly, because the industry consensus says otherwise and the industry consensus is not doing the arithmetic.
Where the line actually falls
The crossover is a single expression:
queries per month at break-even = (build cost ÷ acceptable payback months) ÷ per-query saving
The per-query saving here is $0.0717 − $0.0012 = $0.0705. If you want the pipeline to pay for itself within three months, you need to recoup $1,000 a month, which needs $1,000 ÷ $0.0705 = 14,184 queries a month. Below roughly 14,000 queries a month on this corpus at this price, retrieval is an engineering hobby. Above it, it is a cost control.
Push it further and the argument stops being close. At 100,000 queries a month, stuffing costs 100,000 × $0.0717 = $7,170 and retrieval costs 100,000 × $0.0012 = $120. The $7,050 monthly difference pays for the entire build in the first fortnight and then keeps paying, every month, forever. Run your own corpus size and volume through the context window cost calculator and the RAG pipeline cost calculator rather than trusting the shape of this example.
Two things move that line hard, and both are model choices rather than architecture choices.
The first is which model you stuff. Gemini 2.5 Flash at $0.30 per million is close to the cheapest long-context option in our index. The same corpus on Gemini 2.5 Pro at $1.25 per million costs 239,000 ÷ 1,000,000 × $1.25 = $0.29875 a query — $896.25 a month at 3,000 queries, four times the Flash figure. On Gemini 3.5 Flash at $1.50 per million it is $0.3585 a query, or $1,075.50 a month. "Long context is cheap now" is a statement about one specific model, and the newer, better model in the same family costs five times as much per input token. If your quality bar forces you up a tier, the crossover collapses towards you.
The second is caching. Gemini 2.5 Flash prices cached input at $0.03 per million, a tenth of the standard rate. A corpus that does not change between queries is the ideal cache candidate: 239,000 × $0.03 ÷ 1,000,000 = $0.00717 a query, so $21.51 a month at 3,000 queries. That is only six times the cost of retrieval rather than sixty, and it moves the break-even volume out by roughly an order of magnitude — to something like 140,000 queries a month before a pipeline pays back in three months.
Two caveats keep that from being the end of the argument. Our snapshot prices cached reads but not cache storage, which vendors bill separately by duration; check the storage rate on the vendor's page before you build a plan around it, because for a 239,000-token blob held live it is not trivial. And cache TTLs are short — minutes to hours — so the discount only materialises if your queries arrive in bursts dense enough to keep the cache warm. Three thousand queries a month spread evenly is roughly four an hour, which will miss a short TTL most of the time. The thing that settles this is your own hit rate: log it for a week before you assume the cached price applies. More on the mechanics in prompt caching explained.
The arguments that are not about money
Cost is the part you can compute, which is why it dominates the discussion. It is not the part that usually decides.
Latency. Every stuffed query prefills 239,000 tokens before the first output token appears. Time to first token scales with prefill length, and on a large corpus this is seconds, not milliseconds. Retrieval adds a vector search — tens of milliseconds — and then prefills 4,000 tokens instead of 239,000, roughly sixty times less work. For a batch job nobody cares. For anything with a cursor blinking in front of a human, this is the whole ballgame, and it is the reason plenty of teams above the cost crossover would build retrieval even if stuffing were free.
Mid-context recall. Long-context models do not attend uniformly across their window. Recall of a specific fact is reliably strongest near the start and end of the context and measurably weaker in the middle, and the effect gets worse as you fill more of the window. Stuffing 239,000 tokens means your answer depends on where in that blob the relevant paragraph happens to sit — which is determined by your document ordering, which is arbitrary. Retrieval sidesteps this by putting the relevant passage in the first 4,000 tokens of a short prompt. If you stuff, at minimum order your documents deliberately and put the most-queried material at the edges.
Freshness. A stuffed prompt is a snapshot assembled when the request was built. If a policy document changed this morning, every cached version of that context is wrong until the cache expires, and every prompt built from a stale copy of the corpus is wrong until someone rebuilds it. A retrieval index updates per document: change one chunk, re-embed that chunk. Re-embedding an 800-token chunk with OpenAI's text-embedding-3-small at $0.02 per million costs $0.000016. Embedding the entire 239,000-token corpus from scratch costs $0.00478 — under half a cent, or $0.0287 on voyage-4-large at $0.12 per million. The embedding bill is not a real line item at this scale; the freshness discipline is the thing you are buying.
Citations. Retrieval hands you document identifiers as a side effect of doing the retrieval, so a citation is a fact about your pipeline. With stuffing, a citation is a claim by the model, and verifying it means string-matching back into a 239,000-token blob. In any domain where someone will eventually ask "where does it say that" — legal, medical, compliance, finance — this alone decides it.
When stuffing is the right answer
Low volume, a small and stable corpus, and no engineering capacity to spare. If you are running a few hundred queries a month against a corpus that fits in a window and changes quarterly, a retrieval pipeline is a liability you will maintain badly. Paste the documents in, cache what you can, and revisit when volume grows.
The same applies for the first two weeks of anything. Stuffing is the fastest way to find out whether the feature is worth building at all, and you learn more from a working prototype with a $0.072 per-query bill than from a chunking strategy debate.
And there is one case where stuffing wins on capability rather than convenience: queries that require reasoning across the whole corpus at once. "Which of these 400 contracts contradict one another", "summarise how our refund policy has drifted over three years", "find every clause that is inconsistent with clause 14" — none of these have a relevant chunk to retrieve, because the answer is a property of the entire set. Retrieval returns the top-k passages most similar to the question, and for a global question the top-k is meaningless. If your query mix is mostly global, retrieval will not merely be more expensive to build; it will be wrong, and no amount of chunk tuning fixes it.
Sort your last thousand real queries into local and global before you pick. If more than a fifth are global, you are building a hybrid, and you should budget for the fact that a hybrid costs more than either option alone.
All figures in this guide come from the price index, last verified 15 August 2026. Every entry links to the vendor page it was read from — see the price index and the methodology.