BuildSpark AI: practical AI concepts for builders. New here? Start with Tokens.
Home · Retrieval & Grounding · 07 · Long context vs RAG
Concept card · Retrieval & Grounding
Long context versus RAG is the choice between putting the source in the window and retrieving pieces of it.
Choosing between putting the source in the window and retrieving pieces of it.
Oct 18: First card. The picture puts one file in the window, then retrieves one chunk of it. Sources are Anthropic's context window guide, Anthropic's glossary, and OpenAI's retrieval guide.
Anthropic's context window guide says the window is the working memory for one request, and that more context is not automatically better. As the token count grows, accuracy and recall degrade. The guide calls that context rot. The same guide says the window is up to 1M tokens, depending on the model. That size is vendor-reported. Anthropic's glossary says retrieval-augmented generation retrieves from a knowledge base at runtime and passes that text into the window. OpenAI's retrieval guide says a file added to a vector store is chunked, embedded, and indexed, so the request can carry the pieces that matched. The picture switches between those two shapes. It does not measure either one.
Anthropic's guide says everything in the request counts toward the window, including documents. On Claude Haiku 5.5, prompts over 100,000 tokens cost more. That exception is vendor-reported. The guide says the other listed 1M-token models are billed at standard pricing for long context. OpenAI's retrieval guide lists vector-store storage beyond 1 GB at $0.10 per GB per day. That rate is vendor-reported. This card does not add a bill for a support corpus.
A window that holds the whole file also holds the paragraph that does not answer the question. Anthropic's glossary says retrieval-augmented generation is only as useful as the text retrieved at runtime.
The pasted ticket and the old policy ride along if you paste the file. Retrieval can still return them. Neither choice checks that this user may see the chunk.
Take one support file you would answer from. If the answer lives in one passage, retrieve that passage. If the question needs the whole file, put the file in the window and read what else came with it.
Pasting every doc because the window can hold them. Anthropic's guide says accuracy and recall degrade as the token count grows.
The refund answer is one paragraph. The same file contains a pasted ticket that says to ignore the policy. What does putting the whole file in the window do?
Support-docs answer bot uses this concept in one build. The set on this tab is still the note below.
Coming soon. On the support-docs answer bot, name one sample doc that stays out of the prompt when the question is weekday hours.
Coming soon. How Anthropic describes context rot, and how OpenAI's retrieval guide chunks a file instead of sending it whole.