RAG or long context: the right answer is 'it depends', and here is on what

When context windows started being measured in millions of tokens, a seductive argument appeared: if everything fits, why build a retrieval architecture at all?

The evidence published since does not support that conclusion. It does not support the opposite one either.

What the research says

The LaRA benchmark, presented at ICML, evaluated 2,326 test cases across four question-answering task types and three long-context types over eleven models. Its conclusion is titled, bluntly, “no silver bullet”: the optimal choice depends on the interplay of model capability, context length, task type and retrieval characteristics.

On the enterprise side, EnterpriseRAG-Bench flags a relevant gap: existing datasets focus on web or public sources, and no widely adopted benchmark realistically reflected company-internal knowledge. Which is exactly where most real projects operate.

When long context wins

• The corpus is small and stable. One contract, one tender, one manual. If it fits whole and doesn’t change daily, putting it in context is simpler and usually performs better.
• The question requires seeing everything at once. “Are there contradictions between these clauses?” is not answered by retrieving three fragments.
• You are prototyping. Building RAG before knowing whether the use case works is premature optimisation.

When retrieval wins

• The corpus is large or growing. Thousands of documents, or new ones every day.
• Cost matters. Pushing a million tokens per query is expensive even though the per-token price has fallen. Retrieving five fragments is not.
• You need to cite the source. RAG gives you traceability of which document backed the answer. Long context, far less.
• There is access control. If different users can see different documents, retrieval filters before generating. With everything in context, filtering is a problem.

That last point is constantly underestimated and is usually what decides the architecture at companies with sensitive data.

The common mistake

Building RAG as if it were a closed product: chunk, embed, similarity search, done. When quality falls short, the team changes model. It is almost never the model.

Where quality is lost, in order of frequency:

1. Chunking breaks meaning. Cutting every 500 tokens splits tables, lists and clauses in half.
2. Similarity search doesn’t find the right thing. Semantic similarity fails on proper nouns, codes and references. Combining lexical and vector search fixes more than you’d expect.
3. Too little or too much is retrieved. Both degrade; and only your evaluation tells you the right number.
4. Nothing is reranked. A reranker over the candidates usually delivers more improvement than swapping the generating model.

What we recommend

Start by putting the document in context and measure. If it works and the cost holds, you are done — and you have saved yourself an architecture.

When the corpus grows, cost bites or access control appears, build retrieval. By then you will have something most teams lack at the start: an evaluation set that tells you whether the change actually improved anything.

Sources

• LaRA: Benchmarking Retrieval-Augmented Generation and Long-Context LLMs — No Silver Bullet for LC or RAG Routing (ICML)
• EnterpriseRAG-Bench: A RAG Benchmark for Company-Internal Knowledge — arXiv
• Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach — arXiv
• LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks — arXiv