Generative AI & LLMs
RAG vs CAG: When Cache-Augmented Generation Makes Sense
Written by Tehreem FatimaReviewed by Umaid Asim
Published 15 min read

RAG (retrieval-augmented generation) searches your documents every time someone asks a question and sends the model only the passages it finds. CAG (cache-augmented generation) loads the whole knowledge base into the model's context once, caches the result, and answers every question against it. CAG suits small knowledge bases that rarely change. RAG suits large, changing or permission-controlled ones.
CAG takes its name from a 2024 research paper with a deliberately blunt title, "Don't Do RAG" Source [1]. Since then, million-token context windows and cheap cached input have made the idea practical on hosted models. For teams building an internal assistant or a support bot, CAG vs RAG is now a real choice: build a retrieval pipeline, or skip it.
This article is for the people making that call. It explains what the original research did and did not show and what CAG costs next to RAG at current API prices. It ends with a way to tell which approach your knowledge base needs.
What Is Cache-Augmented Generation (CAG)?
Cache-augmented generation (CAG) is a way to give a language model your knowledge without a search step. You load every relevant document into the model's context window ahead of time and save the model's processed state, called the key-value (KV) cache. Each new question reuses that saved state, so nothing is retrieved at answer time.
The method was proposed by researchers at National Chengchi University and Academia Sinica in Taiwan and accepted at the Web Conference 2025 Source [1]. It runs in three phases:
- Preload. The documents are formatted to fit the model's context window. The model reads them once and produces a KV cache, which is stored in memory or on disk Source [1].
- Answer. Each question is added to the end of the cached state, and the model answers using everything it has already read Source [1].
- Reset. The question's tokens are then trimmed off the end of the cache. The next question starts from the clean, preloaded state, without the documents being read again Source [1].
The KV cache is the model's working notes on text it has already read. OpenAI's documentation describes the same mechanism behind its own prompt cache, which "stores key-value (KV) tensors, not the tokens themselves" Source [2]. Producing those notes is the slow, expensive part of reading a long document. CAG pays that cost once instead of on every question.
Two Things CAG Is Not
CAG is not response caching. A semantic cache stores finished answers and returns one when a new question looks similar enough. Microsoft's documentation for Azure API Management describes returning cached responses "for prompts that are similar in meaning, even if the text isn't identical" Source [3]. CAG stores the processed documents, not answers, and still writes a fresh answer to every question.
CAG also has a second meaning. Some articles use the same letters for context-augmented generation, a loose label for any approach that puts extra material into the prompt. Here, CAG means the cache-based method from the original paper.
RAG vs CAG: The Core Differences
The core difference between RAG and CAG is when knowledge reaches the model. RAG picks a few passages for each question, an approach introduced in 2020 Source [4]. CAG hands the model everything up front and reuses it. That choice decides how large the knowledge base can be, how updates work, who can see what, and where errors come from.
| Aspect | RAG | CAG |
|---|---|---|
| How knowledge reaches the model | A search step picks passages for each question | The whole knowledge base is loaded once and cached |
| Size of the knowledge base | Can grow far beyond the context window | Must fit inside the context window, with room to spare |
| Updating content | Re-index the documents that changed | Rebuild the cache whenever the cached text changes |
| Different access for different users | Search results can be filtered per user | Everyone using one cache can be answered from all of it |
| Main way it fails | The search misses the passage that holds the answer | The model overlooks a fact buried in a long input |
| Parts to build and run | Chunking, embeddings, a search index, often a reranker | A long-context model and cache management |
| Cost per question | A few thousand retrieved tokens at the normal input price | The whole cache, reread at a discounted price |
How knowledge reaches the model
- RAG
- A search step picks passages for each question
- CAG
- The whole knowledge base is loaded once and cached
Size of the knowledge base
- RAG
- Can grow far beyond the context window
- CAG
- Must fit inside the context window, with room to spare
Updating content
- RAG
- Re-index the documents that changed
- CAG
- Rebuild the cache whenever the cached text changes
Different access for different users
- RAG
- Search results can be filtered per user
- CAG
- Everyone using one cache can be answered from all of it
Main way it fails
- RAG
- The search misses the passage that holds the answer
- CAG
- The model overlooks a fact buried in a long input
Parts to build and run
- RAG
- Chunking, embeddings, a search index, often a reranker
- CAG
- A long-context model and cache management
Cost per question
- RAG
- A few thousand retrieved tokens at the normal input price
- CAG
- The whole cache, reread at a discounted price
RAG's moving parts are where most of its tuning effort goes. Our guide to RAG architecture and retrieval quality covers chunking, embeddings, reranking and evaluation in detail. CAG removes those parts, but it moves the risk rather than removing it. Instead of asking whether the search found the right passage, you ask whether the model used the right part of a very long input.
What the CAG Research Actually Found
The original CAG paper found that preloading beat RAG in most of its tests, but the tests were small. It used one model with 8 billion parameters and knowledge sets of 21,000 to 85,000 tokens. On the largest multi-document set, two RAG setups scored higher than CAG, and the simplest RAG setup also answered faster.
The authors compared CAG with keyword (sparse) and embedding (dense) retrieval on two question-answering benchmarks, SQuAD and HotPotQA. Every system used the same model, Llama 3.1 8B, and answers were scored with BERTScore, which measures how closely an answer matches a reference answer Source [1]. Three details matter if you plan to use the paper to justify a build:
- The advantage shrank as the knowledge grew. On the 85,000-token HotPotQA set, CAG scored 0.7407, while keyword RAG using the top three or top five passages scored 0.7463 and 0.7535 Source [1]. The authors connect this to the known drop in performance on very long inputs.
- The benchmarks may have been too easy. Simple keyword search beat embedding search, which the authors say suggests the datasets were not challenging enough to need deeper semantic retrieval Source [1].
- Skipping retrieval is not the same as answering faster. On the largest set, CAG took 2.26 seconds per answer, against 0.67 seconds for keyword RAG with three passages. CAG's clear win was against sending the same text without a cache, which took 92 seconds Source [1].
The paper also notes that Llama 3.1 8B handles about 32,000 tokens effectively, although it accepts 128,000 Source [1]. Its largest test set was more than twice that size. The authors state the limit plainly: loading everything becomes impractical for significantly larger datasets Source [1].
What Other Research Says About Long Inputs
A model that accepts a million tokens does not use all of them equally well. RULER, a benchmark published at COLM 2024, tested 17 long-context models. All of them claimed at least 32,000 tokens, yet "only half of them can maintain satisfactory performance at the length of 32K" Source [5]. The Lost in the Middle study found that models often answered best when the relevant fact sat at the start or end of the input. They did worse when it sat in the middle Source [6]. Anthropic's own documentation warns that "as token count grows, accuracy and recall degrade", an effect it calls context rot Source [7].
Long inputs can still win. A study presented at EMNLP 2024 found that long-context models, given enough resources, outperformed RAG on average across its benchmarks. RAG, though, kept a large cost advantage Source [8]. The evidence points the same way as the CAG paper: preloading works best when the knowledge is small enough for the model to use well.
How CAG Works on a Hosted Model API
On a hosted model such as Claude or GPT, you do not hold the KV cache yourself. Prompt caching does that job: the provider saves the processed state of a repeated prompt prefix and charges less to reuse it. The catch is that the cache expires, costs extra to write, and only matches when the cached text is identical.
The CAG paper kept its cache on its own GPUs Source [1]. Most teams using a hosted model get the same effect by putting the full knowledge base at the start of every request and marking it for caching. Four rules decide how well that works:
- The cache expires. Anthropic's default cache lasts five minutes and is refreshed at no extra cost each time it is used. A one-hour cache is available at a higher write price Source [9].
- Writing costs more than normal input. On Claude, a five-minute cache write costs 1.25 times the base input price and a one-hour write costs 2 times Source [9]. On OpenAI's GPT-5.6 and later models, a 30-minute write costs 1.25 times Source [2].
- Reading costs much less. Cache reads on Claude Sonnet 5.5 and Claude Opus 5.5 cost 0.05 times the base input price. Most other Claude models charge 0.1 times Source [9]. OpenAI lists 0.1 times for most GPT-5.6 and later models, and 0.05 times for GPT-6.1 Sol and GPT-6.1 Terra Source [2].
- Any change breaks the match. The cached prefix must match exactly. If anything before a cache breakpoint changes, the cache entry after it can no longer be reused Source [2]. Put the knowledge base first and everything that varies, such as the user's question, after it.
What CAG Costs Compared With RAG
Once its cache is warm, CAG can cost about the same per question as RAG, because cached tokens are cheap. The risk is the cache miss. At Claude Sonnet 5.5 prices, rereading a cached 100,000-token knowledge base costs about $0.01 per question. Writing the cache costs $0.25 to $0.40, more than sending the text uncached.
The worked example below uses a 100,000-token knowledge base, roughly 75,000 words of English Source [10]. Prices are Claude Sonnet 5.5's listed rates in November 2026, starting at $2 per million input tokens Source [9]. The RAG setup sends 4,000 tokens of retrieved passages per question. The question, the answer and the RAG search index are left out, because the first two cost the same either way and index hosting varies by provider.
| How the knowledge reaches the model | Tokens billed | Price per million tokens | Cost per question |
|---|---|---|---|
| Whole knowledge base, no caching | 100,000 input | $2.00 | $0.20 |
| CAG, writing a five-minute cache | 100,000 cache write | $2.50 | $0.25 |
| CAG, writing a one-hour cache | 100,000 cache write | $4.00 | $0.40 |
| CAG, reading a warm cache | 100,000 cache read | $0.10 | $0.01 |
| RAG, retrieved passages only | 4,000 input | $2.00 | $0.008 |
How the knowledge reaches the model
Whole knowledge base, no caching
- Tokens billed
- 100,000 input
- Price per million tokens
- $2.00
- Cost per question
- $0.20
How the knowledge reaches the model
CAG, writing a five-minute cache
- Tokens billed
- 100,000 cache write
- Price per million tokens
- $2.50
- Cost per question
- $0.25
How the knowledge reaches the model
CAG, writing a one-hour cache
- Tokens billed
- 100,000 cache write
- Price per million tokens
- $4.00
- Cost per question
- $0.40
How the knowledge reaches the model
CAG, reading a warm cache
- Tokens billed
- 100,000 cache read
- Price per million tokens
- $0.10
- Cost per question
- $0.01
How the knowledge reaches the model
RAG, retrieved passages only
- Tokens billed
- 4,000 input
- Price per million tokens
- $2.00
- Cost per question
- $0.008

Scaled to 20,000 questions a month, the traffic pattern decides the bill. Say questions arrive at least once an hour through a 22-day working month, and a one-hour cache is written once each morning. CAG then costs about $209: about $200 in cache reads and $8.80 in cache writes. Sending the knowledge base uncached would cost $4,000. RAG would cost about $160 in retrieved tokens, plus whatever its search index costs to run. If questions are so sparse that the cache has always expired, every question pays the write price, and caching costs more than not caching at all.
The answers themselves add the same amount to every option. Our breakdown of what it costs to run an AI chatbot shows how that side of the bill adds up.
How to Tell if Your Knowledge Base Fits
Count tokens, not pages. For English text, 100 tokens is roughly 75 words, so a 1 million token context window holds roughly 750,000 words. For CAG, aim well below the limit, because accuracy can fall long before the window is full. Then test the model at your real size before you commit.
OpenAI gives the 100 tokens to 75 words figure as a rough estimate for English and notes that other languages differ Source [10]. Claude Sonnet 5.5 and Claude Opus 5.5 accept up to 1 million tokens Source [7]. Anthropic's 2024 rule of thumb was simpler. If your knowledge base "is smaller than 200,000 tokens (about 500 pages of material)", you can put all of it in the prompt with no need for RAG Source [11].
Before choosing CAG, run three checks:
- Measure the real token count. Use your provider's token counting tool on the actual files, including tables and formatting, rather than estimating from a word count Source [7].
- Leave room for everything else. Instructions, conversation history, the question and the model's answer all count toward the same window Source [7].
- Plan for growth. A knowledge base that fits today but doubles every year will not fit for long. Moving from CAG to RAG later means building the retrieval pipeline you skipped.
Then test accuracy at your real size. Ask 50 to 100 real questions with known answers, including some whose answers sit in the middle of the material. Compare CAG with a simple RAG baseline on the same questions. Our post on LLM testing explains how to set that up.
When CAG Makes Sense, and When RAG Is the Better Choice
CAG makes sense when the knowledge base is small and changes rarely. It should also be open to every user, with questions arriving often enough to keep the cache warm. RAG is the better choice when any of those fail: the content is large or growing, updates daily, needs different access for different users, or gets only occasional questions.

Signs CAG Is a Good Fit
- The knowledge is a bounded set, such as a product manual, a policy handbook, a price list or a support FAQ.
- It changes weekly or less, on a schedule you control.
- Every user may see all of it, or there are only a few access groups, each of which can have its own cache.
- Questions need the whole picture, such as comparing two policies or summarizing across several documents.
- Questions arrive steadily during the hours the assistant is used.
- You want fewer parts to build, monitor and maintain.
Signs RAG Is the Better Choice
- The collection runs to thousands of documents, or keeps growing.
- The content changes daily, as tickets, orders or a live knowledge base do.
- Different users may see different documents. Search services can enforce this at query time; Azure AI Search, for example, supports document-level access control for RAG applications Source [12]. A single CAG cache cannot filter what the model has already read.
- You must show exactly which passage supported each answer. RAG records which passages it sent; with CAG, the model's citation is its own pointer into the full text and needs checking.
- Questions are occasional, so the cache would usually be cold.
When questions need several rounds of searching, retrieval can also be run by an agent that decides what to look for next. Our post on agentic RAG covers when that extra complexity is worth it.
Combining CAG and RAG
Most production systems do not have to pick one. A common hybrid preloads a small, stable core, such as policies and product facts, and searches the larger, changing collection only when a question needs it. Prompt caching also helps inside an ordinary RAG system, by caching the fixed instructions that are sent with every request.
The CAG paper itself ends with this idea: preload a foundation context, and use retrieval for edge cases or highly specific questions Source [1]. The EMNLP 2024 study tested a related method, Self-Route. It sends each question either to RAG or to the full long context, based on the model's own judgment. It kept performance comparable to the long-context approach at a much lower cost Source [8].
A simple version works the same way: answer from the cached core first, and fall back to search when the answer is not there.
Where KAG Fits: RAG vs CAG vs KAG
KAG (knowledge-augmented generation) is a third approach that combines a knowledge graph with vector retrieval. It was proposed in a 2024 paper and applied at Ant Group. The graph lets the model follow relationships, numbers, dates and expert rules, not just matching text. It suits specialist domains with strict logic, but someone has to build and maintain the graph.
Ant Group's paper reports relative F1 improvements of 19.6% and 33.5% over earlier RAG methods on two multi-hop question-answering benchmarks. It also describes using KAG for government-service and health question answering at Ant Group Source [13]. Put simply, RAG finds passages, CAG preloads everything, and KAG reasons over structured facts. KAG is the heaviest of the three to build.
How SensViz Can Help
SensViz builds generative AI applications that work with a client's own data, including RAG search and knowledge assistants. The team chooses the smallest practical architecture that can meet the use case. For a small, stable knowledge base, that can mean no retrieval pipeline at all. Builds come with clear tests, approved data sources, human review where it is needed, and running costs considered before launch.
If you are weighing RAG against CAG for a specific assistant, SensViz can help you compare both on your own documents and questions before you commit. See our generative AI development service, or talk to the SensViz team about your project.
Frequently Asked Questions
Short answers to the questions teams ask once they start comparing the two approaches.
Is CAG Better Than RAG?
CAG is not better than RAG in general; it is better for some knowledge bases. In the original study, CAG scored highest on smaller sets and fell behind keyword RAG on the largest one Source [1]. The dependable way to choose is to run both on your own documents and compare accuracy, response time and monthly cost side by side.
Can You Use CAG With Open-Source Models?
You can use CAG with open-weight models by hosting the model yourself and keeping the KV cache in memory or on disk. The original study did this with Llama 3.1 8B on eight Tesla V100 GPUs with 32 GB each. Its authors released their framework publicly Source [1]. Self-hosting means you manage the cache's storage and resets yourself.
Is Cached Data Shared With Other Customers?
Cached prompts are not shared between customers on the major model APIs. Anthropic isolates caches between organizations, and by workspace on the Claude API Source [9]. OpenAI does not share caches across organizations, stores them as KV tensors rather than prompt text, and keeps them for at most 24 hours Source [2]. Check data-retention terms before caching regulated data.
How Often Does a CAG Cache Need to Be Rebuilt?
A CAG cache needs rebuilding whenever the cached documents change and whenever the cache expires. OpenAI's GPT-5.6 and later models keep cache entries for at least 30 minutes by default, or 12 hours in the Responses API Source [2]. Batching content edits into one scheduled update means the cache is rewritten once, not after every small change.
Choosing between RAG and CAG?
Talk to the SensViz team about your documents, your questions and your traffic, and get a view on which approach fits before you build.
Sources
- [1] arXiv. Chan, Chen, Cheng and Huang: Don't Do RAG: When Cache-Augmented Generation is All You Need for Knowledge Tasks (arXiv 2412.15605, Web Conference 2025) (opens in a new tab).
- [2] OpenAI. OpenAI API documentation: Prompt caching (opens in a new tab).
- [3] Microsoft. Microsoft Learn: Enable semantic caching for LLM APIs in Azure API Management (opens in a new tab).
- [4] arXiv. Lewis et al.: Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (arXiv 2005.11401, NeurIPS 2020) (opens in a new tab).
- [5] arXiv. Hsieh et al.: RULER: What's the Real Context Size of Your Long-Context Language Models? (arXiv 2404.06654, COLM 2024) (opens in a new tab).
- [6] arXiv. Liu et al.: Lost in the Middle: How Language Models Use Long Contexts (arXiv 2307.03172, TACL) (opens in a new tab).
- [7] Anthropic. Claude Platform Docs: Context windows (opens in a new tab).
- [8] arXiv. Li et al.: Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach (arXiv 2407.16833, EMNLP 2024) (opens in a new tab).
- [9] Anthropic. Claude Platform Docs: Prompt caching (opens in a new tab).
- [10] OpenAI. OpenAI Help Center: Understanding and counting tokens (opens in a new tab).
- [11] Anthropic. Anthropic: Introducing Contextual Retrieval (September 2024) (opens in a new tab).
- [12] Microsoft. Microsoft Learn: Document-level access control in Azure AI Search (opens in a new tab).
- [13] arXiv. Liang et al.: KAG: Boosting LLMs in Professional Domains via Knowledge Augmented Generation (arXiv 2409.13731) (opens in a new tab).





