Generative AI & LLMs

Agentic RAG: How It Works, and When the Extra Complexity Is Worth It

An agent that chooses its own searches can answer questions a fixed pipeline cannot. It also adds latency, cost, and new ways to fail quietly.

Written by Tehreem FatimaReviewed by Umaid Asim

Published 26 October 202613 min read

Two columns comparing when agentic RAG earns its cost against when a fixed retrieval pipeline is already enough, with four conditions on each side. — Figure 1. The decision this article is about. Blue marks the conditions where an agent earns its cost. Red marks the conditions where a fixed pipeline already wins.

Most explanations of agentic RAG stop at the idea that the agent decides what to search. That is accurate and not much help. It does not tell you which of several quite different architectures a vendor is proposing, what stops the agent searching indefinitely, or how you would know the system was working once it was live.

This article covers those three. If you are still choosing between a fixed retrieval pipeline and an agent-directed one, our comparison of agentic search and RAG covers that decision first. What follows assumes you are past it.

What Is Agentic RAG?

Agentic RAG places an autonomous agent inside the retrieval pipeline, so the system decides what to search, which tool to use, and whether the evidence it has is sufficient. A 2025 survey describes this as integrating “autonomous agents directly into the RAG pipeline,” which creates “an explicit control layer that guides how the system reasons over external evidence” Source [1].

That control layer is the entire difference, and it is also what separates agentic RAG from plain agentic search: agentic search is the retrieval behavior, agentic RAG is the full architecture built around it. In conventional retrieval-augmented generation the retrieval step is fixed: a query goes in, passages come back, the model writes an answer from them. In agentic RAG an agent sits above that step. It can split the question, choose a different tool, judge whether what came back is enough, and search again.

The commercial problem is that agentic RAG is not one thing. At least six architectures share the label, and they differ in cost, latency and failure behavior. A proposal that says “agentic RAG” without naming which one is not telling you what you are buying.

Six Agentic RAG Architectures, and What Each One Trades

Six agentic RAG architectures, each with what it buys you in blue and what it costs in red: single agent router, multi agent, hierarchical, corrective, adaptive and graph based. — Figure 2. The six agentic RAG architectures described in the survey. Blue marks what each pattern buys you. Red marks what it costs.

The survey organizes these designs by how many agents are involved, how they are coordinated, how much autonomy they hold, and how knowledge is represented Source [1]. Six families are worth knowing by name.

A single agent, or router, chooses among knowledge sources such as a SQL database, a semantic index, web search or a recommendation system. The survey describes a design where “a single agent manages the retrieval, routing, and integration of information” Source [1]. It is the simplest to build and the fastest, and the weakest when a question needs reasoning across several domains at once.

A multi agent design splits retrieval across specialized agents running in parallel, distributing “responsibilities across multiple agents, each optimized for a specific role or data source” Source [1]. It scales through parallelism and pays in coordination overhead.

A hierarchical design puts agents in tiers, with “higher-level agents overseeing and directing lower-level agents” Source [1]. This suits cases where some decisions should govern others rather than compete with them. Reliability improves and latency rises.

A corrective design assesses what came back before trusting it. The survey names five roles: context retrieval, relevance evaluation, query refinement, external retrieval and response synthesis Source [1]. Ask about this pattern when wrong answers are expensive, because checking the evidence is part of the loop rather than bolted on afterwards.

An adaptive design classifies how complex a question is and routes it accordingly, to no retrieval, a single lookup, or a multi-step search, “dynamically adjusting query handling strategies based on the complexity of the incoming query” Source [1]. For a mixed workload this is often the most economical choice, because most questions do not need the expensive path.

A graph based design works across a graph knowledge base alongside ordinary document retrieval, which suits relational questions that passage search handles poorly. These depend heavily on the quality of the underlying knowledge Source [1].

The survey adds a seventh family, agentic document workflows, which extends the pattern past question answering into parsing, state and orchestration across a document process Source [1].

The trade-offs follow one shape. Simpler designs are faster and cheaper and handle less. More agents and more checking buy reliability and breadth, and charge for it in latency, in cost, and in how hard it becomes to work out which part went wrong.

What an Agentic RAG Platform Actually Exposes

The taxonomy above is research framing. One commercial implementation shows which parts of it vendors have built and which controls they hand you.

Microsoft describes agentic retrieval in Azure AI Search as “a multi-query pipeline designed for complex questions posed by users or agents in chat and copilot apps” Source [3]. It can “break down a complex query into smaller, focused subqueries for better coverage across proprietary and external content,” with those subqueries able to carry chat history for context, and it “runs subqueries in parallel,” each one semantically reranked Source [3].

The useful detail is that planning, currently marked preview, is a setting rather than a fixed behavior. At low and medium reasoning effort the system sends the query and conversation history to a language model that generates focused subqueries. At minimal effort, “this step is skipped and queries are issued directly to knowledge sources” Source [3].

So the agentic layer can be turned down. That matters when a minority of your questions need planning and the rest do not. When you evaluate any agentic RAG platform, ask whether the planning step can be reduced or disabled for simple questions, and who decides which questions take the expensive path.

One limit is worth holding onto. The survey is explicit that agentic reasoning does not compensate for poor retrieval Source [1]. An agent searching a badly prepared index more cleverly is still searching a badly prepared index. Our guide on RAG architecture and retrieval quality covers what that preparation actually involves, and it is the same groundwork Fossilite builds when connecting an organization’s own sources so an answer can be traced back and corrected.

The Controls That Keep an Agent Bounded

The freedom that makes agentic RAG useful is the same freedom that makes it fail in ways a fixed pipeline cannot. The survey puts it directly: “Unrestricted autonomy risks excessive tool invocation, redundant reasoning loops, or misaligned objectives” Source [1].

An agentic RAG loop from question to plan, retrieve, evaluate and answer, with a dashed return path for searching again, above a red band showing the four limits a team sets. — Figure 3. The loop runs on its own, shown in blue. The four limits in red are the ones to agree before a build starts.

None of those announce themselves. A redundant reasoning loop looks like a slow response. Excessive tool invocation looks like a larger bill at the end of the month. An agent pursuing the wrong objective still returns a fluent answer.

Four controls belong in the scope before a build starts rather than after.

The first is a ceiling on searches: a maximum number of retrieval steps per question, after which the system stops and reports what it has. Without one, a question the evidence cannot answer becomes an expensive loop instead of an honest refusal.

The second is a budget for time and cost per request, tracked as cost per thousand questions rather than per call. Agentic patterns multiply model calls, so the per-question figure is the only one that reflects the bill.

The third is a defined path when evidence is missing. Saying so and stopping, asking a clarifying question, or handing to a person are all defensible. Continuing to search until something turns up is not.

The fourth is attribution. The survey notes that multi-agent settings make error attribution difficult Source [1], which in practice means logging which agent made which decision on what evidence, from the first version rather than during an incident.

The survey also flags a slower risk: “Persistent memory risks knowledge drift and bias reinforcement” Source [1]. If a design carries memory between sessions, ask what expires it and who reviews what it has accumulated.

How to Test a System That Chooses Its Own Path

Teams underestimate this, and the two best sources on it disagree. The disagreement is worth understanding, because it marks where the judgment actually sits.

The survey argues that judging the final answer is not enough. It calls output-level metrics “insufficient for Agentic RAG” and asks for process-aware evaluation, on the reasoning that a system which chose badly and arrived somewhere acceptable has not really worked Source [1].

Anthropic’s guidance on evaluating agents pushes the other way. It recognizes “a common instinct to check that agents followed very specific steps like a sequence of tool calls in the right order,” then concludes: “We’ve found this approach too rigid and results in overly brittle tests.” Its position is that “so as not to unnecessarily punish creativity, it’s often better to grade what the agent produced, not the path it took” Source [2]. The supporting example is an agent that solved a benchmark problem by finding a policy loophole, failed the evaluation as written, and had in fact produced a better result for the user Source [2].

Both are right about different things. Grading a fixed sequence of steps is brittle, and a system that finds a better route should not be penalized for it. But an agentic RAG system that reaches a reasonable answer through twelve searches when two would do has a cost problem the final answer will never show.

The way to hold both is to separate what you grade from what you monitor.

Grade the outcome. Anthropic draws a useful line between the transcript, “the complete record of a trial, including outputs, tool calls, reasoning, intermediate results,” and the outcome, “the final state in the environment at the end of the trial” Source [2]. For retrieval, the outcome questions are whether the answer is supported by what was retrieved and whether the right source was found at all.

Monitor the path, but as an operational signal rather than a pass mark. Track searches per question, tool calls per question, and latency across the whole request including retries. Use those to catch drift and cost, not to fail a run that took an unexpected route.

Then test the cases where the system should stop: questions the evidence cannot answer, conflicting versions of a document, and requests that cross a permission boundary. A fixed pipeline fails these visibly. An agent can keep searching and produce something confident instead, which is the more expensive failure. Our post on LLM testing covers the wider testing picture these checks sit inside.

What Agentic RAG Costs

It costs more than a single-query pipeline, and how much more depends on your question mix, so budget from that rather than from a vendor’s example.

Latency is the first cost. Microsoft states it plainly: “Agentic retrieval adds latency compared to a single-query pipeline, but it handles query complexity that a single query can’t” Source [3]. Running subqueries in parallel limits the damage, but planning and reranking still take time a direct lookup does not.

Token spend is the second, and this is where agentic patterns differ most from conventional RAG. Billing moves from per query to per token across both the search service and the model doing the planning. Microsoft’s worked example puts 2,000 retrievals at roughly 3.30 dollars for search reranking, 0.60 for planning input tokens and 0.42 for planning output tokens, about 4.32 dollars in total, and notes that this excludes answer synthesis and that costs could be lower Source [3]. Read that as the shape of a bill rather than a quote, since the number moves with how many sources you connect and how much planning you ask for.

The levers are the expected ones. Microsoft suggests lowering reasoning effort to reduce model usage during query planning and expansion, reducing the number of knowledge sources, and organizing content so fewer documents need searching Source [3]. The survey makes the same point structurally, noting that iterative retrieval and multi-agent collaboration raise both latency and resource use Source [1].

The figure to ask any provider for is cost per thousand questions at your expected mix of simple and complex queries. A per-call number hides the thing agentic patterns change.

When Agentic RAG Is Not Worth It

Four situations where a conventional pipeline is the better answer.

When the evidence path is already known. If every question of a given type is answered from the same two sources in the same order, planning adds cost and risk without adding capability. A well-configured retriever returning several passages in one step covers more cases than teams expect, and our guide on building or buying RAG covers that decision in more depth.

When retrieval quality is the real problem. The survey is direct that agentic reasoning does not compensate for poor retrieval Source [1]. If content is stale, badly chunked or inconsistently permissioned, fixing that returns more than adding an agent above it.

When latency is part of the product. Anything answering inside a live conversation has a budget that planning and repeated searches will consume.

When nobody owns the evaluation. These systems need someone watching searches per question, cost per thousand questions, and the cases where the system should stop. Without an owner, the failure modes above stay invisible until the bill or a complaint surfaces them.

The survey’s own assessment of the field is worth repeating: most of these systems have been evaluated only in narrow domains Source [1]. Test on your own questions rather than trusting a benchmark.

How SensViz Can Help

SensViz builds retrieval and agentic AI systems, including the evaluation, access controls and monitoring around them. Our GrantMatch work combines language models, recommendation logic and vector search to surface relevant grant opportunities, which is a useful reference point for this article’s main argument: retrieval systems often need more than one mechanism, and the engineering question is how those mechanisms are chosen and controlled rather than which single one is best.

If you are deciding whether a fixed pipeline or an agent-directed one fits the questions you need answered, that is a conversation worth having before the architecture is chosen. Talk to us about your retrieval project, and bring a handful of the real questions the system would need to handle.

Discuss your retrieval project

Frequently Asked Questions

Four questions that come up after the architecture conversation rather than during it.

Does agentic RAG reduce hallucinations?

Not on its own. A corrective design does check retrieved evidence before trusting it, which helps. But the survey is explicit that agentic reasoning does not compensate for poor retrieval Source [1]. If the content is stale or badly chunked, an agent searches the same weak index more times and can produce a confident answer from thin evidence.

Is agentic RAG the same as a multi agent system?

No. Multi agent is one of six agentic RAG families, where retrieval is split across specialized agents running in parallel Source [1]. A single agent router is also agentic RAG and involves one agent. What makes a system agentic is that something decides what to retrieve next, not how many agents are involved.

Can we start with conventional RAG and add agents later?

Usually yes, and it is often the cheaper order. The retrieval layer, content preparation and permissions are the same work either way, and the survey notes agentic reasoning does not compensate for poor retrieval Source [1]. Build and measure that layer first, then add planning where the question mix actually needs it.

Which agentic RAG architecture should we start with?

Not from the taxonomy alone. Sample a few hundred real questions and sort them by how many sources and steps each needs. If most resolve in one lookup, adaptive routing fits. If wrong answers are expensive, corrective fits. Without that sample you are choosing an architecture for a workload you have not measured.