RAG Architecture and Retrieval Quality: A Practical Guide

Retrieval-augmented generation (RAG) depends on both the evidence retrieved and how the model uses it. What a RAG pipeline actually means in practice is the sequence of steps that turns a document collection into grounded, sourced answers: ingesting and chunking content, embedding it, indexing it in a vector database, retrieving and reranking candidates, then assembling context for the model. Failures can arise in retrieval, context assembly, or generation.

That's the part most explainers skip. Plenty of content will tell you what RAG is. Fewer will tell you how to distinguish missing retrieval evidence from a generation error, or why the embedding model on top of a leaderboard isn't automatically the right one for your documents. This guide is about the architecture and, more specifically, about how to know whether the retrieval layer is actually doing its job.

If you're still deciding whether RAG is the right approach at all, our Generative AI & LLM Solutions page covers that groundwork. This guide assumes you're past that question and into the harder one: how do you build a RAG pipeline that's actually reliable.

What a RAG pipeline actually is

A typical vector-based RAG pipeline has two paths. The indexing path prepares the knowledge base and runs again as content is added, changed, or deleted. The query path runs when someone asks a question. Other RAG designs can retrieve through keyword search, SQL, or graph queries without a vector database. [1]

• Ingestion and parsing: pulling source content in from documents, databases, or APIs and extracting usable text
• Chunking: splitting that content into retrievable pieces
• Embedding: converting chunks into vectors that capture semantic meaning
• Indexing: storing those vectors in a database built for similarity search
• Query preparation and retrieval: embedding the question with a compatible query encoder for dense search, applying access restrictions, and retrieving candidates; hybrid search also runs a keyword query
• Optional reranking: reordering retrieved candidates with a more detailed relevance model
• Context assembly: deciding what actually gets handed to the model, and in what form
• Generation: the model producing an answer grounded in that context

Each stage has failure modes that compound. A chunking decision made on day one shapes what's retrievable months later. Good retrieval gives the model useful evidence, but it does not guarantee a correct answer. Check retrieval, context assembly, and generation separately before choosing a fix.

Chunking: the decision that shapes everything downstream

Chunking is how a document gets broken into pieces small enough to retrieve individually but large enough to still mean something on their own. Splitting a table from its heading or a rule from its exception can remove information needed to answer correctly. Compare strategies on the same documents and questions; no single method wins across all workloads. [2]

The common strategies, and where each one earns its place:

Fixed-size chunking splits text into equal token windows (commonly around 512 tokens), regardless of where sentences or ideas end. It's simple to implement and works reasonably well on homogeneous, short-form content like FAQs or support tickets, where each chunk is naturally self-contained. It breaks down on longer, structured documents, where it can split a table, a clause, or an argument in half.

Recursive chunking splits along a hierarchy of separators (paragraphs first, then sentences, then words), falling back only when a chunk is still too large. It respects document structure better than fixed-size splitting and is a reasonable default for mixed content.

Semantic chunking groups text by meaning rather than position, using embedding similarity to decide where one idea ends and another begins. It adds preprocessing work and does not consistently outperform simpler methods. Embedding many sentences also does not necessarily mean making one API call per sentence, because requests can be batched. Test whether the quality gain justifies the extra work. [2]

Proposition chunking breaks content into atomic, self-contained statements, closer to individual facts than paragraphs. It can suit narrow, factoid-style questions, but generating propositions usually requires an LLM pass over the source content. Check that the extracted statements retain qualifications and source references; extraction can introduce errors.

Hierarchical (parent-child) chunking retrieves on small, precise child chunks but returns the larger parent chunk to the model for context. This is a practical middle ground: retrieval stays precise, but the model isn't working from a sentence stripped of everything around it.

There's no universally correct choice. The right one depends on document structure, query type, and how much you're willing to spend on preprocessing. What matters is picking deliberately and testing the decision against your own content, not defaulting to whatever a tutorial used.

Embeddings: past the leaderboard

An embedding model turns a chunk of text into a vector: a list of numbers that represents features useful for comparing meaning. Dense retrieval searches for nearby vectors. Use compatible document and query encoders and the distance metric expected by the model; not every RAG retriever uses vectors.

It's tempting to pick whichever model tops the current MTEB retrieval leaderboard and move on. That's a reasonable starting point, not a finishing one. Leaderboard performance is measured on public benchmark datasets that may look nothing like your documents. A model tuned for general web text can underperform a smaller, more specialized model on dense technical or legal content. The practical considerations that matter alongside raw benchmark rank:

• Dimensionality and cost. Higher-dimensional embeddings can capture more nuance but cost more to store and search at scale. A 1,024-dimension model isn't automatically better than a 768-dimension one for your use case.
• Context length. If your chunks are long, the embedding model needs to actually support that input length without silently truncating it.
• Hosted API vs. self-hosted. API-based embedding models are simple to integrate but add a per-call cost and a network dependency. Self-hosted open-weight models (BGE and E5-family models are common choices) remove that dependency and matter more when data residency or per-query cost at high volume is a real constraint.
• Domain fit. A model evaluated on your own representative queries, not a public benchmark, is the only evaluation that actually tells you whether it'll work for your case.

This is the same principle SensViz applies across generative AI work generally: compare candidates against representative data and the real requirement, rather than choosing by reputation. An embedding model is a component to test, not a foregone conclusion.

Choosing a vector database

The vector database is where embedded chunks live and where similarity search actually happens. Picking the right vector database for RAG depends far more on your scale, existing infrastructure, and filtering needs than on which open source option is most talked about.

pgvector is a PostgreSQL extension that supports vector similarity search alongside SQL filtering and joins. It is worth evaluating if your team already operates Postgres, including a managed service that supports the extension. There is no universal ceiling of a few million vectors: performance depends on indexing, memory, dimensions, filtering, concurrency, and the recall target. [3]

Elasticsearch and OpenSearch support vector search as well as keyword search. If the team already operates one, adding retrieval there can reduce the work of introducing another system. Benchmark the relevant configuration rather than assuming a dedicated vector database will always be faster. [4, 5]

Pinecone offers managed vector search, reducing the database infrastructure your team operates. You still need to design indexes, access controls, and monitoring, and test latency and cost at expected query volume. Managed deployment does not guarantee constant latency or a lower or higher total cost than every self-hosted alternative. [6]

Qdrant supports vector search with metadata filtering and offers both self-hosted and managed cloud deployment. Test filtered-query recall and latency with your actual access patterns. The operational work depends on whether you run the deployment yourself or use the managed service. [7]

Weaviate supports keyword, vector, and hybrid search, with options for text and multimodal applications. It is not limited to multimodal retrieval. Choose it based on the search features, model integrations, and deployment requirements your application needs. [8]

Chroma supports local development and also offers Chroma Cloud for managed deployment. It should not be dismissed as prototype-only. Evaluate the chosen deployment against your requirements for concurrency, data management, access controls, reliability, and cost. [9]

A practical decision framework: test your existing Postgres or search platform first if it supports the required retrieval method. Compare alternatives when measured recall, latency, filtering, or operational requirements are not met. Use the same data, queries, embedding model, and quality target for each comparison; document count alone is not a useful migration threshold.

Migrating between vector databases can require exporting records, rebuilding indexes, and changing query and filtering logic. Re-embedding is needed if you change to an incompatible embedding model or representation, not automatically whenever you change databases. Validate dimensions, distance metrics, metadata, permissions, and retrieval results before switching.

Retrieval: why dense-only search isn't enough

Pure vector (dense) search, sometimes called semantic search, finds chunks that are semantically similar to a query, which works well for conceptual questions and poorly for anything requiring an exact match: product codes, legal citations, specific names, or technical identifiers a dense embedding can blur together.

Hybrid search combines dense vector search with keyword search, commonly BM25. The results can be merged using reciprocal rank fusion, which combines ranks rather than raw scores, or other fusion methods. Compare dense, keyword, and hybrid retrieval on your query set, especially when it mixes conceptual questions with exact identifiers. Hybrid search is an option to test, not a requirement for every RAG system. [8]

Metadata filtering restricts candidates by fields such as date, category, source, or access rights. Filter behavior varies across search engines and configurations, so measure both relevance and latency. Enforce authorization before any restricted content reaches the model or user; do not rely on a prompt to hide it.

Reranking: the underused precision step

Retrieval typically pulls back a broader set of candidates (20 is a common starting point) that are then narrowed down before reaching the model. A reranker takes that candidate set and reorders it using a more precise (and more expensive) relevance signal, typically a cross-encoder model that reads the query and each candidate document together, rather than comparing precomputed vectors the way initial retrieval does.

Reranking can improve retrieval precision, but its quality gain and latency cost depend on the workload. Test it once the initial retrieval stage is otherwise solid. The typical pattern is to retrieve a broader candidate set, then rerank down to the passages that fit the model's context budget. Tune the number of candidates against your own queries rather than treating a fixed count as a requirement.

Context assembly: what the model actually sees matters

Retrieval and reranking decide what could be included. Context assembly decides what actually gets sent to the model, and how. This stage is where hierarchical chunking pays off (retrieving on precise chunks, but assembling the fuller parent context around them), where source references get attached so an answer can point back to what it came from, and where ordering and formatting choices affect how well the model actually uses what it's given rather than skimming past it.

Access controls must already have restricted the candidate set. Recheck authorization before context assembly where needed, including cached results. Treat retrieved text as untrusted data, not instructions: source documents can contain prompt injection. Keep system instructions separate and test whether malicious passages can alter the answer or expose information.

Evaluating retrieval quality: the part most guides skip

This is the actual promise of this guide: not just describing the pipeline, but how to know whether it's working.

A fluent answer can be unsupported because retrieval missed the evidence or because generation ignored or misrepresented it. Even a faithful answer can be wrong if its source is outdated or incorrect. Inspect the source, the retrieved passages, the assembled context, and the answer rather than attributing every failure to retrieval.

The metrics that matter, roughly grouped by what they measure:

On the retrieval side:

• Context precision: whether relevant passages rank ahead of irrelevant ones; the exact calculation depends on the chosen metric implementation
• Context recall: how much of the required reference information is supported by the retrieved context; it normally needs reference answers or relevance labels
• Ranking quality: mean reciprocal rank (MRR) measures how early the first relevant result appears; normalized discounted cumulative gain (NDCG) evaluates ordering using relevance grades

On the generation side:

• Faithfulness: whether answer claims are supported by the retrieved context; this does not establish that the source itself is correct or current
• Answer relevance: whether the response addresses the question. Also check factual correctness against trusted references and whether each citation supports its associated claim.

Use representative queries with reviewed reference answers or relevant source passages. Ragas provides metrics including context precision, context recall, faithfulness, and answer relevancy, but metric variants have different input requirements. If an LLM grades responses, record the model and rubric and compare its judgments with human review. Automated scores are useful signals, not ground truth. [10]

Start with a manageable set of representative and deliberately difficult questions, then expand it as you discover failure modes. Include unanswerable questions, exact identifiers, outdated or conflicting sources, and permission-restricted documents. Keep a held-out set for validation so changes are not judged only on questions used for tuning.

The failure pattern worth watching for specifically: retrieval quality degrading slowly and quietly as a document collection grows or drifts, long before anyone notices from the outside. A system that scored well at launch on a small, curated collection can degrade months later purely because the underlying content changed and nobody re-ran the evaluation. Retrieval quality isn't a one-time check: it's something to monitor on an ongoing basis, the same way you'd monitor latency or error rates.

One pattern this guide doesn't cover in depth: multi-hop retrieval, where answering a question requires chaining multiple retrieval steps together because no single chunk contains the full answer (for example, finding which document is relevant before retrieving a specific fact from it). It's a genuine extension to the architecture above, not a replacement for it, worth a dedicated look once the single-hop pipeline described here is solid.

Where RAG pipelines actually break

When an answer fails, trace it back through the pipeline. Was the source present and parsed correctly? Did a chunk retain the required qualification? Was it retrieved, kept during reranking, and included in the prompt? If the evidence reached the model but the answer was still wrong, inspect generation instructions and model behavior. Fix the earliest demonstrated failure rather than assuming ingestion or the prompt is always responsible.

A realistic latency and cost budget

Measure each query-time stage separately: query preparation, retrieval, optional reranking, context assembly, and generation. Generation can dominate response time, but the bottleneck varies with model, output length, candidate count, network conditions, and load. Track time to first token separately from time to the complete answer.

Set latency targets around the user task, then test median and tail latency, such as p95, under expected load. Streaming can reduce perceived waiting time without reducing total completion time. Cache only when freshness and permissions allow, and parallelize independent work where appropriate. Track indexing and embedding costs alongside query-time search, reranking, and generation costs.

RAG vs. fine-tuning vs. prompt engineering, briefly

This guide is about architecture, not the full comparison between the three: our pillar FAQ covers that question directly. The short version: prompt engineering (better instructions and examples in the prompt itself) is usually the first and cheapest lever to try. RAG is usually the right next move when a system needs access to information that's private or changes over time. Fine-tuning solves a different problem again (teaching a model a consistent behavior, format, or style). None of the three are mutually exclusive; some systems eventually need all three together.

What this looks like applied

SensViz has built retrieval-backed systems into real products rather than demonstrations: GrantMatch, for one, combines language models, recommendation logic, and vector search to match users with relevant funding opportunities based on their profile. The architecture decisions above (chunking approach, retrieval method, evaluation discipline) are exactly the kind of choices that separate a RAG system that works in a demo from one that holds up in front of real users and real, messy documents.

If you're scoping a RAG project and want the retrieval architecture reviewed against your actual data and use case rather than a generic template, Get in touch.

Frequently asked questions

What's the difference between chunking and embedding in a RAG pipeline?

Chunking splits parsed source content into retrievable pieces. Embedding converts those pieces into vectors for dense retrieval. The query also needs a compatible vector representation. Chunking affects which information stays together; the embedding model affects how similarity is represented.

Do I need a reranker, or is retrieval alone enough?

Retrieval alone is often enough for simple, low-stakes use cases with a small, clean document set. Reranking earns its cost once queries get more varied, the document collection grows, or answer accuracy matters enough that the latency cost is worth it. It's an addition to tune in once the basics are solid, not a required first step.

How often should a RAG pipeline's evaluation be re-run?

Whenever the underlying document collection changes meaningfully, whenever the embedding or generation model changes, and on a regular schedule regardless, since retrieval quality can degrade gradually even when nothing obvious has changed. Treat it as ongoing monitoring, not a one-time launch check.

Does RAG need a dedicated vector database?

No. RAG can retrieve through keyword search, SQL, graph queries, or vector search, depending on the task. If vector search is useful, options include a dedicated database, pgvector, or an existing search engine with vector support. Choose by measured retrieval quality, latency, filtering, and operational needs, not a fixed number of documents.

Is a bigger, more expensive embedding model always better?

No. Leaderboard rank on public benchmarks doesn't guarantee the best fit for a specific domain, and larger embeddings cost more to store and search at scale. The right model is the one that performs best on your own representative content and queries, evaluated directly rather than assumed from a general benchmark.

Sources

Primary documentation and research supporting the architecture, product capabilities, and evaluation guidance. GrantMatch is a SensViz product example; no quantitative project outcomes are claimed.

• [1] Microsoft Azure Architecture Center. Design and develop a RAG solution. (opens in a new tab)
• [2] Qu et al. Is Semantic Chunking Worth the Computational Cost? (opens in a new tab)
• [3] pgvector. Official README, indexing, filtering and scaling guidance. (opens in a new tab)
• [4] Elastic. kNN search in Elasticsearch. (opens in a new tab)
• [5] OpenSearch. Vector search documentation. (opens in a new tab)
• [6] Pinecone. Official documentation, search and operational guidance. (opens in a new tab)
• [7] Qdrant. Managed Cloud documentation. (opens in a new tab)
• [8] Weaviate. Hybrid search documentation. (opens in a new tab)
• [9] Chroma. Chroma Cloud documentation. (opens in a new tab)
• [10] Ragas. List of available metrics. (opens in a new tab)