Retrieval-augmented generation, or RAG, has become the default architecture for building LLM applications that need to answer from real, current, or private information. The idea is straightforward: before the language model generates an answer, a retrieval step finds relevant passages from your own knowledge base, and the model is asked to answer based on those passages. The result is more accurate, more current, and less prone to the confident hallucinations that plague pure generation.
What started as a research pattern is now a production discipline, and the good news for practitioners is that you no longer need a proprietary stack to build it. A modern RAG system can be assembled almost entirely from open source components: an embedding model, a vector database, an LLM, and a thin orchestration layer. This tutorial walks through the complete architecture, from indexing and storage to query optimization and generation, with concrete decisions and trade-offs at every step.
Why RAG Matters More Than Ever
Foundation models are impressive, but they have a fundamental limitation: they only know what was in their training data, frozen at a point in time. Ask a model about your company's latest product, a document uploaded yesterday, or a niche technical topic, and it will either guess or refuse. RAG solves this by connecting the model to an external knowledge source at query time.
The practical consequences are significant. RAG systems can cite their sources, so users can verify answers. They can be updated by simply adding documents, without retraining the model. And they can access proprietary data that was never part of the model's training set. These properties make RAG the foundation for customer support bots, internal knowledge assistants, document analysis tools, and research copilots.
The architecture is also remarkably flexible. The same core pattern works whether your knowledge base is a few hundred support articles or millions of legal documents, and whether your model is a frontier API or a small open source model running on your own hardware.
Core Architecture: The RAG Loop
Every RAG system follows the same loop, even if the implementations differ.
Ingest: documents are loaded, cleaned, and split into chunks of a manageable size.
Embed: each chunk is converted into a vector, a numeric representation that captures its meaning.
Index: the vectors are stored in a vector database that supports fast similarity search.
Retrieve: at query time, the user's question is embedded with the same model, and the database returns the most similar chunks.
Generate: the retrieved chunks are inserted into a prompt as context, and the LLM produces an answer grounded in that context.
The details of each step determine the quality of the system. A RAG pipeline is only as strong as its weakest link, and most failures trace back to poor chunking, weak embeddings, or a retrieval step that pulls the wrong passages.
Setting Up the Indexing Pipeline
Indexing is the foundation. Garbage in, garbage out applies to RAG more than almost any other architecture, because retrieval can only find what indexing preserved.
Start with cleaning. Remove boilerplate, navigation, and duplicate content from documents before they enter the pipeline. Raw web pages and exported PDFs are full of noise that will be embedded as if it were signal.
Chunking is the first major decision. The goal is chunks that are small enough to be semantically focused and large enough to contain a complete idea. A common starting point is 300 to 500 tokens per chunk, with a small overlap between neighboring chunks so that no idea is split across a boundary. Tables, code blocks, and lists often deserve special handling; they are hard to split sensibly and are best kept as atomic units.
Metadata is underrated. Store the source document, section heading, and position for every chunk. This metadata powers citation, filtering, and re-ranking later, and it makes the system explainable instead of a black box.
Embedding Models and the Vector Database
The embedding model defines what "similar" means in your system, so the choice deserves care. Open source options have improved dramatically and now cover most use cases.
General-purpose embeddings are a safe default. They work well across domains and languages, and they are easy to serve with open source tooling. If your corpus is broad and your queries are general, start here.
Domain-specific embeddings are worth considering when your content has specialized vocabulary. Legal, medical, and technical corpora benefit from models trained on those domains, because general embeddings may not capture the meaning of niche terms.
Multilingual embeddings matter if your knowledge base spans languages. A good multilingual model will embed documents and queries from different languages into a shared space, so a French query can retrieve an English document.
Whatever you choose, the critical rule is consistency: use the same embedding model for both documents and queries. Mixing models breaks the shared vector space and silently destroys retrieval quality.
The vector database stores embeddings and answers similarity queries. Open source options range from dedicated vector databases to extensions of familiar databases, and the right choice depends on your scale and operational preferences.
For small to medium collections, a simple index running in your application or a lightweight server is often enough. You get fast similarity search, easy setup, and no extra infrastructure.
For larger collections, a dedicated vector database or a database with strong vector support is the better call. These systems handle millions of vectors, support metadata filtering, and offer production features like replication and backups.
One decision matters more than the specific product: the index type. Approximate nearest neighbor indexes, the standard choice for large collections, trade a tiny amount of accuracy for massive speed gains. For most applications this trade is the right one, but it is worth understanding that your results will be approximately, not exactly, the closest matches.
Optimizing Queries: Retrieval and Re-ranking
Retrieval quality is the difference between a RAG system that feels smart and one that feels broken. The raw similarity search is only the first stage.
Hybrid search is a powerful upgrade. Combine vector similarity with keyword matching, so that exact terms, product names, and IDs are found even when embeddings miss them. Many open source systems support hybrid search natively, and it fixes a common failure mode where the vector search returns semantically related but factually wrong chunks.
Re-ranking is the next stage. After retrieving a generous set of candidates, a re-ranker, often a small cross-encoder model, scores each candidate against the query more carefully. The re-ranker looks at the actual text pair rather than comparing pre-computed vectors, so its scores are more accurate. You take the top candidates from the vector search, re-rank them, and keep only the best few for the prompt.
Query rewriting helps when the user's question is vague. A preprocessing LLM step can expand a short query into a fuller expression, extract key entities, or break a multi-part question into several retrieval queries. The retrieved results are then combined for generation.
Prompt Construction and Generation
The generation stage decides how the retrieved context is turned into an answer. The prompt structure has a huge impact on output quality.
Give the model clear instructions about its role and constraints. Tell it to answer based only on the provided context, to say when the context does not contain the answer, and to cite which passages support each claim.
Keep the context focused. More context is not automatically better; too many passages drown the model in noise and increase cost. Five to ten well-chosen chunks usually outperform twenty mediocre ones. Re-ranking pays off here by ensuring the prompt contains the best evidence.
Watch the context window. Long documents and large chunk sets can overflow the model's context limit. Truncate responsibly, keep the most relevant chunks, and design your chunking so that each chunk stands alone reasonably well.
Grounding RAG in a Production Backend
A RAG system does not live in isolation. It sits inside a backend that manages users, documents, tasks, and observability, and the architecture of that backend affects how well RAG performs.
Use a task queue for heavy work. Indexing large document sets and running generation calls can take time. A queue decouples ingestion from serving, so users are not blocked while a document is being processed, and failures can be retried cleanly.
Store document state explicitly. Track which documents are indexed, which versions are current, and which embeddings need refresh. Without this bookkeeping, stale content quietly poisons retrieval results.
Log every retrieval. Record the query, the chunks retrieved, the scores, and the final answer for every request. This log is the raw material for evaluating and improving the system, and it is the only way to diagnose why a specific answer went wrong.
Evaluating a RAG System
Evaluation is where most RAG projects stall, because quality is hard to measure. Still, a basic evaluation loop is essential.
Build a test set of real queries with known good answers. Twenty to fifty representative questions are enough to start. For each query, check two things: retrieval quality, whether the right passages were found, and generation quality, whether the final answer is correct and grounded.
Track retrieval metrics like recall at k, which measures whether the correct passage appears in the top results. Track generation quality with a mix of automated checks and human review. A small, honest test set reviewed by a human is worth more than an elaborate automated benchmark.
Run the evaluation loop every time you change the pipeline. Embedding models, chunk sizes, re-rankers, and prompt templates all shift behavior, and the only way to know whether a change helped is to measure it.
Common Pitfalls and Fixes
RAG projects fail in predictable ways. Here are the patterns worth knowing.
Retrieving the wrong chunks: usually a chunking or embedding problem. Improve chunk boundaries, try a domain embedding, or add hybrid search.
Answers that ignore the context: the prompt is too weak, or the context is too long. Tighten the instructions, shorten the context, and re-rank more aggressively.
Stale or missing information: an indexing pipeline that skipped documents or never updated them. Fix the ingestion bookkeeping and re-index on change.
Latency that kills the product: too many retrieval stages or an oversized context. Cache frequent queries, shrink the candidate set, and consider smaller models for simple answers.
The common thread is that most failures are architectural, not magical. Systematic logging and a real evaluation set will surface them quickly.
FAQ
Do I need a proprietary LLM for RAG? No. Open source models work well, especially with good retrieval. The model quality matters less than the retrieval quality, because the model is grounded in provided context.
What is the best chunk size? There is no universal answer, but 300 to 500 tokens with overlap is a strong starting point. Test 200, 400, and 800 tokens on your own corpus and measure retrieval quality.
How many chunks should go into the prompt? Five to ten well-chosen chunks usually balance quality and cost. Let re-ranking pick the best, rather than stuffing the prompt with everything.
Can RAG replace fine-tuning? For most knowledge-access use cases, yes. Fine-tuning changes the model's behavior and style; RAG provides facts. Many systems use both, with RAG for knowledge and fine-tuning for tone.
How do I handle multi-language documents? Use a multilingual embedding model and keep language metadata on every chunk. The retrieval step then works across languages, and you can translate or summarize at generation time.
RAG is the architecture that makes LLMs useful in the real world, and open source components have made it accessible to teams of every size. The system is not complicated in concept, but it rewards discipline: clean indexing, consistent embeddings, honest retrieval evaluation, and careful prompt construction. Build the loop end to end with a small corpus first, measure it, and then scale. The teams that ship working RAG systems are not the ones with the fanciest models; they are the ones that treat retrieval quality as the product.
