A reliable retrieval-augmented generation (RAG) application is more than a vector database connected to an LLM. This guide explains the complete RAG application architecture—from document ingestion and chunking to retrieval, reranking, citations, evaluation, and failure handling—so you can build an AI workflow that is easier to test, operate, and update.
Overview
Retrieval-augmented generation combines search with language generation. Instead of asking a model to answer entirely from its trained knowledge or a large prompt, the application retrieves relevant content from a controlled knowledge source and places that content in the model’s context. The model then uses the retrieved material to produce an answer.
A typical RAG application architecture has six logical layers:
- Source systems: documents, databases, websites, tickets, code repositories, or internal records.
- Ingestion and normalization: extraction, cleaning, metadata assignment, and access-control handling.
- Indexing: chunking, embedding generation, and storage in a searchable index.
- Retrieval: query processing, vector search, keyword search, filtering, and optional reranking.
- Generation: prompt assembly, model inference, structured output, and citations.
- Evaluation and operations: quality checks, observability, cost controls, security, and feedback loops.
The most important architectural principle is separation of concerns. Retrieval should find evidence; the generation step should interpret that evidence; application code should enforce permissions, output contracts, and business rules. Keeping these responsibilities distinct makes an LLM app easier to debug than a single prompt that attempts to perform every task.
Step-by-step workflow
1. Define the answer contract
Begin with the questions the system must answer and the questions it must decline. Specify the intended audience, acceptable sources, freshness requirements, citation format, and behavior when evidence is missing. For example, a support assistant might be required to answer only from approved product documentation, cite the relevant document title, and say that it cannot verify an answer when retrieval returns insufficient evidence.
This contract becomes the foundation for your prompt, evaluation set, and monitoring rules. It also prevents a common failure mode: optimizing for fluent answers before deciding what counts as a correct answer.
2. Ingest and normalize source content
Collect source material through a repeatable ingestion process rather than manually uploading files into an index. Extract text from each source, preserve useful headings and table context, remove navigation or boilerplate where appropriate, and record metadata such as title, URL, author, publication date, document type, product area, and access scope.
Normalization should be deterministic where possible. If the same source is processed twice, the pipeline should produce a stable document identifier and avoid creating uncontrolled duplicates. Keep the original source reference alongside the processed text so that the application can show users where an answer came from.
3. Choose a chunking strategy
Chunking divides documents into passages that can be retrieved and placed into a model context. There is no universal chunk size that works for every corpus. Short chunks may improve precision but lose context; long chunks preserve context but can dilute the relevant passage and consume more tokens.
Start with structural boundaries such as headings, paragraphs, list groups, and table sections. Add overlap only when a concept regularly crosses boundaries. For technical documentation, a chunk may need the parent heading and product version attached as metadata so that a retrieved code example is understandable on its own.
Store both the chunk text and its provenance. A useful record includes a document ID, chunk ID, source location, heading path, timestamp, permissions, and any version information needed to determine whether the content is current.
4. Generate embeddings and build the index
An embedding converts text into a numerical representation that supports semantic similarity search. Generate embeddings for the indexed chunks, then store them with their metadata in a vector-capable search system. Many production workflows also retain a keyword or full-text index because exact terms, error codes, product names, and identifiers may not be handled well by semantic search alone.
Consider a hybrid retrieval design when users search for both concepts and precise strings. The application can combine semantic results with lexical matches, apply metadata filters, and remove documents the current user is not authorized to see. Permissions should be enforced before content reaches the model context, not merely described in a system prompt.
5. Process the user query
Query processing can include normalization, language detection, query classification, and rewriting. A conversational question such as “What about the annual plan?” may need earlier conversation context to become a searchable query. Query rewriting should be observable and reversible: log the original request and the retrieval query so developers can identify whether failures originated in interpretation or search.
Do not automatically rewrite every query without testing. A rewrite can remove an important identifier or introduce an incorrect assumption. For some workflows, sending the original query and a carefully generated alternative through separate retrieval paths is safer than replacing the original.
6. Retrieve, filter, and rerank
Retrieve more candidates than you plan to show the model, then reduce the set using relevance and policy rules. Metadata filters should handle requirements such as document status, tenant, product version, region, or user permissions. A reranker can compare the query with candidate passages in more detail and improve ordering before prompt assembly.
Set a minimum evidence threshold. If the best available passages are weak, contradictory, or outside the permitted scope, the application should return a controlled “insufficient information” response rather than forcing generation. Retrieval scores are useful signals, but they should be validated against labeled examples instead of treated as universal truth.
7. Assemble the prompt and generate the answer
The generation prompt should clearly separate instructions, user input, and retrieved evidence. Tell the model what the evidence represents, how to handle conflicts, when to decline, and how to cite sources. A compact system prompt is often easier to test than a long set of overlapping rules.
Use structured output when downstream code needs predictable fields such as an answer, citations, confidence category, or follow-up question. Validate the response against a schema and handle invalid output explicitly. For prompt design and regression practices, see the prompt testing frameworks guide and the LLM prompt testing framework.
8. Return citations and record the trace
Each citation should map to a retrieved chunk and a human-readable source location. Avoid presenting a document as support if the cited passage does not actually substantiate the claim. Store a trace containing the query, retrieved IDs, filters, prompt version, model configuration, output validation result, latency, and token usage, subject to your privacy requirements.
This trace connects user feedback to a specific stage of the pipeline. Without it, “the answer was wrong” is difficult to turn into an actionable engineering change.
Tools and handoffs
A practical RAG workflow usually involves several handoffs rather than one all-purpose tool:
- Connectors and parsers acquire files, pages, records, and structured data.
- Transformation code cleans text, preserves structure, assigns metadata, and detects changes.
- Embedding services represent documents and queries for semantic search.
- Search infrastructure supports vector, keyword, hybrid, and metadata-filtered retrieval.
- Reranking and query services improve candidate ordering for difficult questions.
- LLM orchestration code builds prompts, calls models, validates outputs, and manages retries.
- Evaluation and observability tools capture traces, compare versions, and expose regressions.
Keep these interfaces explicit. For example, define a retrieval result object with fields for chunk text, source ID, score, metadata, and access decision. Define a generation input with the user question, conversation state, retrieved evidence, and prompt version. Clear contracts make it possible to replace a model, search backend, or parser without rewriting the entire application.
Context limits, latency, and token costs also shape architecture. Review the context window comparison, AI latency guide, and token usage and cost guide when deciding how many passages to retrieve, whether to stream responses, and where to cache repeated work.
Quality checks
Evaluate retrieval and generation separately before measuring the complete user experience. For retrieval, check whether the correct source appears in the candidate set, whether filtering removes unauthorized content, and whether the top results contain enough context. For generation, check factual support, citation accuracy, completeness, refusal behavior, formatting, and consistency across repeated runs.
Build a small, representative evaluation set with straightforward questions, multi-step questions, ambiguous requests, outdated-content cases, no-answer cases, and adversarial inputs. Keep expected evidence and acceptable answer characteristics with each test. Run the set whenever you change chunking, embeddings, search parameters, reranking, prompts, models, or source connectors.
Test failure handling deliberately. Important cases include empty retrieval results, conflicting documents, parser errors, search timeouts, model timeouts, malformed structured output, and a user attempting to make retrieved text override application instructions. The prompt injection prevention checklist covers controls that should complement, not replace, application-level authorization. For additional guidance on reducing unsupported answers, see how to reduce hallucinations in RAG systems.
Monitor operational signals alongside answer quality. Track ingestion failures, index freshness, retrieval latency, generation latency, empty-result rates, citation coverage, validation failures, user corrections, and cost per request. These measurements help distinguish a search problem from a prompt problem or a source-data problem.
When to revisit
Revisit this RAG architecture whenever the underlying inputs or constraints change. Schedule a review after adding a new source type, changing document structure, introducing another tenant or permission model, replacing an embedding or generation model, or changing the expected citation and output format.
Also review the system when evaluation results drift, users report unsupported answers, retrieval becomes slower, source documents change more frequently, or context and token costs increase. A new model may require prompt and output-validation updates; a new search feature may change ranking behavior; and a new parser may alter chunk boundaries enough to invalidate prior evaluation results.
For a practical maintenance cycle, keep a versioned record of source-processing rules, chunking configuration, embedding model, index settings, retrieval parameters, prompt templates, model settings, and evaluation results. When updating one component, rerun the representative test set and compare both quality and operational metrics. Make the change only after confirming that improvements in one stage have not created regressions elsewhere.
Start with a narrow, well-defined knowledge domain, instrument every handoff, and treat “I don’t have enough evidence” as a valid outcome. That workflow produces a RAG application architecture that can evolve with its tools while remaining understandable to the developers responsible for it.
