A production-ready retrieval-augmented generation application needs more than an LLM call and a vector database. This guide shows how to design a RAG application in Python, estimate its recurring costs with replaceable inputs, evaluate answer quality, add citations and safeguards, and decide when the system needs to be recalculated or updated.
Overview
Retrieval-augmented generation, or RAG, connects a language model to a controlled collection of external documents. Instead of asking the model to answer entirely from its training data, the application retrieves relevant passages and includes them in the prompt. The model then generates an answer grounded in that retrieved context.
A practical RAG system usually contains six stages:
- Ingestion: Read files, web pages, tickets, database records, or other approved sources.
- Preparation: Clean text, preserve useful metadata, and divide documents into searchable chunks.
- Embedding: Convert each chunk into a numerical vector that represents its meaning.
- Retrieval: Compare a user query with stored vectors and return the most relevant passages.
- Generation: Send the question and selected context to an LLM with instructions for answering.
- Evaluation and operations: Measure retrieval, answer quality, latency, errors, usage, and cost.
The goal is not to maximize the number of retrieved documents. It is to provide enough relevant context for a useful answer without creating unnecessary token usage, latency, or ambiguity. Your design should also make uncertainty visible. If the documents do not support an answer, the application should be able to say so rather than fill the gap with an unsupported response.
For a broader development sequence, see the AI app development roadmap. If you want to compare storage options, the vector database comparison provides a useful starting point.
How to estimate
Estimate the application in separate units instead of treating it as one unknown monthly bill. The most useful model has four parts: generation, embeddings, storage and retrieval, and application operations.
1. Estimate generation usage
Record the expected number of requests, average input tokens, and average output tokens. Input tokens include the user question, system instructions, conversation history, and retrieved context. A simple monthly formula is:
Generation cost = requests × [(average input tokens ÷ 1,000,000 × input rate) + (average output tokens ÷ 1,000,000 × output rate)]
Keep the rates as variables because model pricing, billing units, and provider terms can change. If you support several models, calculate each route separately. For example, a short-answer route may use one model while a complex research route uses another.
2. Estimate embedding work
Embedding usage is driven by the amount of text processed, not only by the number of users. Include the initial indexing job and ongoing updates:
Embedding tokens = new or changed chunks × average tokens per chunk
If documents are re-indexed unnecessarily, this number can become much larger than expected. Store a content hash or version identifier for each source and embed only new or changed content when the embedding model and chunking strategy remain compatible.
3. Estimate retrieval and infrastructure
List the services that run outside the model provider: vector storage, database storage, object storage, API hosting, logs, monitoring, queues, and scheduled ingestion jobs. Some services may have a fixed baseline, while others scale with records, requests, storage, or compute time. Use the provider's current calculator or billing documentation for the actual rate rather than copying a price into application code.
For performance, estimate latency as a sequence:
Total latency = query preparation + embedding + vector search + reranking + LLM generation + application overhead
Measure each component in a test environment. A single average can hide slow requests, so record a high-percentile target as well as the median. The exact target depends on whether the application supports chat, internal search, batch analysis, or an asynchronous workflow.
4. Estimate quality outcomes
Cost alone does not tell you whether the system works. Create a small evaluation set containing representative questions, expected source documents, and an acceptable answer description. Track at least:
- Retrieval relevance: whether the returned passages contain the information needed to answer.
- Groundedness: whether the answer is supported by the retrieved context.
- Completeness: whether the response covers the important parts of the question.
- Citation accuracy: whether each citation points to evidence for the associated claim.
- Refusal behavior: whether the application avoids inventing an answer when evidence is missing.
Use a consistent rubric, even if the first version is a human review spreadsheet. A repeatable evaluation is more useful than an impressive demonstration using a few hand-picked questions.
Inputs and assumptions
Before building the calculator, write down assumptions that can be inspected and changed. At minimum, capture:
- Monthly user requests and expected growth range.
- Average question length, conversation history, and retrieved context size.
- Maximum output length and the percentage of requests requiring a longer response.
- Number of source documents, average chunk size, overlap, and metadata fields.
- Initial indexing volume and monthly document change volume.
- Top-k retrieval value, reranking usage, and any filtering rules.
- Model and embedding provider rates, represented as replaceable variables.
- Storage, database, hosting, logging, and monitoring assumptions.
- Target response time, error rate, and evaluation thresholds.
Chunking is a quality and cost decision. Very small chunks may separate related facts and require more retrieval results. Very large chunks may introduce irrelevant text and increase prompt size. Start with a consistent strategy, preserve document titles and section paths, and test several chunk sizes against the evaluation set rather than selecting one by intuition.
Prompt construction should be explicit. A reliable template typically identifies the assistant's role, defines how context may be used, instructs the model to distinguish evidence from uncertainty, and requests citations in a predictable format. Keep retrieved text clearly separated from instructions, and do not assume that retrieved documents are trustworthy instructions. This is one reason to treat document content as data rather than as a new system prompt. The guide to prompt engineering for developers covers related patterns.
For a Python implementation, keep ingestion, retrieval, prompting, and provider calls behind small interfaces. This makes it easier to test a local open-source model, change a hosted model, or replace a vector store without rewriting the entire application. Use configuration variables for model names, limits, top-k values, and rates; do not scatter them across route handlers.
Worked examples
Example A: Estimating monthly model usage
Assume a support application receives 1,000 requests in a month. Each request uses an average of 1,200 input tokens and produces 350 output tokens. Let p-in represent the provider's input rate per million tokens and p-out represent its output rate per million tokens.
Monthly input usage is 1,000 × 1,200 = 1.2 million tokens. Monthly output usage is 1,000 × 350 = 0.35 million tokens. The generation estimate is therefore:
1.2 × p-in + 0.35 × p-out
This formula remains useful when the provider changes its rates. Replace the two variables, then add the embedding and infrastructure estimates. If a conversation history increases the average input from 1,200 to 2,000 tokens, recalculate before launch; the change affects every request.
Example B: Estimating indexing work
Assume an internal knowledge base adds or changes 10,000 chunks each month, with 800 tokens per chunk. The monthly embedding volume is:
10,000 × 800 = 8 million embedding tokens
If the same documents are embedded again during every deployment, the estimate is no longer 8 million tokens. Track document versions so a deployment does not automatically create a new embedding bill or duplicate records.
Example C: Comparing a quality change
Suppose an initial test retrieves five passages and sends all five to the model. A revised pipeline retrieves ten passages, reranks them, and sends the best three. Compare both versions on the same evaluation questions. Record retrieval relevance, citation accuracy, input tokens, and latency. Keep the revised design only if the quality improvement is meaningful for the product and the additional retrieval or reranking work is justified.
This comparison is more informative than judging responses from separate test questions. Change one major variable at a time, retain the evaluation outputs, and document the reason for the chosen configuration.
When to recalculate
Revisit the estimate whenever an input changes, not only when an invoice looks surprising. Recalculate after:
- A model, embedding model, or provider rate changes.
- Traffic, conversation length, output limits, or retry behavior changes.
- Chunk size, overlap, metadata, top-k retrieval, or reranking changes.
- The document collection grows or the ingestion schedule becomes more frequent.
- A new region, hosting service, vector database, logging level, or retention policy is introduced.
- Evaluation results show weak retrieval, unsupported answers, or citation errors.
- Latency or failure rates increase in production.
Use a small change checklist: export the current assumptions, update one variable, recalculate the monthly range, run the evaluation set, and compare quality and latency with the previous version. Keep a dated record of the model, SDK, embedding configuration, prompt version, chunking rules, and vector index version.
Finally, separate a prototype from a production service. A prototype can begin with local files and a simple vector index. A production system needs authentication, access filtering, source tracking, structured logs, retry limits, timeouts, secret management, deletion handling, and a clear fallback when retrieval fails. Review the Python RAG tutorial for a practical foundation, then use this estimation model as a living worksheet. Updating the inputs after every meaningful architecture or pricing change keeps the application understandable, testable, and financially predictable.
