RAG in production: an engineering checklist for enterprise knowledge assistants

Retrieval augmented generation (RAG) gives a language model access to your own documents at the moment it answers a question. Instead of relying on what the model memorized during training, the system retrieves relevant passages and asks the model to answer from them.

A working demo takes an afternoon. A knowledge assistant that hundreds of employees trust takes deliberate engineering. This checklist covers the decisions we review on every RAG project.

1. Treat ingestion as a product, not a script

Most answer quality problems start before retrieval. Documents arrive as scanned PDFs, slide decks, spreadsheets, and wiki pages with navigation clutter.

• Parse by document type. Use layout aware parsing for PDFs, preserve table structure, and run OCR only where text is missing.
• Keep metadata. Source system, title, section headings, author, date, and access permissions are as important as the text.
• Make it incremental. Detect changed and deleted documents so the index reflects the current state of your knowledge, not last quarter's.

2. Chunk along meaning, not character counts

Fixed size chunks split tables and procedures in half. Prefer chunking along headings and paragraphs, with modest overlap, and store the parent section so the model can be given wider context when needed. Test two or three strategies against your evaluation set rather than choosing one by intuition.

3. Use hybrid retrieval

Vector search finds passages that mean the same thing as the question. Keyword search finds exact product codes, clause numbers, and names that embeddings often miss. Combining both, then reranking the merged results with a cross encoder, is a reliable default for enterprise content.

Vector databases such as Qdrant and Pinecone, or PostgreSQL with a vector extension, all support this pattern. The right choice depends on scale, hosting constraints, and what your team already operates.

4. Enforce permissions at retrieval time

An assistant must never reveal a passage the user could not open in the source system. Store access control information with every chunk and filter during retrieval, not after generation. Filtering after the model has seen restricted text is too late.

5. Require citations and allow "I don't know"

Instruct the model to answer only from retrieved passages and to cite them. Display citations as links to the original document and section. Equally important, make it acceptable for the assistant to say the information was not found. A confident wrong answer does more damage to trust than an honest gap.

6. Build the evaluation set first

Before tuning anything, work with subject matter experts to collect representative questions and the passages that answer them. Then measure:

Layer	What to measure
Retrieval	Whether the correct passage appears in the top results
Generation	Whether the answer is correct and supported by the cited passages
Experience	Latency, cost per question, and refusal rate

Run the evaluation on every change to chunking, embeddings, prompts, or models. Without it, improvements in one area quietly cause regressions elsewhere.

7. Monitor in production

Log questions, retrieved passages, answers, and user feedback, with appropriate privacy controls. Review unanswered and poorly rated questions weekly. They show you which documents are missing, outdated, or poorly structured, which is often the highest value improvement available.

8. Plan for change

Models, embedding providers, and prices change. Keep model calls behind a thin interface, version your prompts, and store embeddings with the model that produced them so you can reindex deliberately.

Where to start

Pick one well defined domain, such as HR policy or product documentation, with an owner who can judge answer quality. Build the evaluation set, ship to a pilot group, and expand once the numbers and the feedback support it.

If you are planning a knowledge assistant, our AI engineering team can help you scope the architecture and the evaluation approach before you commit to a build.