An internal assistant answered the launch questions beautifully. A week later it quoted an obsolete policy to one employee and exposed a restricted project name to another. Retrieval was working. The product was not.
RAG describes a technical pattern: retrieve relevant material, place it in context, generate an answer. It does not decide which material is relevant, who may see it, how quickly it changes or what happens when evidence is insufficient. Those decisions are the product.
Retrieval quality begins before vectors
A useful index needs coherent source documents, stable identifiers, meaningful metadata and sensible boundaries. If a handbook page contains several unrelated policies, retrieving the whole page adds noise. If chunks lose their title, owner or effective date, the model cannot judge authority. If duplicates remain, an old copy can outvote the current one.
Test retrieval independently from generation. Given a real question, inspect which passages arrive, in what order and with which metadata. Include exact terms, paraphrases, acronyms, names and language variations. A fluent final answer can hide a weak retrieval result by filling the gap from model memory.
Permissions belong inside retrieval
Filtering after generation is too late. The retriever should only return documents the authenticated user may read. Permission metadata must travel with content and be enforced at query time. Cached answers and conversation history need the same boundary.
Microsoft's document-level access control guidance describes matching caller identity against permission metadata stored with indexed documents. It also highlights the operational catch: permission changes in the source only affect results after that metadata is synchronised. A secure design therefore needs both query-time enforcement and a defined sync path.
Never use a shared service credential to fetch everything and ask the model to hide sensitive parts. The model should not receive what the user cannot access.
Freshness needs an owner and a clock
A RAG system is a second representation of company knowledge. It becomes stale unless ingestion has triggers, monitoring and recovery. Decide which sources update on an event, which are polled and which require human publication. Store source revision and indexed time. Surface them in diagnostics and citations.
Deletion matters as much as addition. When a document is withdrawn or access changes, the index, cache and derived summaries must follow. A successful ingestion job is not proof of freshness if part of the source failed silently. Track lag by source and alert on missing or unusually old content.
"I do not know" is a product feature
The assistant needs an explicit boundary for insufficient evidence. That boundary cannot be a polite sentence added to the prompt. Measure whether retrieved passages support the answer. Require citations that resolve to content the current user can open. When sources conflict, show the conflict or route it to an owner.
A refusal should be useful. It can state what was searched, what evidence is missing and which team or system is authoritative. This turns uncertainty into a next step. Fabricated completeness turns it into hidden risk.
Make the assistant operational this week
Take recurring questions from real teams and build a test set with expected source documents, permission roles and freshness conditions. Run retrieval without generation first. Fix missing metadata, duplicate sources and chunk boundaries before tuning the prompt.
Then test the same question as users with different access. Revoke a document and measure how it disappears from search, cache and citations. Add cases where no source answers the question and define the expected refusal.
Finally, name owners for each source and for the assistant itself. A RAG demo proves that text can be retrieved and rewritten. A product proves that the right person receives current, authorised evidence and that the system declines to improvise when it cannot.