An AI assistant knowledge base: documents, access and answer checks

An AI assistant needs agreed information, access boundaries and an owner who keeps sources current. Uploading a folder does not resolve conflicting price lists or make an old instruction valid. Start with one process: internal policy questions or customer advice based on approved conditions. Users should be able to check a source and reach a person when an answer needs review.

1. Define the pilot and its boundaries

Identify who asks questions, what decisions they make and what the assistant must not do. An internal helper and a customer bot need different documents and permissions. Limit the first release to a defined topic and appoint an answer owner. Where a delivery or returns rule has not been approved, the model should not invent a plausible business decision. Agree the rule or the route to a responsible employee first.

2. Build a register rather than a shared folder

Remove duplicates, mark archives and resolve contradictions. Check readable scans, meaningful tables and headings that explain a rule’s context. Exclude passwords and secret keys. Personal and confidential information needs an explicit handling and access decision; it should not enter a pilot accidentally with the rest of a folder.

3. Separate documents from live data

A policy document is suitable for document search. Stock, order status and customer-specific prices may change after a file is loaded. Discuss reading the operating system and checking the user at the time of the request. If that is outside the pilot, limit the answer and hand off. Preparing documents does not replace CRM integration, and an integration does not make all data available to everyone.

Retrieval-Augmented Generation, or RAG, retrieves relevant passages and adds them to the model’s context. Retrieval quality affects what evidence is available; the method’s name does not establish answer correctness. Mechanism: https://www.anthropic.com/research/contextual-retrieval .

4. Define uncertainty and human handoff

Specify behaviour for missing evidence, contradictory documents and out-of-scope questions. A citation should point to the source used, not merely a related page. Explain what is unknown and how the user can proceed. If a request is recorded for a person, agree the information passed and confirm actual storage. A refusal is useful when the user still has a clear route to resolving the task.

A retrieved document can contain instructions intended to alter model behaviour. OWASP notes that RAG does not eliminate prompt injection. Treat documents as data and enforce action and access restrictions separately: https://genai.owasp.org/llmrisk/llm01-prompt-injection/ .

5. Write evaluation questions before launch

1. A direct question answered in a current document.
2. The same question rephrased or misspelled.
3. A question in another agreed language.
4. A condition changed in a newer document version.
5. A question not answered by the knowledge base.
6. Conflicting conditions in two documents.
7. A request for information outside the user’s role.
8. A request to perform an unauthorised action.

Record the expected outcome, permitted source and handoff rule for each case. This is a suggested starting set, not a claim that eight questions are sufficient for every project. Expand it with permitted real user wording. Check answer correctness, citation relevance, access and the next step separately; one aggregate score can hide a serious failure.

NIST addresses reliability, information security and risk evaluation as distinct generative AI management concerns. Use its profile to assess pilot boundaries, not as a quality certificate: https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf .

6. Own updates and evaluate usefulness

When a policy or price changes, update the version available to the assistant and repeat the affected tests. Assign responsibility for disputed answers and changes to operating limits. Evaluate the chosen process through completed requests, manual handling and answer quality. More conversations do not necessarily mean less work. Operating costs include services, storage, model usage and continuing source review.

Discuss a bounded pilot at https://samsa.agency/en/services/ai-automation . A customer-facing Telegram workflow can start at https://samsa.agency/en/solutions/telegram-bot . Bring one task, approved source documents and evaluation questions to define the development scope.