How to build a company knowledge base for AI

The first deliverable is a question set, not a document dump

The fastest way to make an AI knowledge project vague is to begin with every folder the company owns. A shared drive can contain policy, drafts, exports, personal notes, and copies of records that are authoritative somewhere else. Connecting all of it creates coverage, but it does not create a rule for choosing the right fact.

Begin with questions that already interrupt work. Which agreement governs this client? Can this worker be extended? What did the team promise after the last review? Which policy applies in this country? A useful first set is large enough to expose variation and small enough that a business owner can check every expected answer.

For each question, record the decision or task that follows. This separates curiosity from operational value. If nobody can name what a better answer changes, the question does not belong in the first implementation.

• The exact question in the language employees use
• The facts needed to answer it
• The person accountable for the decision
• The next step a reliable answer makes possible
• The consequence of a wrong, late, or unauthorized answer

Authority belongs to a fact, not to an entire app

A CRM may own the current opportunity stage while a signed agreement owns notice terms. A staffing platform may own placement status while a finance system owns an approved bill rate. Declaring one tool the single source of truth hides these differences.

Create a source-authority map at the level where a conflict can be resolved. Name the fact type, primary source, supporting source, owner, scope, expected update rhythm, and fallback. Scope matters because the authoritative record can change by client, business unit, country, contract type, or date.

This business mapping is consistent with the broader risk approach in the NIST AI Risk Management Framework, which organizes work around governance, context, measurement, and ongoing management rather than a one-time technology selection.

The map is not a data catalog for its own sake. It gives retrieval a business rule. It also gives the team a named person to contact when two approved records disagree.

Enforce access before evidence reaches the model

A polished answer cannot repair a permission leak that already happened during retrieval. The query should carry the user's identity, role, client, department, or other required boundary into the search layer. The candidate evidence set must be filtered before it is assembled into model context.

Microsoft's current RAG guidance recommends retrieval-time access control and production identity controls. Its secure multitenant reference also describes document filtering and data-layer security as part of the architecture, not a prompt instruction.

A Google Cloud example makes the same boundary concrete: an authenticated application retrieves only rows relevant to the user, so the model never receives every user's data at once. The authenticated RAG design is product-specific, but the principle travels well.

Test access with pairs of users, not only an administrator account. Ask the same question as a recruiter and a finance lead, as two client teams, and as a former employee whose access has been revoked. The expected answer may differ, but restricted evidence must not appear in the answer, citation, hidden context, or action payload.

Freshness and conflict need explicit behavior

An authoritative source can still be unsafe when it is overdue for review. Store the source timestamp, effective date, owner, and expected review interval with the retrieved evidence. The answer should say when a consequential source was last checked and whether that falls inside the agreed window.

Do not turn recency into a universal authority rule. A recently edited meeting note does not override a signed agreement. A draft policy dated yesterday does not defeat an approved policy from last month. The source map decides priority; dates decide whether the chosen evidence is still usable.

When approved sources conflict, return the conflicting claims and route the issue to the owner. Any downstream action that depends on the disputed fact should pause. This may feel slower than asking the model to choose, but it converts an invisible information defect into work the business can resolve.

Keep ingestion, retrieval, and answer tests separate

Ingestion asks whether the system captured the right content, metadata, and permissions. Retrieval asks whether the right passages appear for a question. Answer evaluation asks whether the response is supported, direct, and appropriately limited. A single end-to-end demo can pass while one of these layers remains brittle.

Google's private RAG reference architecture separates data processing, retrieval, serving, networking, and IAM concerns. The point is not that every small company needs that cloud design. It is that each layer has a different failure and owner.

Build a compact evaluation set before tuning chunk size or search parameters. Include clean questions, ambiguous wording, missing facts, old documents, conflicting documents, restricted records, and a question that should be refused. Record the expected evidence as well as the expected answer.

Treat retrieved documents as untrusted input

A document can contain instructions as well as facts. That becomes dangerous when a model treats text found in a file, email, ticket, or web page as permission to ignore its operating rules or call a tool.

The OWASP prompt-injection guidance identifies malicious instructions embedded in retrieved knowledge as an indirect prompt-injection path. Microsoft likewise advises treating retrieved content as untrusted.

Reduce the consequence rather than assuming a perfect filter. Separate instructions from retrieved data in the application, limit the tools available for each job, validate structured tool inputs, and require approval for sensitive actions. Log which evidence reached the model so an incident can be reconstructed.

Delay write access until read-only behavior holds up

A useful answer can prepare a CRM change, task, or message without executing it. That review stage exposes mapping errors while the consequence is still reversible. It also gives the business enough examples to write real approval rules.

When action is justified, scope the credential to the smallest required operation. Validate the target record and proposed change outside the model. Apply the approval rule in application logic, then confirm that the destination system accepted the exact change. An API success code is not evidence that the business outcome occurred.

This is where a company knowledge base can become a Business Brain. The transition is earned by proven evidence and control, not by adding a tool-call button to the chat interface.

A focused first month

In week one, collect the question set and name the decision owners. In week two, build the source-authority and access map. In week three, connect the smallest useful evidence set and run retrieval tests. In week four, evaluate answers with the people who own the work and produce a list of failures, missing content, and unresolved authority decisions.

The month should end with evidence about one operating slice, not a promise that the company has been understood. If the read-only system cannot pass missing-evidence, conflict, freshness, and restricted-user tests, keep it read-only. That is a useful result because it shows exactly what the business must fix before automation can carry more responsibility.