How to build a knowledge base that updates over time

Start with a source class such as signed agreements, policy pages, CRM account notes, or meeting decisions. For that class, name what creates a record, which edits matter, what deletion means, who may change access, and how quickly the searchable state must follow. A contract can tolerate a fifteen-minute delay while an emergency procedure may not.

Stable identity is the quiet requirement. If a renamed file arrives as a new object while the old chunk remains indexed, the knowledge base now has two versions and no reliable way to know which one should answer. Store a source record ID, version or modification marker, owner, permission reference, and ingestion status outside the generated chunks.

An addition is the easy case. The pipeline reads the source, extracts content and metadata, creates searchable units, and records the indexed version. An edit needs replacement behavior. Upserting new chunks without removing obsolete ones can leave a convincing old sentence beside the current rule.

AWS documents that its managed knowledge-base sync processes added, modified, and deleted documents incrementally. The source still must be synchronized after those changes. The important design lesson is not the vendor command. It is that deletion belongs in the same freshness contract as addition.

A permission change can be more urgent than a content edit. If a folder becomes restricted, waiting for the next content crawl can leave its old passages searchable. Permission metadata needs its own change path or a retrieval-time check against the live source boundary.

An event-driven path can react quickly when a source changes. It also misses events during outages, permission failures, subscription errors, or deployment mistakes. A scheduled incremental crawl can repair those gaps, but it may leave consequential changes stale for too long when used alone.

Combine them. Let source events place a record into a durable processing queue, then run a periodic reconciliation that compares source identity and version markers with the index ledger. The event path buys speed. The reconciliation path buys recovery.

For sources with no useful event stream, shorten the polling interval only after measuring the cost and business consequence. Polling every minute is not a governance model. It is a resource choice inside one.

Freshness cannot solve authority. A recent Slack message may contradict an older approved policy because the conversation is proposing a change, not publishing one. Mark source roles such as draft, approved policy, executed agreement, operational record, and historical evidence. The answer route can then prefer or disclose sources according to the question.

Direct ingestion creates another authority risk. AWS warns that directly added or changed content can be overwritten when the original data source is later synchronized. If operators can edit the index without updating the owning source, the next successful sync may silently undo the correction.

Choose one durable owner for the fact. Store derived summaries, chunks, and embeddings as rebuildable representations, not as secret alternate truth.

Do not hide the last successful version behind a green interface after a job fails. Record the attempted source version, last successful source version, processing stage, error, retry state, and owner. Expose staleness to the answer route when the deadline is exceeded.

The safe response depends on consequence. A low-impact internal guide may show an answer with a visible last-verified date. A price, safety instruction, or contractual term may need to stop until the current source is confirmed. The freshness policy should make that distinction before the outage.

Retries need idempotency. Reprocessing the same source version should not create duplicate chunks or duplicate downstream actions. Recovery should be testable without manually cleaning the index after every failure.

An ingestion dashboard can show success while retrieval still returns an old chunk. Build a small freshness suite around real questions. Change a known fact, remove a source, revoke one role, create a conflict, and fail a processing step. Check the visible answer, citations, access result, index ledger, and recovery log.

Track time from source change to verified searchable state, not merely job duration. Also track stale-answer escapes, failed deletions, permission lag, reconciliation repairs, and sources past their expected update window. These measures tell the team whether the operating contract holds.

The broader AI knowledge-base implementation service should include this loop in acceptance, because a good launch without a repairable update path is only a temporary index.