Keeping your knowledge base fresh: incremental sync vs. full re-index

A knowledge base is only as good as it is current. The moment a policy changes, a doc gets edited, or a ticket closes, your embeddings are describing a world that no longer exists — and a confidently wrong answer from stale content erodes trust faster than "I don't know."

So you have to re-index. The question is how much, and when. The two extremes are both wrong, and the interesting engineering is in between.

The two extremes

Full re-index every time. Whenever anything changes, re-embed the entire corpus from scratch. It's simple and always correct — the index can't drift because you rebuild it. It's also brutal at any real scale: re-embedding millions of chunks costs real money on every trivial edit, takes hours, and during the rebuild you're either serving a stale index or no index. Fine for a few thousand documents that change daily. Untenable for a large corpus that changes constantly.

Never re-index. Ingest once, never update. Fast, cheap, and wrong. Your knowledge base slowly becomes a museum of what was true at ingestion time.

The right answer is almost always incremental sync: detect precisely what changed and re-process only that, leaving everything else untouched.

Change detection: knowing what actually changed

Incremental sync lives or dies on change detection. You need to answer "what's different since last time?" without reading everything. Three common approaches, roughly in order of preference when available:

1. Content hashing. Store a hash of each document (or chunk) at ingestion. On sync, re-hash and compare. Changed hash → re-process; identical → skip. This is robust because it catches any content change and never re-processes unchanged content, no matter what the source system claims.

new_hash = sha256(document.text)
if new_hash != stored_hash[document.id]:
    reembed_and_upsert(document)
    stored_hash[document.id] = new_hash

2. Timestamps / version numbers. Many sources expose last_edited_time or a revision number. Poll for anything newer than your last sync. Cheaper than hashing (you don't fetch full content to compare), but only as trustworthy as the source's timestamps — some systems touch updated_at on no-op saves, causing needless re-embeds.

3. Webhooks / change feeds. The best option when the source offers it. Instead of polling, the source tells you what changed the moment it changes — Notion webhooks, database change feeds, S3 event notifications. This is what powers near-real-time freshness: an edit fires an event, you re-process just that document, and the next query sees the update seconds later.

In practice you combine them: webhooks for immediacy, a periodic hash-based reconciliation sweep to catch anything the webhooks missed (they will miss some — deliveries fail, systems restart).

Don't forget deletes

Updates are the easy case. Deletes are where incremental sync quietly breaks. If a document is removed from the source and you only ever process changes to existing content, its embeddings live on forever — and your bot keeps citing a document that no longer exists. That's worse than stale; it's a phantom.

Handling deletes needs either a change feed that emits delete events, or a reconciliation pass that diffs the set of source IDs against the set of indexed IDs and removes the orphans. This is a big argument for periodic full-reconciliation even in an incremental system — not a full re-embed, just a full comparison of what should exist against what does.

Make re-processing idempotent

Incremental sync means the same document gets re-processed many times over its life. If re-processing isn't idempotent, retries and overlapping syncs leave you with duplicate chunks — the same passage indexed three times, crowding your top-k with copies of itself.

The fix is deterministic vector IDs plus upserts. Kognita keys each vector as {content_id}_{chunk_index}, so re-ingesting a document overwrites its existing chunks in place rather than appending new ones. Re-process the same content ten times and you land in exactly the same state as processing it once. That property is what lets you retry aggressively without fear.

A practical architecture

Putting it together, a durable freshness pipeline looks like:

1. Webhooks from the source enqueue a re-sync job the moment content changes.
2. A worker fetches the changed document, hashes it, and skips if the content is genuinely unchanged (webhooks fire on no-op edits too).
3. Changed content is re-chunked, re-embedded, and upserted with deterministic IDs.
4. A scheduled reconciliation sweep (nightly, say) diffs source IDs against indexed IDs to catch missed updates and, crucially, removes deletes.

The goal isn't to pick one. It's incremental sync for the hot path — fast, cheap, near-real-time — backed by periodic reconciliation as the safety net that keeps small errors from compounding into a corpus that quietly diverges from reality.