Kognita
All posts
Engineering2026-10-29·9 min read

Scaling a knowledge base from 1,000 to 10,000,000 documents

What works at a thousand documents quietly breaks at ten million. Indexing, latency, ingestion throughput, and cost all shift. Here's what changes at each order of magnitude, and how to stay ahead of it.

A knowledge base at a thousand documents is forgiving — almost any architecture works, latency is fine, cost is a rounding error, and you can rebuild the whole index over lunch. At ten million documents, every one of those assumptions has flipped. The failures aren't dramatic; they're gradual, and each order of magnitude surfaces a different one. Scaling well means anticipating which thing breaks when, so you're upgrading ahead of the wall instead of crashing into it. Here's the tour from 1K to 10M.

scaling a knowledge base to 10 million documents

~1,000 documents: anything works

At this scale, don't over-engineer. A flat (brute-force exact) search is fast enough — you don't even need an ANN index. Ingestion is trivial. Cost is negligible. Latency is instant. The mistake here is premature scaling: standing up a distributed vector cluster and elaborate sharding for a corpus that a single small Postgres handles effortlessly. Build simple, measure, and let real growth pull you to the next stage. Most of what you'd "future-proof" now you'll do differently when you actually get there.

~100,000 documents: the index starts to matter

Somewhere in the tens-to-hundreds of thousands, brute-force search gets too slow for interactive use. This is where you adopt a proper ANN index (HNSW is the sensible default) so search stays fast without comparing against every vector. Other shifts:

  • Chunking discipline starts to bite. Sloppy chunking (tiny chunks, heavy overlap) multiplies your vector count, and now that multiplication matters. Right-size chunks to keep the count reasonable.
  • Ingestion should be async and queue-backed if it isn't already — you're ingesting enough that doing it inline blocks things. (A queue + worker model, like Kognita's, is the right shape from here on.)

Still very manageable, but "just search everything" is behind you and index choice is now a real decision.

~1,000,000 documents: latency and throughput become engineering

At a million documents (often tens of millions of chunks), you're doing real systems work:

  • Query latency needs active management. The ANN index parameters (e.g. HNSW's ef_search) now trade recall against latency meaningfully. You tune them deliberately, and you likely add reranking as a two-stage pattern — cheap wide retrieval, then precise rerank on a candidate set — which also helps quality at this scale where more candidates means more distractors.
  • Ingestion throughput matters. Re-embedding and indrehindexing at this volume takes real time and compute. Incremental sync (re-process only what changed) becomes essential — you can't afford full re-indexing on every change. Idempotent upserts let you parallelize ingestion workers safely.
  • Memory and cost enter the conversation. The index (especially HNSW in RAM) is now a sizable footprint. This is where the storage-cost math starts to be a line item worth forecasting, and where dimension choice and possibly quantization start paying off.
  • Metadata filtering is doing real work. Narrowing the search space by scope before ranking both improves relevance and reduces the effective search cost.

~10,000,000 documents: distribution and cost dominate

At ten million documents (potentially hundreds of millions of vectors), the dominant concerns are storage, memory, and distribution:

  • Storage and index memory are a major cost. This is where dimension reduction (Matryoshka truncation) and quantization (binary/scalar, with a full-precision rerank pass) move from "nice" to "necessary" — they directly attack the biggest cost drivers. A napkin forecast of documents × chunks × dimensions × precision + index overhead is now a budget conversation.
  • You may need to shard / distribute. A single node may no longer hold the whole index comfortably in memory. This is the point where a purpose-built distributed vector database earns its keep if you genuinely have this scale — sharding the index across nodes is what those systems are for. (Below this scale, reaching for them is the premature-scaling mistake.)
  • Freshness at scale is its own problem. Continuously re-embedding a fraction of ten million documents is a steady, significant compute load. Event-driven incremental sync (webhooks → re-embed just the changed content → idempotent upsert) is the only affordable way to stay fresh; full re-indexing is off the table.
  • Reconciliation and deletes matter more. At this volume, drift accumulates. Periodic reconciliation to catch missed updates and remove deletes keeps the corpus from silently diverging from reality.

The through-lines

A few patterns hold across every stage and are worth building toward early — not by over-engineering, but by choosing architectures that extend:

  • Async, queue-backed, idempotent ingestion. Right from ~100K, and it's exactly what carries you to 10M. Deterministic vector IDs and upserts mean you can parallelize and retry safely at any scale.
  • Incremental sync over full re-index. The larger you get, the more full re-indexing becomes impossible. Change-detection and per-document re-processing is the only thing that scales.
  • Two-stage retrieval (wide retrieve → rerank). Helps quality everywhere and becomes structurally important as candidate pools grow.
  • Forecast cost by the formula, not by feel. Storage and index memory scale predictably with documents × chunks × dimensions × precision. Model it before each order of magnitude.
  • Don't scale ahead of need. Each stage's mitigations are appropriate at that stage. Distributed sharding at 10K docs is waste; a single Postgres at 100M is a wall. Match the architecture to where you actually are, while choosing patterns that don't dead-end.

The takeaway

Scaling a knowledge base isn't one big leap — it's a series of specific bottlenecks that surface predictably as you cross orders of magnitude: index choice around 100K, latency and ingestion throughput around 1M, storage/memory/distribution around 10M. The teams that scale smoothly aren't the ones who built for 10M on day one (they overspent and guessed wrong); they're the ones who built simple but extensible — async idempotent ingestion, incremental sync, two-stage retrieval — and upgraded each component just ahead of the wall it was about to hit. Know which thing breaks when, and scaling becomes a roadmap instead of a series of fires.