Kognita
All posts
Engineering2026-06-12·10 min read

Ingestion pipelines that don't fall over: idempotency and fault tolerance

Embedding a document touches a queue, an embedding API, and a vector store, each of which will fail eventually. Here's how to build ingestion that retries safely, never double-writes, and recovers on its own.

Ingesting a document into a RAG system looks trivial in a demo: chunk it, embed it, store the vectors. In production, that path crosses a message queue, a rate-limited embedding API, and a vector database, and every one of them will fail at some point. The difference between a toy and a production pipeline is entirely in how you handle the failure.

Fault tolerant ingestion pipelines

The failure modes you must design for

Before writing any code, enumerate what breaks:

  • The embedding API times out after embedding but before you get the response.
  • The worker crashes halfway through a document with 400 chunks.
  • The same job gets delivered twice by the queue.
  • The vector store accepts writes 1–200, then rejects the rest.
  • A poison document fails every single time and blocks the queue behind it.

A naive pipeline turns any of these into corrupted data or a stuck worker. The two properties that prevent that are idempotency and at-least-once processing with safe retries.

Idempotency: make re-running a no-op

The single most important design decision: running the same job twice must produce the same result as running it once. The enemy is auto-generated primary keys: every retry inserts a fresh copy, and you end up with the same chunk stored three times, each returned separately at search time.

The fix is deterministic IDs. Derive each vector's ID from stable inputs rather than a random value:

vector_id = content_id + "_" + chunk_index
// e.g. "doc_abc123_7"  — same value every single time

Now use upsert instead of insert. A retry overwrites the identical row with identical data. It is harmless. Re-ingesting an edited document overwrites the old chunks in place. This one change eliminates an entire class of duplicate-result bugs and makes retries free.

upsert into vector_store:
    id       = vector_id
    values   = embedding_vector
    metadata = { content_id, chunk_index }

At-least-once delivery with a visibility timeout

Use a queue that guarantees at-least-once delivery; PGMQ, SQS, and most brokers do this. The pattern:

  1. Worker reads a message; the queue hides it for a visibility timeout rather than deleting it.
  2. Worker processes the document to completion.
  3. Only then does the worker delete (acknowledge) the message.
  4. If the worker crashes before step 3, the timeout expires and the message reappears for another worker.

The critical rule: delete the message last. If you ack on receipt, a crash mid-processing loses the job silently. Ack after success, and every job either completes or comes back around. Because processing is idempotent, the redelivery is safe.

Set the visibility timeout longer than your worst-case processing time. A 400-chunk document that takes 90 seconds needs a timeout comfortably above that; otherwise the message reappears and a second worker starts duplicating work (harmless for correctness thanks to upserts, but wasteful).

Partial failure within a document

A big document is itself a small pipeline. If chunk 200 of 400 fails, you do not want to re-embed the first 199 on retry. Two workable approaches:

  • Batch and checkpoint. Embed and upsert in batches of, say, 50. Because upserts are idempotent, a retry re-runs completed batches harmlessly and continues past the failure.
  • Status on the parent record. Track the content's state (PENDING → PROCESSING → READY) and only flip to READY after every chunk is written. Search can filter out anything not READY, so half-ingested documents never surface.

Combining both gives you a document that is either fully searchable or not searchable at all. It is never half-indexed.

Retries, backoff, and the poison message

Transient failures (rate limits, blips) should retry with exponential backoff and jitter so a flood of workers doesn't hammer a recovering API in lockstep:

delay = min(base_delay * 2^attempt, max_delay)
delay = delay + random(0, delay * 0.1)   // jitter

But some documents fail every time: a corrupt PDF, an unsupported encoding, or a chunk that trips the embedding API. Left alone, a poison message redelivers forever and starves everything behind it. Guard against it:

  • Track a delivery/attempt count on the message.
  • After N attempts (5 is a reasonable default), route it to a dead-letter queue and move on.
  • Alert on dead-letter arrivals; that queue should normally be empty.

Observability so you can trust it

A self-healing pipeline is only trustworthy if you can see it heal. Track, at minimum:

  • Queue depth and oldest-message age: rising age means workers can't keep up or are stuck.
  • Success / retry / dead-letter rates: a retry-rate spike is your early warning.
  • End-to-end latency from enqueue to READY.

Putting it together

The mental model is simple once the pieces click: the queue guarantees the job won't be lost, idempotency guarantees running it twice is safe, and status tracking guarantees a document is all-or-nothing. Together they mean you can crash a worker at any instant, restart it, and the system converges to the correct state on its own.

That property, crash anywhere, recover automatically, never corrupt data, is what separates an ingestion pipeline you can run in production from one you have to babysit.