Why Two Jobs Writing to the Same Vector Store Can Permanently Corrupt It

Your RAG pipeline had two writers and nobody noticed. A nightly ingestion job rebuilds embeddings; a request handler upserts new documents as they arrive. Both point at the same on-disk vector store directory. For weeks it works. Then one morning your retrieval starts returning nonsense, or throws on read, and no amount of restarting fixes it. The index is corrupted, and there is no repair command, because there is no repair.

Here is the uncomfortable truth about embedded, file-backed vector stores like a local Chroma PersistentClient: they are single-writer systems. They were never built with the concurrency control a real database has. There is no cross-process lock, no MVCC, no transaction spanning the two on-disk structures they maintain. Two processes writing at once is not a race you occasionally lose. It is a data-integrity bomb with a delayed fuse.

The fix is architectural, not a config flag. You either serialise every write through one process, or you move to a store built for concurrent writers. And whichever you choose, you make the vector store rebuildable, because the day it corrupts, rebuilding is the only way out.

The failure: corrupted after concurrent upserts

The bug report that captures this exactly reads:

[Bug]: PersistentClient is permanently corrupted after concurrent upsert operations

Note the two words that make this dangerous: permanently and concurrent. The corruption is not transient, so a restart does not clear it, and it is triggered by concurrency, so it will never show up in a single-process test. You reproduce it only under the exact conditions production runs in and development never does.

Why file-backed vector stores corrupt under concurrent writes

An embedded vector store keeps two different things on disk. Structured data such as documents, metadata, and ids goes into an embedded SQL database (SQLite, in Chroma's case). The vectors themselves go into a separate on-disk index structure, typically an HNSW graph, in its own segment files. A single logical "upsert" has to write to both, and keep them consistent with each other.

Now look at what serialises those writes. SQLite serialises writes within a single process, and can coordinate multiple processes on a local disk only through its own file locking, which is fragile and easily defeated. The vector index segment has no such protection at all. Critically, there is no transaction that spans both the SQL database and the index, so nothing guarantees they move forward together. When two processes interleave upserts, one can advance the SQL side while the other advances the index side, and the two structures drift out of agreement. Once they disagree, the store is internally inconsistent, and because there is no transaction log to replay or checksum to repair against, the inconsistency is permanent.

This is why the failure is silent until it is catastrophic. The writes appear to succeed. No exception is raised at write time. The damage only surfaces later, on a read, when the index points at data the SQL side does not have, or vice versa, and you get garbage results or a hard error. By then the writer that caused it is long gone from your logs.

The fix: enforce a single writer

Step 1: Route all writes through one process

The most reliable fix for an embedded store is to guarantee that exactly one process ever opens it for writing. Put a queue in front of it and let a single worker drain that queue. Your request handlers and your ingestion jobs enqueue write requests; they never touch the store directly.

# Producers (request handlers, cron jobs) only enqueue:
write_queue.put({"op": "upsert", "ids": ids, "docs": docs, "embeddings": embs})

# Exactly ONE worker process owns the store and drains the queue:
def writer_worker(collection):
    while True:
        job = write_queue.get()
        collection.upsert(
            ids=job["ids"],
            documents=job["docs"],
            embeddings=job["embeddings"],
        )
        write_queue.task_done()

This is not a lock bolted onto concurrent writers. It removes the concurrency entirely: there is one writer, always, so the two on-disk structures can never be advanced by two processes at once.

Step 2: If you need real concurrency, move to server mode

A single-writer queue is fine for most RAG workloads, where ingestion is periodic and modest. If you genuinely have high-volume concurrent writes, stop using the embedded client and run a client-server vector database. In server mode, one server process owns the storage and serialises writes internally, and your application talks to it over the network. Your app processes become clients, and it no longer matters how many of them write, because the server is the single writer.

# Instead of an embedded PersistentClient sharing files across processes:
import chromadb
client = chromadb.HttpClient(host="vectordb", port=8000)
# Every app process is now a client of one server that owns the data.

This is the same reasoning that governs the rest of a production RAG stack, and it fits the architecture I lay out in building RAG over ERP data that survives production: the vector store is infrastructure with an owner, not a file you sprinkle writes at from everywhere.

Step 3: Make the index rebuildable, and treat it as a cache

Whatever you choose, assume the store can be lost, because one day it will be. Keep your source documents and their embeddings as the authoritative record, in object storage or a normal database, and treat the vector index as derived data you can regenerate at any time.

def rebuild_index(target_dir, source_docs):
    # Build into a FRESH directory, then swap atomically. Never rebuild
    # in place over a store that readers are still using.
    client = chromadb.PersistentClient(path=target_dir)
    col = client.create_collection("docs")
    for batch in chunked(source_docs, 500):
        col.upsert(ids=batch.ids, documents=batch.texts, embeddings=batch.embeddings)
    return target_dir  # then point readers at target_dir and retire the old one

Building into a new directory and swapping means a rebuild is never a risky in-place mutation, and recovery from corruption is a routine operation you have already tested, not an emergency. Embeddings are the expensive part to regenerate, so store them alongside the source; the real cost math of running your own models is a good reminder not to recompute them casually.

Verify your write path is single-writer

1. Grep the codebase for every place that opens the store for writing.
   There must be exactly one, or exactly one server client.
2. Confirm ingestion jobs and request handlers enqueue writes rather
   than calling upsert directly.
3. Run your ingestion job AND simulate live upserts at the same time.
   With the queue in place, reads stay correct throughout.
4. Rebuild the index from source into a fresh directory and confirm
   retrieval results match the live store. This proves recovery works
   BEFORE you need it.

Point 4 is the one teams skip and regret. A rebuild path you have never run is not a recovery plan; it is a hope.

What people get wrong

Adding a threading.Lock and calling it fixed. A lock only coordinates threads within one process. Your ingestion job and your web server are separate processes, so an in-process lock does nothing to prevent them clobbering each other. Even within one process, wrapping an embedded store that is not built for concurrent access in a lock is treating the symptom.

Enabling SQLite WAL mode and assuming that solves it. WAL improves concurrent reads against the SQL side. It does not make the separate vector index transactional, it does not coordinate multiple writer processes, and it does nothing for the structure that actually corrupts. It is the right lever for the wrong problem.

Treating the vector store as the source of truth. If the only copy of your documents and embeddings lives inside the vector store, then corruption is not a rebuild, it is permanent data loss. The store must always be reconstructable from an authoritative source you keep elsewhere.

When it is already corrupted, or still breaking

• Already corrupted? There is no repair. Rebuild the collection from your source documents into a fresh directory and swap it in. If you did not keep the source, this is the incident that teaches you to, and you will re-embed from the original documents.
• Check for multiple client instances in one process. Two PersistentClient objects pointed at the same path inside the same app are also unsafe. There should be exactly one, owned by the writer.
• Networked or shared filesystems break it further. An embedded store on NFS or a volume mounted into two containers defeats even SQLite's own locking. Keep the files on a local disk owned by a single process, or use server mode.
• Two pods, one volume. In Kubernetes, a ReadWriteMany volume mounted by two writer pods is the concurrent-writer problem wearing an infrastructure costume. One writer, one volume, or a server deployment.

The principle worth internalising: an embedded vector store is a fast local index, not a database with the guarantees you have come to expect from Postgres. It has no one standing guard over concurrent writers, so you have to be that guard, by design, at the architecture level. Single writer, rebuildable index, source of truth kept elsewhere. Do that and a corrupted store is a ten-minute rebuild instead of a lost knowledge base.