Your Queue Isn't Broken: Background Jobs Need Idempotency, Not Exactly-Once

A customer emails: they were charged twice. You check the logs. The payment job ran twice. You did not write a bug that calls the payment API in a loop, so how? You dig in and find the worker crashed for an unrelated reason (an OOM, a deploy, a network blip) a few hundred milliseconds after it charged the card but before it marked the job done. The queue, doing exactly what it promises, handed the job to another worker. That worker charged the card again.

Your first instinct is that the queue is broken. It is not. Almost every job queue in production use, Sidekiq, Bull, Celery, SQS, gives you at-least-once delivery, and that is a deliberate reliability choice, not a defect. The queue is telling you plainly: I will make sure your job runs, and under failure I may run it more than once. The double charge is not the queue failing. It is the queue succeeding at a guarantee you did not design for.

The fix that actually holds is not to hunt for a queue with exactly-once delivery, because at the level that matters it does not exist. The fix is to make the job itself idempotent, so running it twice has the same effect as running it once. Let me show you why exactly-once is a mirage and how to build the thing that works instead.

Why "just run it once" is impossible, not just hard

Picture the worker's job as three actions: (1) do the side effect (charge the card), (2) record that it is done, (3) tell the queue to remove the message. For exactly-once to hold, all three would have to happen as one atomic step. They cannot, because the side effect happens in a different system (the payment provider) than the bookkeeping (your database) than the acknowledgement (the queue). Between any two of those steps, the worker can die.

So the queue has to pick. If it removes the message before the worker finishes, a crash means the job is lost forever: that is at-most-once, and it is unacceptable for a payment. If it removes the message only after the worker acknowledges success, a crash after the side effect but before the ack means the job runs again: that is at-least-once. Sane queues choose at-least-once, because a duplicate you can defend against is better than a silent loss you cannot detect. The duplicate is the price of never losing work.

The mechanism you actually control: an idempotency key

If you cannot stop the job from running twice, make the second run a no-op. The tool is an idempotency key: a stable identifier for the unit of work, chosen so that the same logical operation always produces the same key. Not a random UUID generated inside the handler (that changes on every run and defeats the purpose), but something derived from the intent: the order id, the invoice id, a hash of the payload, whatever uniquely names "this specific charge".

The worker's contract becomes: before doing the side effect, atomically claim the key. If the claim succeeds, do the work. If the claim fails because the key already exists, someone already did this, so skip it. The atomicity is the whole game. A naive "check if exists, then insert" has a race: two workers both check, both see nothing, both proceed. You need the database to make the check-and-claim a single indivisible operation.

Step 1: Make the key claim atomic with a unique constraint

The simplest correct mechanism is a table with a unique constraint on the key. Inserting is the claim; the database rejects the second insert.

CREATE TABLE processed_jobs (
    idempotency_key TEXT PRIMARY KEY,
    processed_at    TIMESTAMPTZ NOT NULL DEFAULT now()
);

def handle_charge(order_id, amount):
    key = f"charge:{order_id}"
    try:
        with db.transaction():
            db.execute(
                "INSERT INTO processed_jobs (idempotency_key) VALUES (%s)",
                (key,),
            )
    except UniqueViolation:
        # another run already claimed this key; nothing to do
        log.info("charge %s already processed, skipping", key)
        return
    # we hold the claim; safe to do the side effect
    charge_card(order_id, amount, idempotency_key=key)

Expected behaviour: the first worker inserts the row and charges; the second worker's insert raises a unique-violation, it catches it, and it returns without charging. The database, not your application logic, is the thing preventing the race.

Step 2: Pass the same key to the payment provider

Notice the idempotency_key=key passed into charge_card. Good payment APIs (Stripe, and the pattern generalises) accept an idempotency key on the request and guarantee that two requests with the same key charge only once. This is your safety net for the ugliest race of all: the worker inserts the claim row, charges the card, then crashes before committing, so the claim row rolls back. Now the claim is gone but the charge happened. Without a provider-side key, the retry would claim the key freshly and charge again. With the provider key, the retry's charge request is deduplicated by the provider. Belt and braces, because the money side is the one you cannot undo.

Step 3: For created-then-committed effects, use a transactional outbox

The claim-then-act pattern has a residual gap: the side effect (an external API call, an email) is not part of your database transaction, so a crash between them can leave the two out of sync. When the effect is "notify another system", the clean fix is the transactional outbox. Instead of calling the external system inside the handler, you write the intent to an outbox table in the same transaction that changes your business data. A separate relay process reads unsent outbox rows and delivers them, marking each sent. Because the business change and the outbox row commit together, you can never have one without the other.

-- committed atomically with the order status change
INSERT INTO outbox (id, topic, payload)
VALUES (gen_random_uuid(), 'order.paid', '{"order_id": 4471}');

The relay still delivers at-least-once (it can crash after sending, before marking sent), so the consumer of the outbox must also be idempotent, using the outbox row id as its idempotency key. This is the same principle applied one layer out: you never achieve exactly-once, you achieve at-least-once plus idempotent consumers, everywhere.

Verification: prove a double delivery is harmless

Do not trust that it works, force a duplicate and watch. The cleanest test is to invoke the handler twice with the same input, in the same test, and assert the side effect happened once:

def test_charge_is_idempotent(fake_provider):
    handle_charge(order_id=4471, amount=1500)
    handle_charge(order_id=4471, amount=1500)  # simulated redelivery
    assert fake_provider.charge_count(order_id=4471) == 1

Run it. A count of 1 after two invocations is your proof. Then do the harder test: kill the worker process mid-job in staging (a SIGKILL right after the charge) and let the queue redeliver, then confirm the customer was charged once. If your unique constraint and provider key are right, the redelivery lands as a skip. This is the same discipline behind handling an M-Pesa callback that fires twice: assume the duplicate will happen and make it boring.

What people get wrong

Chasing exactly-once at the infrastructure layer. Teams enable every "deduplication" and "exactly-once" toggle the broker offers and consider the problem solved. These features deduplicate within a window, on a best-effort basis, and they cannot cover the case that actually bites you: the crash between your side effect and your acknowledgement. They reduce duplicates, they do not eliminate them, and building on the assumption that they do is how you ship the double charge anyway. Idempotent handlers are the only approach that holds under a real crash.

Using a random UUID as the idempotency key. If the key is generated fresh inside the handler on each run, the retry generates a different key and dedup never triggers. The key must be stable across runs, derived from the work, not from the attempt.

Check-then-act without atomicity. "SELECT to see if it exists, then INSERT if not" is a race, not a fix. Two concurrent workers both pass the check and both act. Use a unique constraint or a single atomic upsert so the database arbitrates, never your application code.

Ignoring retry storms. A job that fails 100 percent of the time (a poison message) will be retried forever by default, and with enough of them the retries consume the majority of your worker capacity, starving healthy jobs. Cap retry attempts, use exponential backoff, and route exhausted jobs to a dead-letter queue for a human, rather than letting them spin. Auto-retrying a permanent failure is the queue equivalent of the mistake dissected in CI auto-retry hiding your flaky tests.

When it is still broken

If duplicates survive an idempotency key, look here:

• The claim and the side effect are in different transactions. If you commit the claim, then in a separate transaction do the work, a crash in between loses the work while keeping the claim, so the retry skips real work. Keep the claim uncommitted until the work is safely durable, or lean on the provider-side key to catch the gap.
• Your key is not as unique as you think. Two logically different operations that collide to the same key will make the second silently skip. Log every skip; a skip that should have been real work is your signal the key is too coarse.
• The provider does not support idempotency keys. For a provider without them, you cannot fully close the money gap. Reduce the exposure by making the durable claim the very last thing before the call and the very first thing checked, and add reconciliation: a scheduled job that compares your records against the provider's and flags mismatches. Reconciliation is the backstop when prevention has a residual gap, especially for M-Pesa-style flows discussed in why retrying an M-Pesa transaction instantly fails.

The mental shift that fixes this for good is to stop treating a duplicate delivery as an error to be prevented and start treating it as a normal event to be absorbed. Your queue is not broken. It is honest. Design the handler so that running twice is indistinguishable from running once, and the 2am "I was charged twice" email stops arriving.