The Webhook That Charged a Customer Twice: Timeout-Induced Duplicate Processing

The demo went perfectly. You wired up the payment webhook, ran a test charge, watched the order flip to paid, and shipped it. For three weeks nothing happened. Then a customer emails to say they were charged twice for one subscription, and when you open the logs the same event ID appears twice, four seconds apart, both processed cleanly.

Nothing is broken. Your code did exactly what you told it to. The problem is that you built the handler as if a webhook arrives exactly once, and webhooks do not work that way. Every serious provider, Stripe, Paystack, GitHub, M-Pesa, guarantees at-least-once delivery, which is a polite way of saying "we will sometimes send this twice and that is your problem to handle."

This is the bug that cannot be reproduced in development, because in development your handler answers in 40 milliseconds and the provider never has a reason to retry. The duplicate path only runs when something is slow, and things are only slow in production.

The symptom: the same event ID processed twice, seconds apart

There is usually no error message at all, which is what makes this so nasty. If you log the event ID on entry you will see something like this:

12:04:11.201  webhook received  evt_1P9zXk2eZvKY  charge.succeeded
12:04:11.320  charge recorded   order_8841  amount=4500
12:04:15.640  webhook received  evt_1P9zXk2eZvKY  charge.succeeded   <-- same event
12:04:15.702  charge recorded   order_8841  amount=4500              <-- charged again

Four seconds between the two arrivals is the tell. That is the provider's delivery timeout expiring on the first attempt and the retry landing. The event ID is identical, which is the good news: it means the fix is available to you.

Why this happens: acknowledgement, not delivery, closes the loop

A webhook is not fire-and-forget from the provider's side. When Stripe or Paystack sends you an event, it holds that event in a "pending" state and waits for your endpoint to return a 2xx status inside a timeout. Stripe's timeout is on the order of a few seconds. If the timeout expires, or the TCP connection drops, or you return a 500, or your load balancer returns a 502 because your worker was busy, the provider concludes that delivery failed and schedules a retry with exponential backoff.

Here is the part that trips everyone: the retry is judged on your response, not on whether your code ran. If your handler received the event, created the charge row, called the fulfilment service, and then took eight seconds and the connection timed out, the provider never saw your 200. From its point of view the event was never delivered, so it sends it again. Your side effects already happened. Now they happen twice.

Out-of-order delivery is the same family of problem. Under retries and parallel dispatch, a subscription.updated can land before the subscription.created it depends on. If your handler assumes events arrive in the order they were generated, it will occasionally act on a state that does not exist yet.

The fix, in numbered steps

Step 1: Verify the signature and read the top-level event ID

Before anything else, confirm the request actually came from your provider, using the raw request body and the signature header. If you are not already doing this, stop and fix it first, because an unauthenticated webhook endpoint is a bigger problem than duplicates. I wrote about the exact trap that makes this fail intermittently in your webhook endpoint trusts anyone who can curl it and the closely related body parser order bug.

Once verified, pull out the stable ID. For Stripe this is event.id (the evt_... value), not the object ID inside data and not anything you compute yourself.

const event = stripe.webhooks.constructEvent(rawBody, sig, endpointSecret);
const eventId = event.id; // evt_...  stays identical across all retries

Step 2: Record the event ID atomically, and stop if it is a repeat

Create a table whose entire job is to remember which events you have seen. The unique constraint on the event ID is what does the real work. An INSERT that hits the constraint is your signal that this is a duplicate.

CREATE TABLE processed_webhook_events (
    event_id    text PRIMARY KEY,
    received_at timestamptz NOT NULL DEFAULT now(),
    status      text NOT NULL DEFAULT 'received'
);

// INSERT ... ON CONFLICT DO NOTHING returns 0 rows if we have seen it before
const result = await db.query(
  `INSERT INTO processed_webhook_events (event_id)
   VALUES ($1) ON CONFLICT (event_id) DO NOTHING`,
  [eventId]
);

if (result.rowCount === 0) {
  // Duplicate. We already accepted this event. Acknowledge and stop.
  return res.status(200).send('duplicate ignored');
}

Note that this check and the write live in the same statement. If you do a SELECT then a separate INSERT, two retries arriving at the same instant can both pass the SELECT before either inserts, and you are back where you started. Let the database enforce uniqueness for you.

Step 3: Return 200 immediately, then process on a queue

Now that the event is durably recorded, acknowledge it and hand the slow work to a background worker. This is the step that stops the retries happening at all, because the provider gets its 200 in milliseconds.

await queue.add('process-webhook', { eventId });
return res.status(200).send('accepted');

The worker loads the event, does the charge, fulfilment and email, and flips the row's status to processed. If the worker crashes halfway, the row is still received, and a small reconciliation job can retry it safely, because every step it performs is itself idempotent (upserts keyed on the order ID, not blind inserts).

Step 4: State-machine the consumer for out-of-order events

Do not assume order. Model the valid transitions and reject the ones that do not make sense yet. If subscription.updated arrives for a subscription you have never seen, either fetch the current state from the provider's API, or park the event and let the reconciliation job pick it up once the created event lands. This is the same discipline I argue for in your migration will run twice, write it that way: assume repetition and design for it, rather than hoping the timing holds.

Verification: prove one event produces one side effect

The whole point is that you can now replay an event and nothing bad happens. Test it deliberately.

# Resend a real event from Stripe's CLI to your live handler
stripe events resend evt_1P9zXk2eZvKY

# Then check that the side effect happened exactly once
psql -c "SELECT count(*) FROM charges WHERE order_id = 'order_8841';"
 count
-------
     1

If that count is 1 after two or three resends, you are done. Add it as an automated test so it stays true: post the identical payload twice in your test suite and assert a single row.

What people get wrong

"I will dedupe on a hash of the payload." Tempting, and wrong. Retries can carry slightly different headers or metadata, and some providers re-serialise the body, so the hash changes and your dedupe silently stops working. Use the provider's declared stable event ID, which exists precisely for this.

"I will just make the handler faster so it never times out." This reduces the frequency of the bug without removing it. A GC pause, a slow database, a cold Lambda, or a provider that retries on a 502 from your load balancer will still produce a duplicate eventually. Speed is not correctness. You need the dedupe key regardless.

"I will process first, then insert the event ID at the end so a failed run gets retried." Now a run that succeeds but crashes before writing the ID gets fully reprocessed, and a run that fails midway leaves partial side effects. Record the ID up front and make the individual side effects idempotent instead. The order matters.

When it is still broken

If you have all of the above and still see doubles, check these in order:

• Two endpoints registered. A leftover webhook URL in the provider dashboard (old staging URL, or a duplicate) means every event is genuinely sent to you twice as two separate deliveries. Delete the stale one.
• You are keying on the wrong ID. Confirm you are storing the top-level event.id, not the inner object ID. A single charge object can appear in several distinct events (charge.succeeded, charge.updated), each with its own evt_ id, and each is legitimately a different event.
• Your dedupe table and your side effect are in different databases with no shared transaction, so a crash between them leaves them disagreeing. Keep the received insert and the enqueue in one commit where you can.
• The provider is retrying even after a 200. A few providers retry on connection reset even after you sent the response. Check their dashboard's delivery log to see what status they actually recorded, then trust that over your own logs.

Duplicates are not an edge case you can hope to avoid. They are a guarantee you design around once, and then never think about again.