Your M-Pesa Callback Will Fire Twice: Idempotency, Retries and the 1037 Problem

The M-Pesa integration that works perfectly in your staging environment will fail in production, and it will fail in a specific, boring, expensive way: a customer will be charged and your system will not know, or a customer will be charged once and your system will credit them twice.

The root cause is almost never your code being wrong. It is your code assuming the callback is a reliable, exactly-once, ordered delivery channel. It is none of those things.

The thesis: Daraja callbacks are an unreliable notification, not a source of truth. Your system must be able to arrive at the correct balance even if every callback is lost, duplicated, or delivered out of order. Everything below follows from that.

Failure mode 1: the callback simply never arrives

Safaricom does not run an aggressive retry queue like Stripe does. If your endpoint is slow, returns a non-200, is behind a firewall that drops their IP range, or your TLS chain is incomplete for their client, the notification is gone. There is no dashboard where you can replay it.

The fix is not a better callback handler. The fix is a poller.

import time
from datetime import datetime, timedelta, timezone

def reconcile_pending_stk(db, daraja):
    cutoff = datetime.now(timezone.utc) - timedelta(seconds=90)
    rows = db.query("""
        SELECT id, checkout_request_id, attempts
        FROM payment_intents
        WHERE status = 'PENDING'
          AND created_at < %s
          AND attempts < 8
        ORDER BY created_at
        LIMIT 200
    """, [cutoff])

    for row in rows:
        res = daraja.stk_query(row["checkout_request_id"])
        code = res.get("ResultCode")

        # Still in flight - Daraja returns errorCode 500.001.1001 for this.
        if res.get("errorCode") == "500.001.1001":
            db.execute("UPDATE payment_intents SET attempts = attempts + 1 WHERE id = %s", [row["id"]])
            continue

        if code is None:
            continue

        apply_result(db, row["id"], int(code), res.get("ResultDesc"), source="query")

The backoff matters. Query too eagerly and you get 500.001.1001 forever. I use 90 seconds, then 3 minutes, then 10, then hourly out to 24 hours before marking the intent UNKNOWN and routing it to a human queue.

Failure mode 2: ResultCode 1037 is not a failure

This is the one that costs real money. 1037 is "DS timeout - user cannot be reached". Everyone maps it to FAILED. But 1037 means Safaricom's dialogue service gave up waiting; it does not always mean the debit did not happen. On congested days I have seen 1037 callbacks followed, minutes later, by a real C2B confirmation for the same amount from the same MSISDN.

Treat these codes as terminal-failure: 1 (insufficient funds), 1032 (cancelled by user), 2001 (wrong PIN), 1019 (expired). Treat 1037 and 1001 (unable to lock subscriber) as indeterminate: keep the intent open, keep querying, and do not let the user retry into a second STK push until you have resolved it. That last part is what stops double charges.

Failure mode 3: duplicates

Duplicates happen for mundane reasons. Load balancers retry. Safaricom occasionally redelivers. Your own reconciler races the callback and both try to settle the same payment. Your worker crashes after writing the ledger entry but before marking the job done.

Do not solve this with an application-level "have I seen this before?" check. That is a read-then-write race and it will lose. Solve it in the database, with a constraint.

CREATE TABLE payment_events (
    id              BIGSERIAL PRIMARY KEY,
    provider        TEXT        NOT NULL,
    event_kind      TEXT        NOT NULL,
    provider_ref    TEXT        NOT NULL,   -- MpesaReceiptNumber or TransID
    intent_id       BIGINT      REFERENCES payment_intents(id),
    amount_minor    BIGINT      NOT NULL,   -- cents. never a float.
    currency        CHAR(3)     NOT NULL DEFAULT 'KES',
    occurred_at     TIMESTAMPTZ NOT NULL,
    raw             JSONB       NOT NULL,
    created_at      TIMESTAMPTZ NOT NULL DEFAULT now()
);

-- The whole article, in one line:
CREATE UNIQUE INDEX payment_events_dedupe
    ON payment_events (provider, event_kind, provider_ref);

Then let the insert decide whether this is the first time:

def settle(conn, ev):
    with conn.transaction():
        cur = conn.execute("""
            INSERT INTO payment_events
                (provider, event_kind, provider_ref, intent_id,
                 amount_minor, currency, occurred_at, raw)
            VALUES (%(provider)s, %(kind)s, %(ref)s, %(intent_id)s,
                    %(amount_minor)s, %(currency)s, %(occurred_at)s, %(raw)s)
            ON CONFLICT (provider, event_kind, provider_ref) DO NOTHING
            RETURNING id
        """, ev)

        row = cur.fetchone()
        if row is None:
            return "duplicate"      # already applied. do nothing. return 200.

        post_ledger_entries(conn, event_id=row[0], ev=ev)
        conn.execute(
            "UPDATE payment_intents SET status = 'PAID', paid_at = %s WHERE id = %s",
            [ev["occurred_at"], ev["intent_id"]],
        )
        return "applied"

The ledger write and the dedupe insert are in the same transaction. If the process dies between them, nothing happened. If a duplicate arrives, the insert conflicts and returns nothing. There is no window.

One caveat: MpesaReceiptNumber is your natural key for STK and C2B. For B2C use TransactionID, and set OriginatorConversationID yourself so a retried request maps to the same payout.

Failure mode 4: the amount is not what you asked for

With C2B, the customer types the amount. They will underpay, overpay, and pay to the wrong account reference. With STK you specify the amount, but the callback is still authoritative. Never assume they match.

from decimal import Decimal

def to_minor(value) -> int:
    # Daraja sends Amount as a JSON number in STK and a string in C2B.
    return int((Decimal(str(value)) * 100).to_integral_value())

paid  = to_minor(items["Amount"])          # e.g. 149950
owed  = intent["amount_minor"]

if paid == owed:
    fulfil(intent)
elif paid < owed:
    record_partial(intent, paid)           # do NOT fulfil
else:
    fulfil(intent)
    record_credit(intent, paid - owed)     # overpayment becomes a liability

If you ever wrote float(amount), go and fix it now. 0.1 + 0.2 is not 0.3, and in a reconciliation report against a Safaricom statement that discrepancy will haunt you for weeks.

Failure mode 5: nothing is signed

Daraja sends no HMAC signature and no shared secret header. Anyone who learns your callback URL can POST a fabricated success payload. I have seen this exploited on a Kenyan e-commerce site whose callback path was /mpesa/callback and was linked from a public GitHub repo.

Defence in depth:

1. Use an unguessable path segment: /hooks/c/9f3a2c81-collections. Not secret, but not scrapeable either.
2. Allowlist Safaricom's egress ranges at the edge. Nginx example below.
3. Re-verify every callback against stkpushquery or Transaction Status before releasing anything expensive. Yes, it doubles your API calls. It also makes a forged callback worthless.

location /hooks/c/ {
    allow 196.201.214.0/24;
    allow 196.201.213.0/24;
    deny  all;

    proxy_read_timeout 10s;
    proxy_pass http://app_upstream;
}

Confirm the current ranges with Safaricom support before you deploy this - they change, and a stale allowlist is an outage that looks exactly like "callbacks stopped working".

Failure mode 6: slow handlers

Your callback handler must do three things and nothing else: read the body, write a raw row, return 200. No sending SMS. No calling KRA eTIMS. No generating PDFs. No third-party HTTP at all.

// Express. The whole handler. Under 20ms p99.
app.post("/hooks/c/9f3a2c81-collections", express.json(), async (req, res) => {
  try {
    await db.query(
      `INSERT INTO raw_webhooks (provider, path, body, received_at)
       VALUES ($1, $2, $3, now())`,
      ["daraja", req.path, req.body]
    );
  } catch (err) {
    logger.error({ err }, "raw webhook persist failed");
  }
  // 200 regardless. A retry we cannot service is worse than a lost log line.
  res.json({ ResultCode: 0, ResultDesc: "Accepted" });
});

A separate worker reads raw_webhooks and does the real work. This gives you something else that is priceless: replay. When you find a bug in your settlement logic six weeks later, you can re-run every raw payload through the fixed code path.

Failure mode 7: timestamps

TransactionDate arrives as the integer 20191219102115. That is yyyyMMddHHmmss in Africa/Nairobi (UTC+3), with no timezone marker. If your server runs UTC and you parse it naively, every transaction is stamped three hours early. Daily cut-offs break. Month-end reports disagree with Safaricom's. Fix it once, at the edge:

def parse_daraja_ts(value) -> datetime:
    return datetime.strptime(str(value), "%Y%m%d%H%M%S").replace(
        tzinfo=ZoneInfo("Africa/Nairobi")
    )

B2C uses dd.MM.yyyy HH:mm:ss instead. Same timezone, different format. Write two parsers and a test.

The invariant to hold onto

Write this on a sticky note above your monitor:

A payment is settled when a row exists in payment_events with a unique provider reference and matching ledger entries. Not when a callback arrives. Not when the HTTP response said ResponseCode 0. Not when the user saw a green tick.

Callbacks become a latency optimisation - a way to settle in two seconds instead of ninety. When they fail, you get slower, not wrong. That is the only design that survives a Safaricom incident, and Safaricom has incidents.

What to instrument

• Ratio of intents settled by callback versus by poller. A rising poller share is your early warning.
• Count of ON CONFLICT DO NOTHING hits per hour. Steady is fine. A spike means something upstream is retrying.
• Age of the oldest PENDING intent. Alert past 10 minutes.
• Count of intents in UNKNOWN. This should be near zero. If it is not, you have a manual reconciliation problem, and the next article is about exactly that.