Bolting a phone onto ERPNext: custom routes, guest endpoints and HMAC auth in Frappe

The SMS gateway on the till phone posts to three fixed paths. That is a deliberate constraint: the operator configures a base URL and nothing else, because the fewer things there are to type into a phone in a shop, the fewer support calls there are later.

POST  /api/sms/incoming
POST  /api/sms/heartbeat
GET   /api/app/version

Point that at a FastAPI service and you write three route decorators and go home. Point it at ERPNext and the first request 404s, because Frappe has opinions about what lives under /api/, and none of them are yours.

This article is how the real integration works, in a shipping UpeoRetail installation. The routing trick is worth knowing on its own, and the auth model is worth arguing about.

Why the path 404s

Frappe's request handling checks for the /api/ prefix early and dispatches to its own API layer before website routing is consulted. That layer recognises a small, fixed set of shapes: /api/method/<dotted.path>, /api/resource/<DocType>, and the versioned /api/v1 and /api/v2 namespaces. Anything else under /api/ is not a website route that Frappe has not found yet. It is a malformed API call, and it is rejected as one.

Which means there is no place to put a www page, no website_route_rules entry, and no clever @frappe.whitelist decorator that makes /api/sms/incoming reachable. The path never gets far enough for any of those to be consulted.

The rewrite, and the cache you will miss

The hook is registered in hooks.py:

before_request = [
    "upeoretail.api.sms_gateway.route_sms_gateway",
]

And the function itself is short, which makes the one non-obvious line easy to spot:

GATEWAY_ROUTES = {
    "/api/sms/incoming": "upeoretail.api.sms_gateway.incoming",
    "/api/sms/heartbeat": "upeoretail.api.sms_gateway.heartbeat",
    "/api/app/version": "upeoretail.api.sms_gateway.app_version",
}


def route_sms_gateway():
    req = getattr(frappe.local, "request", None)
    if req is None:
        return
    path = (req.path or "").rstrip("/")
    target = GATEWAY_ROUTES.get(path)
    if not target:
        return
    new_path = "/api/method/" + target
    req.environ["PATH_INFO"] = new_path      # what the URL map matches
    req.__dict__["path"] = new_path          # bust the cached property
    req.__dict__.pop("full_path", None)

Setting PATH_INFO is the obvious half. It is the WSGI environment value that the URL map actually matches against, so without it nothing routes anywhere.

The second and third lines are the half that costs an afternoon. In werkzeug, request.path is a cached property: the first read computes it from the environment and stores the result on the instance. Frappe's init_request runs before before_request hooks and reads the path along the way, so by the time your hook runs the value is already memoised. Change PATH_INFO alone and you get a request whose environment says one thing and whose .path says another, and the resulting behaviour depends entirely on which of the two any given piece of downstream code happens to consult. Writing straight into req.__dict__ replaces the cached value. Popping full_path clears the sibling cache that would otherwise still carry the old value with its query string.

Reaching into an object's __dict__ is not something to do casually. It is justified here because the alternative is a request in two minds about its own URL, and because the three lines sit together with a comment explaining exactly what each one is for. If a future werkzeug stops caching, the lines become harmless no-ops rather than breakage.

Note also the allowlist shape. The hook does not pattern-match a prefix, it looks up an exact path in a three-entry dictionary and returns immediately for everything else. A before_request hook runs on every single request that hits the site, including every desk page load, so it has to be nearly free and it has to be impossible to accidentally capture a path that belongs to somebody else.

Escaping the JSON envelope

Frappe wraps whitelisted return values in its own structure and maps exceptions onto its own status codes. That is helpful when the caller is Frappe's own client, and it is a problem when the caller is a phone app implementing a contract that was written before ERPNext was chosen as the backend.

The contract says a duplicate returns 200 with a body of {"status": "duplicate"}. It does not say 200 with {"message": {"status": "duplicate"}}, and the phone will not unwrap it. So the endpoints construct their own responses:

def _json(payload, status=200):
    from werkzeug.wrappers import Response

    return Response(json.dumps(payload), status=status,
                    content_type="application/json")

Every return path in the module goes through that helper, which has a pleasant side effect: the endpoint reads as a straight sequence of guard clauses that each return a status code, rather than a mix of frappe.throw calls whose HTTP mapping you have to remember.

row, secret = _device_and_secret(payload.get("device_id"))
if not secret:
    return _json({"status": "error", "detail": "unknown device"}, 401)

# 1) integrity - recompute message_hash
if _message_hash(payload["sender"], payload["message"],
                 payload["received_at"]) != payload["message_hash"]:
    return _json({"status": "error", "detail": "message_hash mismatch"}, 400)

# 2) signature
if not hmac.compare_digest(_sign(_incoming_sts(payload), secret),
                           str(payload["signature"])):
    return _json({"status": "error", "detail": "invalid signature"}, 401)

Compare that with a frappe.throw(..., frappe.PermissionError), which gets you a 403, an HTML error page in some configurations, and a traceback in the error log for something that is a routine authentication failure rather than an exception.

Guest endpoints that are not unauthenticated

All three endpoints are declared like this:

@frappe.whitelist(allow_guest=True, methods=["POST"])
def incoming(**payload):
    ...

That reliably makes people uncomfortable, and the discomfort is worth unpacking, because it comes from conflating two different things.

allow_guest=True means Frappe will not require a session cookie or an API key before dispatching to the function. It does not mean the function is unauthenticated. It means Frappe is not the thing doing the authenticating.

What authenticates a request here is the HMAC contract: a signature over a byte-exact canonical string computed with a per-device secret, a sent_at inside a five-minute window, and a nonce that has never been used. A caller with none of those gets a 401 on the first check. A caller who somehow captured a valid request off the wire gets a 409 on the nonce.

Session auth would actually be worse here. A session means a credential that lives on the phone, gets refreshed, expires at inconvenient moments and can be replayed for its whole lifetime if stolen. The shared secret never crosses the wire at all: it is only ever used to compute a signature, so possession is proved without disclosure. The endpoint is guest-accessible and strongly authenticated at the same time, and those two statements are not in tension.

The one thing this does require is that the secret is stored properly. It is a Password fieldtype on the device DocType, which Frappe keeps out of the main table and encrypted at rest, and it is read through the dedicated helper:

def _device_and_secret(device_id):
    """Return (device_row, raw_secret) for an enabled device, else (None, None)."""
    if not device_id:
        return None, None
    row = frappe.db.get_value(DEVICE_DT, device_id, ["name", "enabled", "branch"],
                              as_dict=True)
    if not row or not cint(row.enabled):
        return None, None
    secret = get_decrypted_password(DEVICE_DT, device_id, "device_secret",
                                    raise_exception=False)
    if not secret:
        return None, None
    return row, secret

Three failure shapes collapse into one return value on purpose. Unknown device, disabled device and missing secret all produce (None, None) and the same 401. An attacker probing device ids learns nothing about which ones exist, and the operator disabling a stolen phone gets an instant, total cut-off with a single checkbox.

Three DocTypes

The data model is deliberately small.

The device DocType carries a health block that the heartbeat writes into on every beat: last heartbeat, app version, connectivity, last SMS timestamp, battery, charging state, and the pending and failed queue depths reported by the phone. That means an ordinary Frappe list view of devices, with no custom page and no chart, is a usable fleet monitor. Somebody in the office can see which till has gone quiet, which one is on 4% battery, and which one has 200 messages stuck in its queue.

The heartbeat writes with update_modified=False, which is a small thing that matters at fleet scale. Without it, a device beating every five minutes touches the modified timestamp 288 times a day and makes every version-tracking and sync mechanism in Frappe think the record changed constantly.

Parse inline, then parse again on a schedule

The accept path stores the raw message and then tries to make sense of it, and the ordering there is a rule rather than a convenience:

doc.insert(ignore_permissions=True)

# Best-effort inline parse; the scheduled backstop retries anything left
# in "Received". A parse failure must never fail the accept.
try:
    parse_message(doc)
except Exception:
    frappe.log_error(title="Upeo SMS gateway parse failed", message=frappe.get_traceback())

frappe.db.commit()
return _json({"status": "accepted", "name": doc.name}, 200)

A parse failure must never fail the accept. If the parser throws, the phone must still be told the message was received, because otherwise the phone retries, and the retry hits the same broken parser, and now a bug in a regular expression is causing a message to bounce between a phone and a server forever. The SMS is safely stored; the parse is a separate concern that can be fixed and re-run.

Which is what the scheduled backstop does:

def process_pending_messages(limit=200):
    """Backstop: parse any messages still in 'Received' (inline parse missed)."""
    names = frappe.get_all(
        MESSAGE_DT, filters={"status": "Received"}, pluck="name",
        order_by="creation asc", limit=cint(limit),
    )
    for name in names:
        doc = frappe.get_doc(MESSAGE_DT, name)
        try:
            parse_message(doc)
        except Exception:
            doc.db_set("status", "Error", update_modified=False)
            doc.db_set("parse_error", frappe.get_traceback(with_context=False)[:500],
                       update_modified=False)
            frappe.log_error(...)
        frappe.db.commit()
    return {"processed": len(names)}

Anything still sitting in Received gets another go. If it fails again it moves to Error with a truncated traceback stored on the row itself, which is the difference between "somebody will find this in the error log eventually" and "there is a filtered list view of messages that need a human". The commit is inside the loop so one poisonous message does not roll back the batch.

The parser itself is conservative on purpose. It only treats a message as a payment if the body contains both "received" and "confirmed", so balance messages, debit notifications and marketing SMS end up as Ignored rather than being coerced into a payment with a zero amount:

low = text.lower()
if "received" not in low or "confirmed" not in low:
    return None

It is not a complete parser for every Safaricom message format and it is not trying to be. It covers the credit confirmations that a till actually receives and deliberately ignores the rest, which is a much easier thing to keep correct than a parser that claims to handle everything.

The punchline: one payment table, two sources

Here is the part that makes the whole integration worth building rather than bolting a second payment system onto the side of the first.

name, _created = mpesa.upsert_mpesa_payment(
    transaction_code=parsed["code"],
    amount=parsed["amount"],
    phone_number=parsed.get("phone"),
    payer_name=parsed.get("name"),
    transaction_time=parsed.get("time"),
    raw_payload=doc.message,
    source="SMS Gateway",
    gateway_message=doc.name,
)

That function is not part of the SMS gateway. It is the same function the Daraja webhook calls, and it deduplicates on a unique transaction code:

def upsert_mpesa_payment(transaction_code, amount=0, ...):
    """Idempotently create an Upeo Retail M-Pesa Payment.

    Shared by the Daraja webhook and the SMS-gateway parser. Dedups on the
    unique transaction_code; returns (name, created).
    """
    if not transaction_code:
        frappe.throw(_("transaction_code is required."))
    if frappe.db.exists(MPESA_DT, transaction_code):
        return transaction_code, False
    ...

So a shop can start on the SMS gateway, get approved for Daraja six months later, and run both simultaneously through the transition without producing a single duplicate payment, because the transaction code is the transaction code no matter which door it came through. Reconciliation, matching against invoices, the whole downstream flow: one code path, one set of bugs, one thing to test. The source field records which door, which is useful for reporting and irrelevant to correctness.

That design decision is the subject of its own article, because it is the reason the phone stays dumb.

Housekeeping

Two scheduled jobs keep the thing from growing without bound. process_pending_messages runs on a cron entry as the parse backstop. prune_old_nonces runs daily and drops consumed nonces well past the five-minute freshness window:

def prune_old_nonces(hours=24):
    """Daily: drop consumed nonces well past the freshness window."""
    from frappe.utils import add_to_date

    cutoff = add_to_date(now_datetime(), hours=-cint(hours))
    frappe.db.delete(NONCE_DT, {"created_at": ["<", cutoff]})
    frappe.db.commit()

The replay ledger only has to remember a nonce for longer than the window in which a stale timestamp would be rejected anyway. Keeping 24 hours against a 5-minute window is a margin of nearly three hundred, which costs almost nothing and removes any chance of the two schedules interacting badly.

What I would do differently

The path rewrite is clever, and clever is a warning sign. It works, it is commented, and it is three lines, but it depends on an implementation detail of when Frappe reads the request path and on werkzeug caching a property. Both could change in a version bump, and the failure would be a 404 on a payment endpoint, discovered by a shop rather than by a test. If I were starting again I would push harder on the phone side to make the base URL include the full method path, so that the operator types a longer URL once and no rewrite is needed at all. The reason it works this way is that the contract was written against a generic backend before ERPNext was the target, which is a very ordinary way for a wart to happen.

The endpoints also have no rate limiting of their own. They are cheap to reject, the first thing they do is a device lookup, and the signature check comes before any write. But an unauthenticated caller can still make Frappe do a database read per request, and the only thing standing between that and a bad afternoon is the reverse proxy. A per-device counter in the nonce table would be a natural place to add it.

And I would add a test that asserts the canonical string on the Python side against the same literal the Dart test uses. Today the two implementations are kept in step by a comment saying they must be. That has held, because the module is small and the comment is loud. It is still weaker than the Dart side, which has a test asserting the exact bytes, and the asymmetry is not a good look.