Frappe Socket.IO Not Working in Production: Diagnose Before You Reinstall

The invoice submits, but the little realtime notification never appears. The bulk import runs, but the progress bar sits at zero and then the page just reloads to reveal it finished minutes ago. In development everything updated live. In production, realtime is dead, and the browser console has a lonely error about failing to connect to a socket.

Open any Frappe forum thread on this and you will be told to run the ritual: bench setup socketio, bench setup supervisor, bench setup nginx, reload everything, pray. That is not diagnosis, it is a blind full reinstall, and it wastes an hour rebuilding config that was probably fine while never telling you what was actually wrong. Realtime in Frappe production breaks for one of about three specific reasons, and each has a specific check.

You can find the real cause in five minutes. The chain is short: browser, to nginx, to the socketio Node process, on port 9000. Something in that chain is broken. Walk it in order and it will tell you where.

The symptom in the browser

Open devtools, go to the Network tab, filter by WS or search for socket.io, and reload. A working setup shows a socket.io request that upgrades to a WebSocket (status 101) and stays open. A broken one shows repeated failed polling requests, often to the wrong host or port, with an error like:

GET http://localhost:9000/socket.io/?EIO=4&transport=polling net::ERR_CONNECTION_REFUSED
socket.io Failed to load resource: the server responded with a status of 502

The exact error tells you a lot already: a request to localhost:9000 from the browser means the client is trying to reach the socketio port directly instead of going through nginx, and a 502 means nginx tried to proxy but the upstream did not answer. Hold onto whichever one you see.

Step 1: is the socketio process even running?

This is the fastest check and a frequent culprit, so do it first.

sudo supervisorctl status

Look for the socketio process by name:

frappe-bench-web:frappe-bench-frappe-web              RUNNING
frappe-bench-workers:frappe-bench-frappe-default-worker RUNNING
frappe-bench-node-socketio                            STOPPED   Jul 24 01:02

If frappe-bench-node-socketio is anything other than RUNNING, that is your answer. Read the log it writes before you restart it, because a process that keeps dying will just die again:

tail -50 ~/frappe-bench/logs/node-socketio.error.log

The most common reason this process is FATAL is that supervisor is invoking the wrong Node binary, for example a system /usr/bin/node that is too old, instead of the newer Node the bench expects. If the log complains about syntax or an unsupported feature, fix the node path in the supervisor config and re-run bench setup supervisor, which is a targeted use of that command rather than the blind ritual.

Step 2: is the backend answering on port 9000?

Bypass nginx entirely and hit the socketio process directly on the server. This isolates backend from proxy in one command.

curl "http://localhost:9000/socket.io/?EIO=4&transport=polling"

A working backend returns a short handshake payload that starts with a number and a JSON object containing a session id, something like 0{"sid":"...","upgrades":["websocket"],...}. If you get that, the socketio process is healthy and the problem is upstream of it, in nginx or the network, so move to step 3. If you get Connection refused, the process is not listening, and you are really still in step 1.

The port comes from your bench config. Confirm it if 9000 is not answering:

grep socketio_port ~/frappe-bench/sites/common_site_config.json
# "socketio_port": 9000

Step 3: does nginx proxy /socket.io with the Upgrade headers?

A WebSocket connection starts as HTTP and then upgrades. nginx will not pass that upgrade through unless you explicitly tell it to. The Frappe-generated nginx config includes a block for exactly this; if yours is missing it or it was overwritten, realtime breaks with 502s or silent fallback. The block should look like this:

upstream frappe-bench-socketio-server {
    server 127.0.0.1:9000 fail_timeout=0;
}

server {
    # ... your existing TLS server block ...

    location /socket.io {
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header X-Frappe-Site-Name $host;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_pass http://frappe-bench-socketio-server;
    }
}

The three lines that matter most are proxy_http_version 1.1, proxy_set_header Upgrade $http_upgrade, and proxy_set_header Connection "upgrade". Without them nginx speaks HTTP/1.0 or drops the upgrade, and the socket can never open. After any edit, test and reload, exactly as you would for any nginx change (the discipline is the same one in Nginx Reverse Proxy, TLS and Rate Limiting):

sudo nginx -t && sudo systemctl reload nginx

Step 4: is anything blocking the path?

If the process is up, the backend answers on 9000, and nginx has the block, the connection is being blocked somewhere in between.

• Firewall: port 9000 should not be open to the world; the browser talks to 443 and nginx talks to 9000 over localhost. But a host firewall or cloud security group that blocks localhost traffic, or a misconfigured rule, can sever the nginx-to-socketio hop. Confirm nginx can reach 9000 with the curl from step 2, run from the same host.
• Docker: if Frappe runs in containers, nginx and socketio may be in different containers, so 127.0.0.1:9000 in the nginx config points at nginx's own container, not the socketio one. The upstream must use the socketio service name on the shared Docker network, not localhost. This is the single most common Docker-specific cause.
• Wrong host in the browser: if the browser is trying to reach localhost:9000 directly, the site's socketio_port or host config is telling the client to connect to the raw port instead of going through the proxy. In production the client should connect to the same origin as the site, and nginx routes /socket.io internally.

How to verify realtime is fixed

1. Reload the app in the browser with devtools open on the Network tab. You should now see a socket.io request return status 101 Switching Protocols and stay in a pending, connected state. That 101 is the WebSocket handshake succeeding.
2. Trigger something realtime, a bulk import or a background job, and watch the progress bar move live without a manual refresh. If the bar animates, the socket is delivering events.
3. From the server, confirm the process is stable, not flapping:

   sudo supervisorctl status frappe-bench-node-socketio
   # frappe-bench-node-socketio   RUNNING   pid 20481, uptime 0:05:12

   Rising uptime across a few checks means it is not crash-looping.

What people get wrong

The blind reinstall ritual. Running bench setup socketio && bench setup supervisor && bench setup nginx and reloading is the forum default, and it occasionally works by regenerating a config that had drifted. But it teaches you nothing, it can overwrite customisations you made to nginx, and if the real cause was a blocked port or a Docker networking issue, it fixes nothing at all while costing you an hour. Diagnose the three links first; only regenerate the specific config that a check proved is wrong.

Opening port 9000 to the internet. Some guides tell you to open 9000 in the firewall. Do not. In a correct production setup the browser never touches 9000; it connects to 443 and nginx proxies internally over localhost. Exposing 9000 publicly is an unnecessary attack surface and is not what fixes realtime.

Assuming HTTPS is the problem. Realtime over a proxied WebSocket works fine behind TLS as long as the Upgrade headers are passed. If you disabled SSL to test and it started working, you did not prove TLS was the issue, you proved you changed the nginx config path being used, which likely restored the /socket.io block. Keep TLS on and fix the block.

When it is still broken

• 101 handshake succeeds but no events arrive. The transport is fine but the app is not emitting or subscribing. Check that the Redis instance Frappe uses for realtime is running, since socketio relays events through Redis, and a stopped redis-socketio means an open socket with nothing to send.
• Works on the apex domain but not a subdomain or behind Cloudflare. A CDN or extra proxy in front of nginx must also pass WebSocket upgrades. Enable WebSocket support at that layer too.
• Intermittent disconnects. A short proxy_read_timeout can drop idle sockets. Raise it on the /socket.io location if connections die after a fixed interval.
• Assets or the whole desk also misbehave. If more than realtime is off, the problem may be broader than socketio; start with Frappe / ERPNext Assets not Loading Properly and the production setup in Setup ERPNext for Production.

Realtime failures feel mysterious because they are silent, but the chain is short and each link answers a single yes-or-no question. Walk it, and you will fix the one thing that is actually broken instead of rebuilding the four things that were fine.