502 Bad Gateway: It Is Almost Never Nginx

There is a ritual I have watched in three different offices. The site returns 502 Bad Gateway, and somebody restarts Nginx. It does not help. They restart it again with -s reload. Then they restart the whole server, the site comes back for eleven minutes, and everyone agrees the server "needed a reboot".

Here is the thing worth internalising: a 502 is Nginx reporting that your application failed it. Nginx accepted the request, tried to hand it to an upstream, and either could not connect or did not get a usable response back. The web server is the messenger. In several years of doing this I can count on one hand the times the fault was actually in Nginx, and each of those was a configuration error I had made myself.

The good news is that Nginx tells you precisely what happened, in one line, in a file most people never open.

sudo tail -n 30 /var/log/nginx/error.log

The four causes, and the line that identifies each

1. Nothing is listening

2019/03/19 09:14:02 [error] 1187#1187: *4 connect() failed (111: Connection refused)
while connecting to upstream, client: 41.90.x.x, server: app.example.co.ke,
request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:8000/", host: "app.example.co.ke"

Or, for a Unix socket that is not there at all:

connect() to unix:/run/gunicorn/app.sock failed (2: No such file or directory)
while connecting to upstream

The application is down, or it is up and listening somewhere else. Check both halves:

systemctl status app.service
sudo ss -lntp | grep 8000
ls -l /run/gunicorn/app.sock

A frequent variant: the app binds 127.0.0.1 and Nginx proxies to localhost, which resolves to the IPv6 address ::1 first. Connection refused, application perfectly healthy. Use the literal 127.0.0.1 in proxy_pass, or bind both.

2. Permission denied on the socket

connect() to unix:/run/gunicorn/app.sock failed (13: Permission denied)
while connecting to upstream

The socket exists and the Nginx worker cannot open it. Sockets obey ordinary file permissions, and Nginx workers run as www-data. Look at the whole path, not just the socket, because a directory the worker cannot traverse produces the same error:

namei -l /run/gunicorn/app.sock

The fix belongs in the application's service definition, where the socket is created, not in a chmod you run by hand and lose on the next restart. For gunicorn under systemd:

[Service]
User=app
Group=www-data
RuntimeDirectory=gunicorn
ExecStart=/srv/app/venv/bin/gunicorn --workers 3 --umask 007 \
          --bind unix:/run/gunicorn/app.sock app.wsgi:application

RuntimeDirectory=gunicorn makes systemd create /run/gunicorn with the right ownership on every start, and recreate it after a reboot clears /run. The --umask 007 makes the socket group readable and writable, and the shared group www-data does the rest.

3. The upstream died mid-request

upstream prematurely closed connection while reading response header from upstream

Nginx connected fine, sent the request, and the other end hung up before sending a complete response. Three usual reasons: the application server's own worker timeout fired, the kernel killed the process for memory, or your code crashed hard enough to take the worker with it.

sudo journalctl -u app.service --since "10 minutes ago"
dmesg | grep -i "killed process"

If gunicorn logs WORKER TIMEOUT, a request exceeded its timeout, which defaults to thirty seconds, and gunicorn killed its own worker. Raising the timeout is occasionally right and usually a way of hiding an eight second query. If dmesg shows the kernel killing your process, the machine is out of memory and no Nginx setting will help.

This is also where the difference between 502 and 504 becomes useful. A 504 is Nginx giving up after proxy_read_timeout, sixty seconds by default. A 502 with "prematurely closed" means somebody else gave up first, and that somebody is usually your own application server, configured with a shorter timeout than Nginx.

4. The headers are too big

upstream sent too big header while reading response header from upstream

Nginx buffers response headers in a fixed sized buffer, and a large session cookie or a long redirect chain overflows it. The application is fine; the buffer is small:

proxy_buffer_size   16k;
proxy_buffers       8 16k;
proxy_busy_buffers_size 32k;

Before you paste that, ask why your headers are that large. It is almost always an oversized cookie, and putting a session's worth of JSON in a cookie is a problem you will meet again elsewhere.

The 502 that only happens during a deploy

A special case worth naming, because it is so common that people stop reporting it. Every deploy produces thirty seconds of 502s, someone says "it is just the restart", and eventually a customer is the one who sees it.

The cause is that your restart stops the old process before the new one is listening. For the length of that gap, Nginx has nowhere to send requests and answers 502 to everybody. Three ways out, in increasing order of effort:

• Retry at the proxy. proxy_next_upstream error timeout non_idempotent; with more than one upstream defined lets Nginx try the other one. This needs two application instances to be worth anything.
• Reload rather than restart. Gunicorn reloads its workers on SIGHUP, one at a time, keeping the listening socket open. systemctl reload does this when the unit defines ExecReload=/bin/kill -s HUP $MAINPID.
• Hand the socket to systemd. A socket unit holds the listening socket itself and queues connections while the service restarts, so requests arriving mid-deploy wait instead of failing.

The first one is a plaster, the second is usually enough, and the third is the one that makes deploy time 502s genuinely impossible.

Verification

Prove the upstream works without Nginx in the way. If it answers here and 502s through Nginx, the fault is between them, which is permissions or address:

curl -i http://127.0.0.1:8000/health
sudo -u www-data curl -i --unix-socket /run/gunicorn/app.sock http://localhost/health

The second command is the one that settles arguments, because it makes the request as the same user Nginx runs as. Then, after any change:

sudo nginx -t && sudo systemctl reload nginx
curl -o /dev/null -s -w '%{http_code}\n' https://app.example.co.ke/

Finally, reboot. A socket in /run that nothing recreates at boot is the classic 502 that appears weeks later, when the machine restarts for an unrelated reason and nobody connects the two events.

What people get wrong

chmod 777 on the socket. It makes the error go away and it makes every local user able to talk directly to your application, bypassing whatever Nginx was doing in front of it. Use a shared group.

Raising every timeout until the error stops. Now the user waits three minutes instead of thirty seconds for the same failure, and your workers are all occupied doing it. Timeouts are a symptom. Look at what is slow.

Adding more gunicorn workers on a small VPS. Each worker is a full copy of your application in memory. On a 2GB machine, going from three workers to twelve turns intermittent 502s into constant ones, because now the kernel is killing them. The rough starting point of two workers per core exists for a reason.

Blaming Nginx. If restarting Nginx appears to help, the real cause was a stale configuration or a socket path that changed under it. It will return.

When it is still broken

• Intermittent 502s under load only. You are running out of workers or of file descriptors. Check the application's queue depth, then LimitNOFILE in the unit and worker_connections in Nginx.
• 502 only on one route. That route crashes the worker. Reproduce it with curl against the upstream directly and read the traceback.
• Nothing in error.log at all. Then Nginx is not the one returning the 502. A load balancer or CDN in front of it is, and its logs are the ones you need.
• Managed stacks such as a Frappe bench. The processes are under supervisor, not systemd, so the equivalent commands are supervisorctl status and the per process logs in the bench's logs/ directory, as in the ERPNext install notes. The four phrases above mean exactly the same things.