Certbot Bypasses systemd for Nginx: Why Renewals Leave Them Out of Sync

Here is a failure that will make you doubt your monitoring. Your certificate renews fine at 3am. Certbot exits zero. And then, hours later, the site is down and systemctl status nginx cheerfully reports active (running) while nothing is listening on 443. Or the reverse: nginx is serving traffic perfectly and systemd insists it is dead.

The root of this is a design decision most people never notice: Certbot's nginx plugin does not go through systemd. It drives the nginx binary directly with signals, the same way you would with nginx -s reload. systemd, which is supposed to be the single source of truth for whether nginx is up, gets bypassed and its tracked state drifts away from reality. Mix that with a well-meaning renewal script that also runs systemctl stop nginx, and you have two control paths fighting over the same PID file.

The error you see when the two control paths collide

Hook command "nginx -s stop" returned error code 1
nginx: [error] invalid PID number "" in "/run/nginx.pid"

That invalid PID number "" is the smoking gun. It means something asked the nginx binary to signal the master process using the PID file, but the file was empty or stale, because the process it referred to had already been stopped by the other path (systemd, or a previous signal). Neither side owns the truth, so one of them ends up shouting at a corpse.

Why systemd and Certbot disagree about nginx

When systemd starts nginx, it forks the master process, records that PID as the unit's MainPID, and watches it. If that exact process dies, systemd knows, and (depending on your unit) may restart it. This is the entire point of running under a service manager.

Certbot's nginx installer plugin does not ask systemd to do anything. To apply a renewed certificate it tells the running nginx to re-read its config with a signal, effectively nginx -s reload. A reload is graceful: the master keeps the same PID, spins up new workers with the new certificate, and retires the old workers. As long as everyone only ever reloads, this is fine and nginx never actually goes down.

The trouble starts when a stop enters the picture from a different path. The classic case is someone using the standalone authenticator, which needs port 80 to itself, so they wrote a wrapper:

# The anti-pattern that causes the race
systemctl stop nginx
certbot renew
systemctl start nginx

Now imagine a renewal hook, or a leftover Certbot deploy hook, that also issues nginx -s stop or nginx -s reload. systemd stopped the process and cleared its state, but the hook still tries to signal via the PID file, which systemd's stop left empty. You get invalid PID number "". In the opposite ordering, the raw signal kills nginx while systemd still believes its MainPID is alive, so systemctl status reports active over a dead port until the next explicit systemd action reconciles it.

Step 1: Stop wrapping renewals in systemctl stop/start

Delete any cron entry or script that does systemctl stop nginx around certbot renew. That single change removes the most common trigger. You do not need to stop nginx to renew if you use the right authenticator.

Step 2: Renew through nginx, with a reload not a restart

If nginx is already serving your site, let Certbot use it for both the challenge and the install. It will place the HTTP-01 challenge file, get the cert, and finish with a graceful reload:

sudo certbot renew --nginx

Or set it as the default for a certificate at issue time so unattended renewals inherit it:

sudo certbot --nginx -d example.com -d www.example.com

With this, nginx stays up throughout. There is no stop, so there is no empty PID file, so there is no race.

Step 3: If you are stuck on the standalone authenticator, make hooks idempotent and use systemctl

Standalone binds port 80 itself, so nginx has to release it for the length of the challenge. Do that through systemd on both sides, and write the hooks so running them twice is harmless:

sudo certbot renew \
  --pre-hook  "systemctl stop nginx" \
  --post-hook "systemctl start nginx"

Use systemctl in the hooks, never nginx -s. Keep both paths on the same control plane. The post-hook runs whether or not a renewal happened, so systemctl start nginx (which is a no-op if it is already running) is the safe choice, not nginx -s reload against a possibly-stopped master. See the Certbot renewal hooks documentation for how pre, deploy and post hooks fire.

Verification: prove nginx is actually serving, not just marked active

Never trust one signal. Check three things after a renewal:

# 1. systemd's opinion
systemctl is-active nginx

# 2. reality: something is listening on 443
ss -ltnp | grep ':443'

# 3. the cert is served and its dates are fresh
echo | openssl s_client -connect example.com:443 -servername example.com 2>/dev/null \
  | openssl x509 -noout -dates

If is-active says active but ss shows nothing on 443, that is the drift: systemd's state is stale. Reconcile it with an explicit action:

sudo systemctl restart nginx

The notBefore date from the last command should be recent. That, not Certbot's exit code, is proof the renewal reached live traffic.

What people get wrong

Treating certbot renew --dry-run as proof the real thing works. The dry run exercises the ACME conversation against the staging server, but it does not fully reproduce how your hooks interact with a live nginx and a real PID file under production timing. Test the actual hook behaviour once in a maintenance window. If your dry run and your live renew disagree, the hooks are the difference. I wrote separately about a related failure where renewals fail silently and nobody is alerted.

Adding Restart=always to nginx and calling it resilience. If systemd auto-restarts nginx while a Certbot signal is mid-flight, you have added a third actor to the race, not removed one. Fix the control-path conflict; do not paper over it with automatic restarts.

Using nginx -s stop in a post-hook. A post-hook runs even when nothing was renewed, and it may run when nginx is already stopped by a pre-hook, which is exactly the empty-PID-file condition. Hooks that signal the binary directly are the trap.

When it is still broken

If you still see the PID error or intermittent drift after all of the above:

1. Find every place nginx is controlled: grep -r "nginx -s" /etc/letsencrypt/renewal-hooks/ /etc/cron.d/ /etc/cron.* and remove or convert each to systemctl.
2. Check the renewal config itself: cat /etc/letsencrypt/renewal/example.com.conf and confirm the authenticator and any pre_hook/post_hook lines match what you intend. Old issuance leaves stale hooks here.
3. Confirm the PID path nginx uses matches the unit: nginx -V 2>&1 | tr ' ' '\n' | grep pid against the PIDFile= in the systemd unit. A mismatch guarantees drift.
4. Remove the challenge dependency entirely by moving to DNS-01. With DNS validation Certbot never touches port 80 or nginx to renew, so this whole class of race disappears. It is the right answer for production boxes where a renewal must never risk the web server. If you are also fighting the challenge itself failing, see why the HTTP-01 challenge gets connection refused.

The unit file underneath all of this matters too. If nginx is going to be supervised, supervise it properly and let one manager own it; my piece on a systemd unit that actually keeps your app running covers the surrounding fields.