Certbot renewal fails silently: alerting before the certificate expires

Nobody finds out about an expired certificate from a monitoring system. They find out because a client calls to say the site is "showing a virus warning", or because a payment callback started failing at 4am and the provider's logs say the TLS handshake was rejected. By then the certificate has been dead for hours and the renewal that was supposed to happen has been failing quietly for eleven weeks.

That is the part worth sitting with. Certbot was not broken for a day. It ran twice a day, every day, for nearly three months, failed every single time, and nothing anywhere told anyone. The default installation gives you automation without observation, which feels like reliability right up until the moment it is the opposite.

Since 4 June 2025 Let's Encrypt no longer sends expiration notification emails at all, so the one accidental safety net most teams were relying on without realising it is gone. If you have not replaced it with something deliberate, you currently have no expiry alerting.

Why the failure is silent

Two mechanisms, both working exactly as designed.

On Debian and Ubuntu the certbot package installs both a cron entry and a systemd timer, and the cron entry deliberately stands down when systemd is running. The scheduled command is a quiet renewal:

certbot -q renew

The -q means "print nothing unless something goes wrong". Cron's only notification mechanism is emailing the output of a job to the owner, which requires a working local mail transport agent and a mailbox somebody reads. On a modern cloud VPS there is usually no MTA installed at all, so that output goes nowhere. Under systemd it is worse in a subtle way: the output goes to the journal, which is a file nobody has opened in a year.

The second mechanism is certbot renew's exit code, which is correct but easy to misread. The documentation is explicit: the exit status is 1 only if a renewal attempt failed, and 0 if no certificate needed updating. That is exactly right for automation, and it means that for 60 of every 90 days a completely broken setup exits 0 and looks healthy.

Add the most common breakage on top of that: a redirect or rewrite rule that swallows the ACME challenge path. Your site works. HTTPS works. Only http://example.com/.well-known/acme-challenge/ is broken, and nothing on earth requests that path except Let's Encrypt, twice a day, in silence.

Step 1: Make the renewal job page you when it fails

Write one notification script. Mine posts to a chat webhook; a mail command or an ntfy topic works just as well. What matters is that it reaches a human on a phone.

sudo tee /usr/local/bin/notify-unit-failure >/dev/null <<'EOF'
#!/bin/bash
unit="$1"
host="$(hostname -f)"
body="$(journalctl -u "$unit" -n 25 --no-pager)"
curl -fsS -X POST "$ALERT_WEBHOOK_URL" \
  --data-urlencode "text=FAILED: ${unit} on ${host}
${body}"
EOF
sudo chmod 750 /usr/local/bin/notify-unit-failure

Wire it to a template unit so any unit on the box can use it:

[Unit]
Description=Notify that %i failed

[Service]
Type=oneshot
Environment=ALERT_WEBHOOK_URL=https://hooks.example.com/xxxx
ExecStart=/usr/local/bin/notify-unit-failure %i

Save that as /etc/systemd/system/notify-failure@.service, then attach it to certbot with a drop-in rather than editing the packaged unit:

sudo systemctl edit certbot.service

[Unit]
OnFailure=notify-failure@%n.service

sudo systemctl daemon-reload

The %n expands to the full unit name, so a failing certbot.service starts notify-failure@certbot.service.service, and inside that instance %i is certbot.service. It looks odd and it is the standard idiom.

Step 2: Probe the certificate that is actually being served

Step 1 tells you the renewal command failed. It cannot tell you the renewal succeeded but nginx was never reloaded, so the old certificate is still in memory and still being served. That failure mode is common and completely invisible on the host, because the file on disk is perfectly fresh.

So check the handshake, from somewhere else:

#!/bin/bash
# expiry-check.sh example.com 14
host="$1"; days="${2:-14}"
secs=$(( days * 86400 ))

cert=$(echo | openssl s_client -servername "$host" -connect "$host":443 2>/dev/null)
echo "$cert" | openssl x509 -noout -checkend "$secs" >/dev/null || {
  enddate=$(echo "$cert" | openssl x509 -noout -enddate)
  echo "ALERT: certificate for $host expires within $days days ($enddate)"
  exit 1
}
echo "OK: $host has more than $days days left"

openssl x509 -checkend N exits non-zero when the certificate will expire within N seconds, which makes it trivially usable from any monitoring system that understands exit codes. Run it from a different machine, or from your CI on a schedule. The point is to be independent of the box that is failing.

Step 3: Alert on silence, not just on failure

A timer that has been disabled, a machine that was rebuilt without certbot, a snapshot restored from before the setup: none of these generate a failed unit, because nothing runs at all. The fix is a dead-man's switch, a URL you ping on success where the monitoring service alerts you if the ping does not arrive on schedule.

Certbot has the right hook for this. A deploy hook runs only after a successful renewal, and hook scripts dropped into the renewal-hooks directories are picked up automatically for every certificate:

sudo tee /etc/letsencrypt/renewal-hooks/deploy/10-reload-and-ping >/dev/null <<'EOF'
#!/bin/bash
set -e
systemctl reload nginx
curl -fsS -m 10 --retry 3 https://hc.example.com/ping/<uuid> >/dev/null
EOF
sudo chmod 750 /etc/letsencrypt/renewal-hooks/deploy/10-reload-and-ping

Certbot's hook directories are pre, deploy and post under /etc/letsencrypt/renewal-hooks/. Use deploy for anything that should only happen when a certificate actually changed, and remember that pre and post hooks run around every renewal attempt, including the ones that renew nothing.

One caution on the reload: if the hook fails, certbot prints the error and carries on. A deploy hook that quietly fails to reload nginx leaves you serving the old certificate, which is exactly the scenario step 2 exists to catch.

Step 4: Test the renewal on a schedule, not just at setup

sudo certbot renew --dry-run

This runs the whole renewal against the Let's Encrypt staging server without writing a new certificate, so it exercises the part that actually breaks: DNS, the challenge path, your web server config, your plugin. Run it weekly from a small timer or cron entry and route its failure into the same alerting as step 1. A dry run in January proves nothing about the redirect somebody added in March.

While you are there, make sure the challenge path cannot be swallowed by a redirect. In nginx, the ^~ prefix modifier stops regex locations from taking precedence:

location ^~ /.well-known/acme-challenge/ {
    root /var/www/html;
    default_type "text/plain";
    allow all;
}

Place that above any catch-all redirect in the port 80 server block. If the challenge is failing before you even get to alerting, the connection-level causes are covered separately in certbot connection refused on http-01.

Verification

Confirm the timer exists and when it next fires:

systemctl list-timers certbot.timer

NEXT                        LEFT     LAST                        PASSED   UNIT           ACTIVATES
Fri 2026-07-24 21:14:03 UTC  9h left  Fri 2026-07-24 06:41:22 UTC  5h ago   certbot.timer  certbot.service

Confirm what certbot thinks it manages, and how long is left:

sudo certbot certificates

  Certificate Name: example.com
    Serial Number: 3f9a...
    Domains: example.com www.example.com
    Expiry Date: 2026-09-21 04:12:07+00:00 (VALID: 58 days)
    Certificate Path: /etc/letsencrypt/live/example.com/fullchain.pem

Then the step everybody skips. Break it deliberately and confirm you get told:

1. Temporarily rename the certbot binary, or add a --dns-fake style invalid argument to a copy of the unit, and run sudo systemctl start certbot.service. You should receive a notification within seconds. Undo it.
2. Run your expiry probe against a host with a certificate you know is close to expiry, or set the threshold to 200 days so it trips. You should get the alert.
3. Pause the dead-man's switch check-in for one cycle and confirm the "missing ping" alert arrives.

An alert you have never seen fire is not an alert. It is a hope.

What people get wrong

Relying on Let's Encrypt's expiry emails. That service ended on 4 June 2025. Anyone still treating those emails as their backstop currently has no backstop at all. This is worth raising with clients explicitly, because it silently changed the risk profile of every certificate they own.

Monitoring the file on disk instead of the served certificate. Checking /etc/letsencrypt/live/example.com/cert.pem tells you certbot did its job. It tells you nothing about whether nginx, HAProxy or your Node process picked it up. Probe the handshake.

Putting --force-renewal in the timer. It renews on every run regardless of expiry, which burns through Let's Encrypt's duplicate certificate rate limit and can leave you unable to issue at all for days, sometimes at the worst possible moment. Certbot only renews inside the 30-day window by design; let it.

chmod -R 777 /etc/letsencrypt. This appears in far too many forum answers as a fix for a permission error. Those directories contain your private keys. World-readable private keys turn a renewal inconvenience into a compromise you will not detect. If you have a permission problem, fix the specific ownership on the specific path, as root.

Assuming cron emails somebody. It requires an MTA, a delivered mailbox, and a person who reads it. On a typical VPS, none of the three exist.

When it is still failing

1. Read the log properly. /var/log/letsencrypt/letsencrypt.log holds the full trace, and it rotates, so check the numbered files for the first failure rather than the most recent one. The date of the first failure usually names the change that caused it.
2. Check the renewal configuration, not the command you remember running. /etc/letsencrypt/renewal/example.com.conf records the authenticator, the webroot path and any hooks that were saved at issuance. A webroot that moved during a site rebuild is a classic cause of a renewal that fails while the site is perfectly healthy.
3. Test the challenge path by hand. Put a file in the webroot under /.well-known/acme-challenge/ and fetch it over plain HTTP from another network. If a redirect, a WAF rule or Cloudflare returns anything other than the file contents, that is your failure.
4. Check the account, not just the server. If the machine was restored from an image, or /etc/letsencrypt was partially copied between hosts, the account key may not match the certificate lineage. Re-running certonly for the domain will tell you quickly.

None of this is exotic. It is one drop-in, one script and one scheduled check, and it takes about twenty minutes. Compare that with explaining to a client why their checkout page showed a security warning for six hours on a Saturday. If you are already reviewing your edge configuration, the nginx reverse proxy and TLS config is the right place to put the challenge location block permanently.