Certificates for a Multi-Server Cluster: Issue Once, Distribute Everywhere

You have three web nodes behind a load balancer, all serving shop.example.com. Someone set them up the sensible-looking way: each node runs its own certbot, each renews its own certificate, each is self-contained. It works in staging. Then one morning renewal fails on two of the three nodes and your monitoring lights up with certificate expiry warnings, or worse, certbot starts returning a rate-limit error and refuses to issue anything at all.

The instinct is to treat this as a certbot bug. It is not. It is a design mistake. A certificate for shop.example.com is one logical object, and you have three machines fighting over who gets to create it. The fix is not a better cron schedule or a longer backoff. The fix is to stop issuing three times.

Issue the certificate once, in one place, and distribute the resulting files to every node. That is the whole idea. Everything below is how to do it without dropping a single connection.

Why per-node issuance breaks in a cluster

Each of your nodes talks to the same ACME endpoint and asks for a certificate covering the same set of names. From Let's Encrypt's point of view these are not three different certificates. They are the same certificate requested three times, and the account is charged against shared rate-limit buckets that are keyed on the domain, not the machine.

The most relevant one here is the Duplicate Certificate limit: requesting the exact same set of hostnames is capped at five per week. A three-node cluster that reissues on every renewal cycle, or that panics and retries after a transient failure, burns through that budget fast. Once you hit it, every node is locked out, including the one that was healthy.

There is a second, quieter problem even when you stay under the limit. Two nodes that issue near-simultaneously end up holding two different certificates with two different validity windows and two different private keys. Your load balancer now hands clients a different certificate depending on which backend answered. Certificate pinning breaks, OCSP behaviour diverges, and debugging "it works on refresh but not always" becomes miserable. A cluster should present one identity, not a lottery.

Why DNS-01 is the right challenge for a cluster

The HTTP-01 challenge asks Let's Encrypt to fetch a token from http://your-domain/.well-known/acme-challenge/.... In a load-balanced cluster that request lands on whichever backend the balancer picks, and the token only exists on the node that started the challenge. So validation succeeds or fails depending on routing luck. People paper over this by pinning the ACME path to one backend, but now you have a fragile special case in your load balancer config that breaks the day someone reorders the rules.

DNS-01 sidesteps all of it. You prove control of the domain by writing a TXT record, which has nothing to do with which node answers HTTP. One credential to your DNS provider's API, issued from one place, validates for the whole cluster. If you have ever fought this on a load balancer, the same reasoning is in ACME TLS handshake failure behind a load balancer and use DNS-01, not http-01, behind Cloudflare.

The fix, step by step

Step 1: Choose the issuer and stop certbot on the others

Pick one host to own issuance. It can be one of the web nodes, but a small dedicated automation host (the same box that runs your deploy tooling) is cleaner because it removes the certificate lifecycle from your web tier entirely. On every other node, disable the certbot timer so it can never issue again:

sudo systemctl disable --now certbot.timer
sudo systemctl list-timers | grep certbot   # should now be empty

Leave certbot installed there if you like, but it must never renew. Its job now is only to receive files.

Step 2: Issue once on the issuer, with DNS-01

On the issuer, use the DNS plugin for your provider. The exact plugin name depends on where your DNS lives (Cloudflare, Route 53, and many others have official or community plugins). With Cloudflare it looks like this:

sudo certbot certonly \
  --dns-cloudflare \
  --dns-cloudflare-credentials /etc/letsencrypt/cloudflare.ini \
  -d shop.example.com -d www.shop.example.com \
  --cert-name shop.example.com

The credentials file holds a scoped API token that may edit DNS records for that zone and nothing else. Lock it down: chmod 600 /etc/letsencrypt/cloudflare.ini. Setting --cert-name explicitly means renewals keep updating this one lineage instead of spawning shop.example.com-0001 directories, a trap covered in certbot keeps making new certificates instead of renewing.

Test the whole thing against staging first so a config mistake does not eat your real rate limit. That habit is worth building permanently, as argued in your cert automation should default to Let's Encrypt staging.

Step 3: Distribute the certificate with a deploy hook

certbot runs a --deploy-hook script only when a certificate is actually renewed, which is exactly when you want to push files. Put your distribution logic there instead of in a separate cron job that copies stale files on every tick:

#!/usr/bin/env bash
# /etc/letsencrypt/renewal-hooks/deploy/distribute.sh
set -euo pipefail

CERT_DIR="/etc/letsencrypt/live/shop.example.com"
NODES=("web-a.internal" "web-b.internal" "web-c.internal")

for node in "${NODES[@]}"; do
  rsync -a --chmod=600 \
    "${CERT_DIR}/fullchain.pem" "${CERT_DIR}/privkey.pem" \
    "deploy@${node}:/etc/ssl/shop/"
  ssh "deploy@${node}" 'sudo systemctl reload nginx'
done

sudo chmod +x /etc/letsencrypt/renewal-hooks/deploy/distribute.sh

If you run configuration management (Ansible, Salt, a GitOps sync), an even better pattern is to have the deploy hook drop the new files into your secrets store or trigger a playbook, and let the existing tooling fan them out. The principle is the same: one source of truth, pushed outward.

Step 4: Reload, never restart, the TLS terminator

This is the line people skip and regret. A restart tears down the listening socket and every in-flight connection with it. A reload tells the process to re-read its configuration and swap the certificate while keeping existing connections alive. For nginx that is nginx -s reload or systemctl reload nginx; for HAProxy it is a hitless reload; for Caddy it is caddy reload. Wire the reload into the same hook, as above, so a node never serves the old certificate after the new one has landed.

Verifying the whole cluster serves the same certificate

The point of all this is that every node presents an identical certificate. Prove it by asking each backend directly for its certificate serial number and comparing:

for host in web-a.internal web-b.internal web-c.internal; do
  echo -n "$host: "
  echo | openssl s_client -connect "$host:443" \
    -servername shop.example.com 2>/dev/null \
    | openssl x509 -noout -serial
done

Expected output is the same serial on every line:

web-a.internal: serial=03A1F9...C4
web-b.internal: serial=03A1F9...C4
web-c.internal: serial=03A1F9...C4

If one serial differs, that node did not receive the latest file or was not reloaded. Check the deploy hook log and the file modification time in /etc/ssl/shop/ on that node. To confirm renewal itself is healthy, run sudo certbot renew --dry-run on the issuer and watch it exercise the DNS challenge and the deploy hook end to end.

What people get wrong

"Just add jitter so the nodes do not renew at the same second." This treats a design problem as a timing problem. Staggering the requests does not reduce how many certificates you issue; you still burn the duplicate-certificate budget, just more slowly. And you still end up with three different keys across the cluster. Jitter hides the collision, it does not remove it.

"Raise the rate limit." You cannot on demand, and you should not want to. Hitting the limit is the symptom; the disease is issuing the same certificate many times. If you find yourself reading how to fix Let's Encrypt rate limits for a cluster, the durable answer is almost always to consolidate issuance, not to ask for more headroom.

"Put certbot in every container image so each replica is self-contained." In an autoscaling group this is the worst version of the problem: every new replica issues on boot, and a scale-up event can exhaust your weekly limit in minutes. Certificates belong to the platform, not to the replica. Mount them in, or terminate TLS at the load balancer.

"Copy the whole /etc/letsencrypt directory to every node." Tempting, but it drags along account keys and renewal config that only the issuer should have, and it invites a node to think it can renew. Ship only fullchain.pem and privkey.pem to the consumers. Renewal state stays on the issuer alone.

When it is still broken

• Terminate TLS at the load balancer instead. If your balancer (an ALB, a managed HAProxy, an nginx edge) does TLS itself, the backends can speak plain HTTP internally and the whole distribution problem disappears. This is often the simplest answer for a cluster you control end to end.
• The DNS challenge times out. Some providers propagate TXT records slowly. Add --dns-cloudflare-propagation-seconds 60 (or the equivalent for your plugin) so certbot waits before asking Let's Encrypt to validate.
• A node serves the old cert after reload. Confirm the reload actually happened (journalctl -u nginx --since "5 min ago") and that the file paths in your server block point at /etc/ssl/shop/, not the old /etc/letsencrypt/live path that no longer updates on that node.
• Permissions on the private key. If the web process cannot read privkey.pem it will fall back to no certificate or fail to start. The key should be 600 and owned by whatever user reads it; do not loosen it to 644 to make an error go away.

One certificate, one issuer, pushed to many nodes, reloaded in place. That is the shape of certificate management that scales past a single box without ever surprising you at renewal time.