ACME TLS Handshake Failure Behind a Load Balancer: When Only Some Backends Answer

Your certificate renewal failed. You rerun it, and it works. A week later it fails again, and again a rerun fixes it. Normal traffic to the site is fine the entire time; browsers are happy, the padlock is green, nobody is complaining. Only the ACME challenge, only sometimes, throws a TLS handshake failure.

So you go where everyone goes: the ACME client. You re-read the certbot flags, you check the webroot path, you second-guess the DNS. All of it is fine, because none of it is the problem.

The problem is that you have more than one backend behind a load balancer, and they are not identical. The challenge request lands on whichever backend the load balancer happens to pick, and only some of them are configured to answer it. Which one answers is nondeterministic, so the failure is nondeterministic. "It works when I retry" is not luck; it is the load balancer routing you to a good node that time.

The exact error: tls: handshake failure during domain verification

Let's Encrypt reports it back to your client, and it reads like this:

The server experienced a TLS error during domain verification ::
remote error: tls: handshake failure

Read it literally. Let's Encrypt's validation server opened a TLS connection to your domain and the other side aborted the handshake. That is not a certificate-expiry problem and not a DNS problem. Something answered on port 443, started a TLS handshake, and then refused to complete the specific handshake the ACME challenge needs. The word "sometimes" is the whole diagnosis: a subset of your backends refuse it.

Why one bad backend breaks the whole challenge

To fix this you need to know what the challenge actually asks each backend to do, because the two common challenge types fail differently.

http-01 is the simple one. Let's Encrypt connects to http://your-domain/.well-known/acme-challenge/<token> on port 80 and expects the exact key authorization string back. Your ACME client wrote that token file on the machine it ran on. If that machine is one of five identical app servers and the load balancer sends the validation request to any of the other four, they have no such file, and you get a 404 or a redirect that fails validation. The handshake-failure variant appears when those other nodes force an HTTPS redirect and then cannot complete TLS.

tls-alpn-01 is where the tls: handshake failure text comes from directly. This challenge happens entirely inside the TLS handshake on port 443. The responder must present a special self-signed certificate for your domain and negotiate the ALPN protocol named acme-tls/1. A backend that has never heard of acme-tls/1 cannot select it, so the handshake aborts. If only the node running your ACME client knows how to answer, every other node fails the handshake the moment the load balancer routes there.

The trap underneath both is where TLS is terminated. If the load balancer terminates TLS and forwards plain HTTP to the backends, the ACME client has to live on the load balancer, not the app servers. If TLS is passed through to the backends, then each backend terminates its own TLS and each one must be able to answer. Teams get this wrong because normal traffic does not care which node answers, so the asymmetry stays invisible until the one operation that depends on a specific node runs.

Diagnose it: test each backend, not the front door

Step 1: Confirm the failure is per-node, not per-client

Reissue against Let's Encrypt staging a few times in a row. If some attempts pass and some fail with the same config, the input that is changing is the backend, not your command. That alone rules out the ACME client.

Step 2: Find where TLS is terminated

Check your load balancer config. Is it doing TLS termination (an HTTPS listener with a certificate on the balancer) or TCP pass-through (a TCP listener on 443 forwarding raw bytes)? This decides where the responder must live. Do not skip this; every later step depends on the answer.

Step 3: Probe each backend directly

Bypass the load balancer and talk to each node's real IP. For http-01, place a known token and fetch it from every node:

# On the node running certbot, note the token file it wrote, then from outside:
curl -H "Host: your-domain.com" http://NODE_A_IP/.well-known/acme-challenge/TESTTOKEN
curl -H "Host: your-domain.com" http://NODE_B_IP/.well-known/acme-challenge/TESTTOKEN
# The node that 404s or redirects to HTTPS is your failing backend.

For tls-alpn-01, check whether each node can negotiate the challenge ALPN protocol at all:

openssl s_client -connect NODE_C_IP:443 -alpn acme-tls/1 -servername your-domain.com </dev/null
# A node that cannot do the acme-tls/1 handshake is the one failing validation.

Step 4: Pick a fix that matches your termination model

If the load balancer terminates TLS, run the ACME client on the load balancer (or use its managed-certificate feature) so there is exactly one responder. If TLS is passed through, you have two honest choices: make every backend able to answer (a shared challenge directory, or routing every /.well-known/acme-challenge/ request and the acme-tls/1 handshake to a single designated node), or stop using the HTTP/TLS path entirely.

Step 5: The durable fix is dns-01

The dns-01 challenge proves control by publishing a _acme-challenge TXT record. Let's Encrypt queries DNS, not your servers, so the number of backends and how the load balancer routes become irrelevant. It is also the only challenge that issues wildcards. If your DNS provider has an API, a plugin automates the record:

certbot certonly \
  --dns-cloudflare \
  --dns-cloudflare-credentials /etc/letsencrypt/cloudflare.ini \
  -d your-domain.com -d '*.your-domain.com'

Confirm the plugin and flag names against the certbot DNS plugin docs for your provider before running it in production.

Verify the fix

Run one staging issuance, then a real one, and only trust it after it passes several times consecutively:

certbot renew --dry-run
# Repeat 3-4 times. Every run must succeed. One failure in five means
# a backend is still slipping through unfixed.

If you moved to dns-01, also watch the next scheduled renewal actually fire, because a challenge type change can quietly break automation. Alerting on that is worth setting up; see catching certbot's silent renewal failures.

What people get wrong

Turning on sticky sessions to "fix" it. Session affinity makes the failure rarer, which feels like progress and is actually the worst outcome. The broken backends are still broken; you have only reduced the odds of hitting one during validation. The renewal will fail eventually, probably at the least convenient moment, and now with a config change that hid the real cause.

Adding a retry loop around issuance. Retrying until you land on a good node treats Let's Encrypt's rate limits as free. They are not, and you can lock yourself out; see the "too many certificates already issued" rate limit. Retries paper over the asymmetry instead of removing it.

Chasing certbot flags. The client ran fine. The bytes it sent were correct. The failure is downstream of the client, in how your fleet answers, so no flag will fix it.

When it is still broken

• Look for a default or catch-all server block. A node with a self-signed fallback vhost will intercept SNI for your domain and present the wrong certificate before the real config is consulted. That is a classic source of an intermittent handshake failure.
• Check IPv6. If your domain has an AAAA record pointing at a node (or path) without the responder, Let's Encrypt may prefer IPv6 and fail there while IPv4 succeeds. Either fix the IPv6 path or make it answer too.
• Check the load balancer's own TLS. If the balancer terminates TLS with SNI-based routing and your domain is not mapped, the balancer itself may present a default certificate and abort the acme-tls/1 handshake before any backend is involved.
• Reduce the variables. Move to dns-01 for the initial issuance to get a working certificate, then diagnose the HTTP/TLS path without a broken production site over your head.

The habit that prevents all of this: never assume the nodes behind a load balancer are identical for operations that depend on a specific node answering. Normal traffic hides the difference. Certificate validation, health checks, and anything with server-side state will find it. If you run TLS behind a proxy at all, the nginx reverse proxy and TLS config I actually run is a good baseline to compare each node against.