Certbot http-01 Fails Behind Your Force-HTTPS Redirect: It Is Not HSTS

The renewal timer fires at 03:12. Certbot fails. You read the error, search it, and the first three results all say the same thing: your HSTS header and your force-HTTPS redirect are blocking the ACME challenge, so turn them off, renew, then turn them back on. You do it. The cert renews. Three weeks later you notice HSTS is still disabled on a site that takes card payments, because nobody put it back.

Here is the thing that advice gets wrong at the root. HSTS cannot break ACME validation. The Let's Encrypt validation server does not implement HSTS. It is a browser policy, delivered in a response header, honoured by browsers. A validation client that fetches one URL and reads the body has no HSTS store and no reason to consult one.

And the redirect does not break it either. Let's Encrypt documents that HTTP-01 follows redirects, up to ten deep, to http: or https: on ports 80 or 443, and it explicitly does not validate the certificate on an HTTPS redirect target. Your expired cert is not the problem either. So if your renewal is failing behind a force-HTTPS rule, something else is wrong, and it is almost always one of four things.

The exact error

Modern certbot reports the CA's own words back to you, and the Detail: line is the only part worth reading:

Certbot failed to authenticate some domains (authenticator: webroot).
The Certificate Authority reported these problems:
  Domain: example.com
  Type:   unauthorized
  Detail: 203.0.113.10: Invalid response from
  http://example.com/.well-known/acme-challenge/mV3n-9wQ...: 404

The other common variant, which means something completely different:

  Type:   connection
  Detail: 203.0.113.10: Fetching
  http://example.com/.well-known/acme-challenge/mV3n-9wQ...:
  Timeout during connect (likely firewall problem)

People searching this often describe it in their own words as a permanent redirect to https preventing them from authorizing the request. That description is what leads them to the wrong fix, because it names the redirect as the cause when the redirect is only the vehicle.

Where HSTS actually applies, and where it does not

That left-hand box is the whole misdiagnosis in one place. You try to check the challenge URL in a browser, the browser upgrades it to HTTPS from its own HSTS cache, the expired certificate throws an interstitial, and you conclude that HSTS is blocking the challenge. Use curl instead. It does not do HSTS unless you explicitly ask it to with an HSTS cache file.

Why it actually fails

Cause 1: the redirect discards the path

This is the one, and it comes straight out of the tutorials people copy:

# Wrong. Sends every request to the homepage.
server {
    listen 80;
    server_name example.com;
    return 301 https://$host/;
}

The CA asks for /.well-known/acme-challenge/mV3n-9wQ, gets a 301 pointing at https://example.com/, follows it obediently, and receives your homepage with a 200 or your app's 404. Either way the response body is not the token, so validation fails with unauthorized. The redirect worked perfectly. It just sent the CA somewhere useless.

Cause 2: the HTTPS vhost does not serve the challenge

Your port 80 block redirects correctly, path intact, and the CA lands on the port 443 block. But that block is a reverse proxy to a Node or Frappe application, and the application has never heard of /.well-known/acme-challenge/, so it returns its own 404. The challenge file exists on disk. Nothing is reading it.

Cause 3: the TLS handshake on the redirect target fails

Let's Encrypt does not check the certificate on the redirect target, but it still has to complete a TLS handshake to read the response. A missing default server block for that SNI name, a protocol floor that rejects the client, or a firewall that drops 443 from outside will all fail here, and the error looks like a generic connection problem rather than a TLS one.

Cause 4: port 80 is closed

Somebody hardened the firewall and dropped port 80 because "everything is HTTPS anyway". HTTP-01 can only be done on port 80. There is no flag, no workaround, no configuration on the CA side. Either port 80 accepts the connection or you use a different challenge type. This produces the Timeout during connect variant, not the 404 one.

The fix, in numbered steps

Step 1: Reproduce exactly what the CA does

Put a file where the challenge would go, then fetch it from a machine that is not the server. Fetching from the server itself proves nothing, because you bypass the firewall and often the DNS too:

sudo mkdir -p /var/www/certbot/.well-known/acme-challenge
echo "probe-ok" | sudo tee /var/www/certbot/.well-known/acme-challenge/probe

# from your laptop, not the server:
curl -ksSL -D - -o /dev/null "http://example.com/.well-known/acme-challenge/probe"

-D - prints the headers of every hop, -k skips certificate verification exactly as the CA does, -L follows the chain. A healthy result looks like this:

HTTP/1.1 301 Moved Permanently
Location: https://example.com/.well-known/acme-challenge/probe

HTTP/2 200
content-type: text/plain

A broken one gives you the diagnosis in a single line:

HTTP/1.1 301 Moved Permanently
Location: https://example.com/

HTTP/2 404

Step 2: Make the redirect preserve the request URI

server {
    listen 80;
    listen [::]:80;
    server_name example.com www.example.com;

    location ^~ /.well-known/acme-challenge/ {
        root /var/www/certbot;
        default_type "text/plain";
    }

    location / {
        return 301 https://$host$request_uri;
    }
}

Two things are doing work here. $request_uri carries the full original path and query string into the redirect. And location ^~ is a prefix match that tells nginx to stop searching once it matches, so a regex location further down cannot steal the request. A plain location / would still lose to the longer prefix, but ^~ makes the intent explicit and survives the next person editing the file.

Reload and check the config parsed before you trust it:

sudo nginx -t && sudo systemctl reload nginx

Step 3: Serve the challenge from the HTTPS block too

Because the CA follows the redirect, the port 443 block is where the file actually gets read. Add the same location there, above your proxy pass:

server {
    listen 443 ssl;
    server_name example.com www.example.com;

    add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;

    location ^~ /.well-known/acme-challenge/ {
        root /var/www/certbot;
        default_type "text/plain";
    }

    location / {
        proxy_pass http://127.0.0.1:8000;
    }
}

Note that the HSTS header stays exactly where it was. It is doing its job for real visitors and it is invisible to the CA.

Step 4: Dry run before you burn rate limit

sudo certbot certonly --webroot -w /var/www/certbot \
  -d example.com -d www.example.com --dry-run

A dry run hits the staging environment, so a failure costs you nothing. Production issuance is rate limited per registered domain, and a loop of failing renewals is a good way to lock yourself out for a week at exactly the wrong moment. Never debug against production issuance.

Step 5: Consider DNS-01 and stop having this problem

If your host blocks port 80, if the domain sits behind something you do not control, or if you want wildcards, HTTP-01 is the wrong tool. DNS-01 proves control by writing a TXT record, so no inbound port is involved at all:

sudo certbot certonly --dns-cloudflare \
  --dns-cloudflare-credentials /etc/letsencrypt/cloudflare.ini \
  -d example.com -d '*.example.com'

Create the credentials file with a scoped API token, then sudo chmod 600 it. Certbot warns loudly if the permissions are wrong, and it is right to. That file can rewrite your DNS. There are provider plugins for most registrars, and a token restricted to a single zone's DNS edit permission is the correct scope.

Verification

Three checks, in order. First, the dry run above must complete with a success line. Second, confirm the real certificate actually rotated rather than certbot deciding it was still fresh:

sudo certbot certificates

Read the Expiry Date and the days remaining, not just the presence of the cert. Then confirm what the world sees, which is not always what is on disk:

echo | openssl s_client -connect example.com:443 -servername example.com 2>/dev/null \
  | openssl x509 -noout -dates -subject

Finally, prove the renewal path itself works unattended rather than only when you run it by hand:

sudo systemctl list-timers certbot.timer
sudo certbot renew --dry-run

A certificate that renews only when a human is watching is not renewed, it is postponed. This is the same discipline as testing restores rather than backups.

What people get wrong

Common advice	Why it is wrong
Disable HSTS to let the challenge through	The CA never reads the header. You have removed a real protection for real users to influence a client that ignores it. Worse, browsers cache the old policy for the full max-age, so removing the header does not even take effect immediately.
Comment out the HTTPS redirect during renewal	Works by accident, and now your renewal depends on a manual edit that will be forgotten. Fix the redirect once instead of disabling it twelve times a year.
Close port 80 entirely for security	HTTP-01 only runs on port 80. Closing it does not remove an attack surface, since the same nginx serves both ports, and it does break every plain-HTTP visitor who would otherwise have been redirected to you.
chmod -R 777 /var/www so the CA can read the file	The CA reads over HTTP, not the filesystem. The permission that matters is whether the nginx worker user can read the token file, which is 644 on the file and 755 on the directories. 777 is world-writable and hands anyone with a shell a way to serve content from your webroot.
Switch to --standalone because webroot is fiddly	Standalone binds port 80 itself, so it needs nginx stopped. People wire that into a pre-hook, the post-hook fails one night, and the site is down until morning. Use webroot while nginx keeps running.
Retry certbot renew in a loop until it works	Failed validations count against your rate limits. A tight retry loop is how a broken renewal becomes a week-long outage.

When it is still broken

1. Check DNS from outside. The CA resolves your name from the public internet, not from your /etc/hosts. If the A record points at an old server, the challenge lands on a machine that has never seen the token. dig +short example.com from somewhere unrelated to your network.
2. Check every name in the request. A cert for example.com and www.example.com requires both to validate. If www redirects to the apex and the apex block lacks the acme location, one name passes and the whole order fails.
3. Dump the effective nginx config with sudo nginx -T and search it for acme-challenge and return 301. Included files and a stale sites-enabled symlink are a frequent cause of editing a file that is not being used.
4. Watch the access log during the attempt. Run sudo tail -f /var/log/nginx/access.log | grep acme in one terminal and the dry run in another. If no request appears at all, the traffic is not reaching nginx and the problem is DNS, the firewall or a proxy in front, not your vhost. If a request appears with a 301 followed by nothing, you are back to the redirect and proxy configuration.

The wider point: security controls that people disable to make something else work will stay disabled. If a control is genuinely in your way, replace it with one that is not, do not switch it off with a note to yourself. That note is the least reliable component in the system. Whatever you land on, write the working config down where the next person will find it, ideally next to the rest of your certificate setup.