Free HTTPS on Nginx With Let's Encrypt, and the Renewal Everyone Forgets

A client called me on a Sunday because their site had gone from working to a full page browser warning about an untrusted connection. Nobody had deployed anything. Nobody had touched the server. The certificate had simply reached day ninety and expired, three months after a consultant set it up, demonstrated it, and moved on.

Free certificates from Let's Encrypt changed the economics of HTTPS in Kenya more than anywhere I have worked, because the old answer to "should we buy a certificate for the staging site" was no, and now the question does not arise. But free certificates come with a ninety day life, and that is the part people skip. Issuing the certificate is the demo. The renewal is the product. If you only automate one of the two, automate the second.

There is also a deadline forming. Google has said Chrome will start labelling plain HTTP pages that ask for a password or a card number as "Not secure" from January. Any site with a login form is on that list.

sudo certbot certonly --webroot -w /var/www/html \
  -d example.com -d www.example.com \
  --email you@example.com --agree-tos

# then, in root's crontab
15 3,15 * * * /usr/bin/certbot renew --quiet --renew-hook "systemctl reload nginx"

Step 1: Install certbot

Ubuntu 16.04 carries the client in universe under its old name, but it is well behind. Use the PPA:

sudo apt-get update
sudo apt-get install -y software-properties-common
sudo add-apt-repository -y ppa:certbot/certbot
sudo apt-get update
sudo apt-get install -y certbot

There is an Nginx plugin that will edit your configuration for you. It is still marked experimental, and I do not use it on servers I care about. It rewrites server blocks in ways that are hard to review, and when it guesses wrong you are debugging generated configuration at the exact moment your site is down. Issue the certificate, then write four lines of Nginx by hand.

Step 2: Make the challenge path reachable

Let's Encrypt proves you control the domain by asking your web server for a file. Before requesting anything, make sure a plain HTTP request for that path is served, and is not swallowed by a redirect:

server {
    listen 80;
    server_name example.com www.example.com;
    root /var/www/html;

    location /.well-known/acme-challenge/ {
        allow all;
    }
}

sudo nginx -t && sudo systemctl reload nginx
echo hello | sudo tee /var/www/html/.well-known/acme-challenge/test
curl -i http://example.com/.well-known/acme-challenge/test

You want HTTP/1.1 200 OK and the word hello. If you get a 301, a 404, or somebody else's site, stop here. The certificate request will fail for exactly the same reason, only with a longer error message.

The error you will meet, and what it means

Failed authorization procedure. example.com (http-01): urn:acme:error:unauthorized ::
The client lacks sufficient authorization :: Invalid response from
http://example.com/.well-known/acme-challenge/lB7x9... [41.90.x.x]: 404

Read the two useful parts. The IP address in brackets is where the ACME server actually landed, so if that is not your server, your DNS is the problem. The status code at the end is what it got: a 404 means the file was not found at the webroot you passed to -w, and a 301 means something redirected the challenge, usually a blanket HTTP to HTTPS rule that you added before the certificate existed.

The root cause in almost every case is a mismatch between the directory certbot writes into and the directory Nginx serves for that hostname. Certbot does not read your Nginx configuration in webroot mode. It writes where you tell it to, and if your server block for www.example.com has a different root from the one for example.com, one name validates and the other does not.

Step 3: Request the certificate

sudo certbot certonly --webroot -w /var/www/html \
  -d example.com -d www.example.com \
  --email you@example.com --agree-tos --no-eff-email

On success:

IMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at
   /etc/letsencrypt/live/example.com/fullchain.pem
   Your key file has been saved at
   /etc/letsencrypt/live/example.com/privkey.pem
   Your cert will expire on 2017-02-06.

Use the paths under live/, never the numbered files in archive/. The live directory holds symlinks that certbot repoints on every renewal, so pointing Nginx at archive/ gives you a configuration that works today and serves an expired certificate in three months.

Step 4: Configure Nginx

server {
    listen 80;
    server_name example.com www.example.com;
    location /.well-known/acme-challenge/ { root /var/www/html; }
    location / { return 301 https://$host$request_uri; }
}

server {
    listen 443 ssl http2;
    server_name example.com www.example.com;
    root /var/www/html;

    ssl_certificate     /etc/letsencrypt/live/example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem;

    ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
    ssl_prefer_server_ciphers on;
    ssl_session_cache shared:SSL:10m;
    ssl_session_timeout 1d;
    ssl_dhparam /etc/nginx/dhparam.pem;

    ssl_stapling on;
    ssl_stapling_verify on;
    ssl_trusted_certificate /etc/letsencrypt/live/example.com/chain.pem;
}

Note that the redirect in the first block sits below the challenge location, so renewals keep working after you force HTTPS. That ordering is the single most common cause of "it issued fine but it will not renew".

Generate the DH parameters once. It takes a few minutes on a small VPS and there is no output until it finishes:

sudo openssl dhparam -out /etc/nginx/dhparam.pem 2048
sudo nginx -t && sudo systemctl reload nginx

Step 5: Prove the renewal works before you need it

This is the step that separates a site that stays up from the Sunday phone call:

sudo certbot renew --dry-run

The dry run talks to the staging environment, performs a real authorisation, and issues nothing. Expected ending:

Congratulations, all renewals succeeded. The following certs have been renewed:
  /etc/letsencrypt/live/example.com/fullchain.pem (success)

Then add the cron entry from the short answer above, and confirm what the browser is actually being served:

echo | openssl s_client -connect example.com:443 -servername example.com 2>/dev/null \
  | openssl x509 -noout -dates -issuer

What people get wrong

Renewing by hand with a calendar reminder. Ninety days is exactly long enough for the person who set the reminder to have changed jobs. Automate it or accept the outage.

Renewing without reloading Nginx. Certbot replaces the files on disk; Nginx keeps the old certificate in memory until it is reloaded. The renewal log says success, the browser says expired, and you lose an hour to disbelief. That is what the --renew-hook is for.

Testing against production until you hit a rate limit. Let's Encrypt currently allows twenty certificates per registered domain per week, and only five identical ones. If you loop on a failing command you will exhaust that quota and then be locked out for days on the one week you actually needed the certificate. Use --dry-run and --staging while you are experimenting.

Copying an ssl_ciphers line from a five year old blog post. Cipher advice ages badly and half of what is on the internet still enables RC4. Take the current recommended configuration from Mozilla's generator, and read it before you paste it.

When it is still broken

• The IP in the error is not yours. The domain still points at the old host. Check with dig +short example.com from a machine that is not the server.
• Timeout rather than 404. Port 80 is closed. Even if your site is HTTPS only, the http-01 challenge needs port 80 open, so leave it open and redirect.
• One name works and the other does not. Issue with only the failing name to isolate it. Usually www and the bare domain resolve to different servers.
• You need a certificate for a name that is not publicly reachable. Today the only alternative is the DNS challenge, which proves control by putting a TXT record in the zone. There are no wildcard certificates from Let's Encrypt at the moment, so a certificate must name every hostname it covers.

Full detail of every failure is in /var/log/letsencrypt/letsencrypt.log, which is verbose, ugly, and always contains the answer.