Let’s Encrypt: Reload Nginx after Renewing Certificates

Today I had an incident that caused my webserver to serve expired certificates. My blog relies on Let’s Encrypt for SSL/TLS certificates, which have to be renewed every 3 months. Usually, the cronjob that runs certbot --renew takes care of it automatically. However, there is one step missing: the server must reload the renewed certificates. Most of the time, the server gets reloaded often enough, so everything is okay, but today, it’s been quite a while since the last time the nginx server was restarted, so expired certificates were served and the blog became unavailable.

To work around it, we can make sure nginx reloads its configuration after each successful certificate renewal. The automatic renewal is defined in /etc/cron.d/certbot. The default contents under Debian Jessie are as follows:

# /etc/cron.d/certbot: crontab entries for the certbot package
#
# Upstream recommends attempting renewal twice a day
#
# Eventually, this will be an opportunity to validate certificates
# haven't been revoked, etc.  Renewal will only occur if expiration
# is within 30 days.
SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin

0 */12 * * * root test -x /usr/bin/certbot && perl -e 'sleep int(rand(3600))' && certbot -q renew

The last line makes sure certificate renewal runs twice a day. Append --renew-hook "/etc/init.d/nginx reload" to it, so it looks like this:

0 */12 * * * root test -x /usr/bin/certbot && perl -e 'sleep int(rand(3600))' && certbot -q renew --renew-hook "/etc/init.d/nginx reload"

The --renew-hook runs the next argument after each successful certificate renewal. In our case, we use it to reload the nginx configuration, which also reloads the newly renewed certificates.

Update 2019-02-27:

renew-hook has been deprecated in recent versions of certbot. Plus, Debian moved from using cronjobs for automatic renewals to a systemd timer if they are available. On the other hand, now certbot supports having hooks in configuration files. So, instead of what is described above, I would suggest creating a file /etc/letsencrypt/renewal-hooks/deploy/01-reload-nginx with the following content:

#! /bin/sh
set -e

/etc/init.d/nginx configtest
/etc/init.d/nginx reload

don’t forget to make the file executable.

nginx and SNI

Server name indication (SNI) allows you to serve multiple sites with different TLS/SSL certificates using a single IP address. Nginx has supported SNI for quite some time, and actually setting it up is easy: simply add server entries for the corresponding sites. There is one caveat: the server_name entry must come before the server_certificate for SNI to be activated:

server {
    listen          443 ssl;
    server_name     www.example.com;
    ssl_certificate www.example.com.crt;
    ...
}

server {
    listen          443 ssl;
    server_name     www.example.org;
    ssl_certificate www.example.org.crt;
    ...
}

is good, but

server {
    listen          443 ssl;
    ssl_certificate www.example.com.crt;
    server_name     www.example.com;
    ...
}

server {
    listen          443 ssl;
    ssl_certificate www.example.org.crt;
    server_name     www.example.org;
    ...
}

will serve the wrong certificate for www.example.org.

Skip Updates When Using the Let’s Encrypt `letsencrypt-auto` Client

To use Let’s Encrypt CA to issue free certificates, you need to use their client. The recommended method to install it is to use letsencrypt-auto, a script that automatically fetches and installs all the required dependencies. There is no doubt that letsencrypt-auto is the fastest and simplest way to get a Let’s Encrypt client up and running. I’ve used it myself when I wrote a guide to get Let’s Encrypt up and running easily.

Automatically updating required dependencies has its downside. As letsencrypt-auto does it every time you run it, it quickly gets annoying. Running a simple ./letsencrypt-auto --help takes a whopping 15 seconds just figuring out that there are no updates available. Supposing that you know that no updates are available, and you wish to save some time, you can run the letsencrypt executable directly, skipping the update process of letsencrypt-auto:

~/.local/share/letsencrypt/bin/letsencrypt

Most actions require you to be root, so you might need to run it with sudo.

You can expect this issue to be resolved in the future. There is already an open issue for it and active work that will resolve it.

Getting Started with Let’s Encrypt – Tutorial

A few days ago, I got my invitation to Let’s Encrypt Beta Program. For those of you who are not familiar with Let’s Encrypt:

Let’s Encrypt is a new free certificate authority, built on a foundation of cooperation and openness, that lets everyone be up and running with basic server certificates for their domains through a simple one-click process.

This short tutorial is intended to get you up and running with your own Let’s Encrypt-signed certificates.

The first thing is to get the Let’s Encrypt client:

git clone https://github.com/letsencrypt/letsencrypt
cd letsencrypt

The main command we will be working with is ./letsencrypt-auto. The first time you run it, it will also ask for sudo, install various dependencies using your package manager, and set up a virtualenv environment.

The next step is to issue the certificate and prove to Let’s Encrypt that you have some control over the domain. The client supports two methods to perform the validation. The first one is the standalone server. It works by setting up a web server on port 443 and responding to a challenge from the Let’s Encrypt servers. However, if you already have your own web server running on port 443 (the default for TLS/SSL), you would have to temporarily shut it down. To use the standalone method, run:

./letsencrypt-auto --agree-dev-preview --server https://acme-v01.api.letsencrypt.org/directory certonly

The second method is called Webroot authentication. It works by placing a folder (.well-known/acme-challenge) in the document root of your server with files corresponding to responses for challenges.

./letsencrypt-auto --agree-dev-preview --server https://acme-v01.api.letsencrypt.org/directory -a webroot --webroot-path /var/www/html/ certonly

Whichever method you choose, it will ask for a list of domains you want to validate and your email address. You can enter multiple domains. The first one will be the Common Name (CN), and the rest will appear in the Subject Alt Name field.

This slideshow requires JavaScript.

The newly generated certificates will be placed in

/etc/letsencrypt/live/yourdomain.com/

The important files in this directory are fullchain.pem, which contains the full certificate chain to be served to the browser, and privkey.pem, which is the private key.

An example Nginx configuration will now look like:

        listen 443 ssl;
        ssl_certificate /etc/letsencrypt/live/guyrutenberg.com/fullchain.pem;
        ssl_certificate_key /etc/letsencrypt/live/guyrutenberg.com/privkey.pem;

Just don’t forget to reload the web server so configuration changes take effect. No more government snooping on my blog 😉 .

certificate

Securing Access using TLS/SSL Client Certificates

This tutorial will guide you in setting up authentication using TLS/SSL Client Certificates. It is a simple one, as it does not delve into details about integration with server-side apps. Instead, it simply gives you instructions on how to set up Client Certificates as a means to prevent unwanted parties from accessing your website.

For example, one scenario where it may come in useful is limiting access to sensitive things on the server, like phpMyAdmin. phpMyAdmin, for example, handles sensitive data (such as database authentication) and does so in plain HTTP, which may pose several security risks. Even if the data were encrypted, someone with access to the application might find vulnerabilities in it and exploit the relatively high privileges it has to compromise the server. The solution to this is to also limit who has access to the application at all. A possible solution, which I’ve used, is to limit access only to the local machine and use SSH or a VPN tunnel to access phpMyAdmin. A better solution would be to use TLS/SSL Client Certificates. They operate on the connection level and provide both encryption and authentication. They are easier to set up than VPN tunnels and easier to use.

Note that limiting access based on a TLS/SSL Client Certificate can only be done on the subdomain level, because it happens as part of the connection, before any specific HTTP request can be made.

Most of the tutorial is not HTTP server-specific; however, the server configuration part relates to Nginx. As other servers (such as Lighttpd) use a very similar configuration for Client Certificates, adapting the instructions should be straightforward.

Creating a CA

The two commands below will create a CA private key and a corresponding self-signed certificate for you to sign the TLS client certificates with.

openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:2048 -aes-128-cbc -out ca.key
openssl req -new -x509 -days 365 -sha256 -key ca.key -out ca.pem

The first command will ask you for a passphrase for the key. It is used to protect access to the private key. You can decide not to use one by dropping the -aes-128-cbc option from the command.

The second command will ask you to provide some details to be included in the certificate. Those details will be sent to the browser by the web server to let it know which client certificate to send back when authenticating.

Server Configuration

Upload the ca.pem that was just generated to your server. You should not upload the private key (ca.key).

The following instructions are for Nginx.

ssl_client_certificate /path/to/ca.pem;
ssl_verify_client on; # we require client certificates to access

Assuming you already enabled TLS/SSL for the specific subdomain, your configuration should look something like this:

server {
        server_name subdomain.example.com;

        # SSL configuration
        #
        listen 443 ssl;
        listen [::]:443 ssl;

        ssl_certificate /etc/nginx/example.pem;
        ssl_certificate_key /etc/nginx/example.key;

        ssl_client_certificate /etc/nginx/ca.pem;
        ssl_verify_client on;

After reloading the server, check that everything is configured correctly by trying to access your site via HTTPS. It should report “400 Bad Request” and say that “No required SSL certificate was sent”.

Creating a Client Certificate

The following commands will create the private key used for the client certificate (client.key) and a corresponding Certificate Signing Request (client.csr), which the owner of the CA certificate can sign (which, in the case of this tutorial, will be you).

openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:2048 -out client.key
openssl req -new -key client.key -sha256 -out client.csr

You will be asked again to provide some details, this time about you. Those details will be available to the server once your browser sends it the client certificate. You can safely leave the “challenge password” empty1.

You can add the flag -aes-128-cbc to the first command if you want the private key for the client certificate to be encrypted. If you opt for it, you will be prompted for a passphrase just like before.

Signing a Client Certificate

The next step is to sign the certificate signing request from the last step. It is a good practice to review it and make sure all the details are as expected, so you do not sign anything you would not intend to.

openssl req -in client.csr -text -verify -noout | less

If everything looks just fine, you can sign it with the following command.

openssl x509 -req -days 365 -in client.csr -CA ca.pem -CAkey ca.key 
    -set_serial 0x`openssl rand 16 -hex` -sha256 -out client.pem

You will be prompted for the passphrase for ca.key if you chose one in the first step.

Installing Client Key

Now comes the final part, where we take the signed client certificate, client.pem, and combine it with the private key so it can be installed in our browser.

openssl pkcs12 -export -in client.pem -inkey client.key -name "Sub-domain certificate for some name" -out client.p12

Adjust the -name parameter to your liking. It will be used to identify the certificate in various places, such as the browser. If your private key was encrypted, you will be prompted to enter a passphrase for it. Encryption for certificates in p12 format is mandatory, so you will be prompted to enter a password for the generated file as well. It is OK to reuse the same password here, as those files are practically equivalent. Once imported to your browser, you will not need the password for normal usage, until you would like to import it to another browser.

GlobalSign provides instructions on how to actually install the p12 client certificate for browsers in Linux and Windows.

References


  1. It is used to “enhance” the security of a certificate revocation request, by requiring not only knowledge of the private key, but also the challenge password. Thus, someone who got hold of your private key cannot revoke the certificate by himself. However, this is also the reason why this option is not used more often: When someone steals your private key, usually they will prefer the certificate not to be revoked. 

Creating a Self-Signed ECDSA SSL Certificate Using OpenSSL

Before generating a private key, you’ll need to decide which elliptic curve to use. To list the supported curves, run:

openssl ecparam -list_curves

The list is quite long, and unless you know what you’re doing, you’ll be better off choosing one of the sect* or secp* curves. For this tutorial, I chose secp521r1 (a curve over a 521-bit prime).

Generating the certificate is done in two steps: first, we create the private key, and then we create the self-signed X.509 certificate:

openssl ecparam -name secp521r1 -genkey -param_enc explicit -out private-key.pem
openssl req -new -x509 -key private-key.pem -out server.pem -days 730

The newly created server.pem and private-key.pem are the certificate and the private key, respectively. The -param_enc explicit option tells OpenSSL to embed the full parameters of the curve in the key, as opposed to just its name. This allows clients that are not aware of the specific curve name to work with it, at the cost of slightly increasing the size of the key (and the certificate).

You can examine the key and the certificate using:

openssl ecparam -in private-key.pem -text -noout
openssl x509 -in server.pem -text -noout

Most web servers expect the private key to be chained to the certificate in the same file. So run:

cat private-key.pem server.pem > server-private.pem

And install server-private.pem as your certificate. If you don’t concatenate the private key to the certificate, at least Lighttpd will complain with the following error:

SSL: Private key does not match the certificate public key, reason: error:0906D06C:PEM routines:PEM_read_bio:no start line 

Manually Install SSL Certificate in Android Jelly Bean

Apparently it’s pretty easy, but there are some pitfalls. The first step is to export the certificate as a DER-encoded X.509 certificate. This can be done using Firefox (on a PC) by clicking the SSL lock icon in the address bar, then More Information -> View Certificate -> Details -> Export. The exported certificate needs to be saved in the root directory of the phone’s internal storage, with a *.cer extension (or *.crt). Other extensions will not work.

Afterward, on the phone, click on “Install from device storage” under Settings -> Security -> Credential Storage. If you did everything correctly in the previous step, it will display the certificate name and ask you to confirm its installation. If you’ve exported the certificate in the wrong format, given it the wrong extension, or placed it somewhere other than the root of the internal storage, it will display the following error:

No certificate file found in USB storage

If you see it, just make sure you are exporting the certificate correctly and saving it in the right place.

More details: Work with certificates (geared toward the Galaxy Nexus, but should apply to any Android 4.0 and above).

Updated Aug 2015: Fixed a broken link.

WordPress Administration over SSL on Lighttpd

In this tutorial, we’ll walk through the steps of enabling SSL (https) for the WordPress admin panel when using Lighttpd as a web server. The tutorial consists of two stages: the first is enabling SSL at the Lighttpd level, and the second is at the WordPress level.


Continue reading WordPress Administration over SSL on Lighttpd

Extract Public Key from X.509 Certificate as Hex

X.509 certificates are a common way to exchange and distribute public key information. For example, most Open Social containers use the OAuth RSA-SHA1 signature method and distribute their public keys in the X.509 format.

While working on an AppEngine application, I needed to verify requests from such containers. However, there is (currently) no pure Python library capable of parsing the certificates. This meant that I needed to extract the public key out of the certificate manually and store it in some parsed way inside the Python code.

Fortunately, parsing public keys from an X.509 certificate and representing them as a hex number turned out to be simple and easy.
Continue reading Extract Public Key from X.509 Certificate as Hex