Protecting OpenSSL private keys using PKCS#8

When generating private keys with OpenSSL, by default they are unprotected by a passphrase. For example:

$ openssl genrsa
-----BEGIN PRIVATE KEY----
MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQCY+q1kOPM4RF5T
...
Rjet4T2TrJmFzIL1dsgJACU=
-----END PRIVATE KEY-----

generates a plaintext private key. By using the -aes256 parameter, the generated key is protected according to PKCS#8:

$ openssl genrsa -aes256
Enter PEM pass phrase:
Verifying - Enter PEM pass phrase:
-----BEGIN ENCRYPTED PRIVATE KEY-----
MIIFLTBXBgkqhkiG9w0BBQ0wSjApBgkqhkiG9w0BBQwwHAQITQpYPEgrGN0CAggA
...
XIdi3hSPE8NoPWnROVGMnGbFOMm36g6064nZzQBD4r+4
-----END ENCRYPTED PRIVATE KEY-----

However, is it actually secure?

The default protection used by genrsa is PBKDF2 with 2048 (0x800) iterations of HMAC-SHA256. This can be verified by examining the PKCS#8 key, which is ASN.1 encoded.

$ openssl asn1parse -in key.priv 
    0:d=0  hl=4 l=1325 cons: SEQUENCE          
    4:d=1  hl=2 l=  87 cons: SEQUENCE          
    6:d=2  hl=2 l=   9 prim: OBJECT            :PBES2
   17:d=2  hl=2 l=  74 cons: SEQUENCE          
   19:d=3  hl=2 l=  41 cons: SEQUENCE          
   21:d=4  hl=2 l=   9 prim: OBJECT            :PBKDF2
   32:d=4  hl=2 l=  28 cons: SEQUENCE          
   34:d=5  hl=2 l=   8 prim: OCTET STRING      [HEX DUMP]:EAD8B97C8EAD1414
   44:d=5  hl=2 l=   2 prim: INTEGER           :0800
   48:d=5  hl=2 l=  12 cons: SEQUENCE          
   50:d=6  hl=2 l=   8 prim: OBJECT            :hmacWithSHA256
   60:d=6  hl=2 l=   0 prim: NULL              
   62:d=3  hl=2 l=  29 cons: SEQUENCE          
   64:d=4  hl=2 l=   9 prim: OBJECT            :aes-256-cbc
   75:d=4  hl=2 l=  16 prim: OCTET STRING      [HEX DUMP]:290DB8F1786A9C8667829A4A394A8FB7
   93:d=1  hl=4 l=1232 prim: OCTET STRING      [HEX DUMP]:D828C11F170D22B82801A567A9136AA293D630692B88C1F07987ED256C29DC45C4625709A0D36039B22E5A8BCA2B400C590C2560CED3629D1F8C5D77AD...CC5F8027A3A5ED533A00D626E7DDC4ABC85547

A meager 2048 SHA256 iterations provide very little added protection on top of the password’s intrinsic entropy.

Better protection can be achieved by using scrypt to protect the PKCS#8 encoded key. You can upgrade an existing key’s protection to scrypt:

openssl pkcs8 -in key.priv -topk8 -scrypt -out key-scrypt.priv

The command will prompt for the key’s passphrase and re-encrypt it using a key derived from the same passphrase with scrypt.

openssl pkcs8 -in key3.priv -topk8 -scrypt -out key5priv

The default openssl scrypt parameters, N=0x4000=2^14, r=8, p=1, are simply weak. They are what scrypt’s author recommended to achieve 100ms per iteration on his 2002-era laptop. While better than the PBKDF2 protection offered by OpenSSL, you might want stronger protection.

Choosing better scrypt parameters

Before we dive into the parameters, let’s understand the scrypt algorithm briefly. Scrypt takes three input parameters: password, salt, and cost parameters. The password and salt are user-provided inputs, whereas the cost parameters are fixed for a given implementation.

The cost parameters determine the computational cost of the algorithm. Higher cost parameters result in increased computational requirements, making it harder to brute-force the derived key. The cost parameters are as follows:

  • N: This is the CPU/memory cost parameter. It determines the number of iterations the algorithm performs. The value of N must be a power of two.
  • r: This is the block size parameter. It determines the size of the blocks of memory used during the algorithm’s execution.
  • p: This is the parallelization parameter. It doesn’t affect the memory consumption of the algorithm. However, despite its name, OpenSSL doesn’t perform any parallelization, instead opting to run the algorithm serially.

The original article recommends r=8 and p=1 as a good ratio between memory and CPU costs. This allows us to scale N almost arbitrarily, increasing both CPU and memory costs. However, OpenSSL, by default, limits its memory usage.

$ openssl pkcs8 -in key.priv -topk8 -scrypt -scrypt_N 0x8000 -out key-scrypt2.priv
Enter pass phrase for key.priv:
Error setting PBE algorithm
40E778DD277F0000:error:030000AC:digital envelope routines:scrypt_alg:memory limit exceeded:../providers/implementations/kdfs/scrypt.c:482:
40E778DD277F0000:error:068000E3:asn1 encoding routines:PKCS5_pbe2_set_scrypt:invalid scrypt parameters:../crypto/asn1/p5_scrypt.c:59:

This limitation severely restricts our parameter choices. The only option to increase security is through the CPU cost parameter (p). We can set it (almost) arbitrarily large, gaining added security. For example:

$ openssl pkcs8 -in MOK.priv -topk8 -scrypt -scrypt_p 32 -passin pass:1234 -passout pass:1234 -out key-scrypt3.priv

Split DNS using systemd-resolved

Many corporate environments have internal DNS servers that are required to resolve internal resources. However, you might prefer a different DNS server for external resources, for example 1.1.1.1 or 8.8.8.8. This allows you to use more secure DNS features like DNS over TLS (DoT). The solution is to set up systemd-resolved as your DNS resolver and configure it for split DNS resolution.

Starting with systemd 251, Debian ships systemd-resolved as a separate package. If it isn’t installed, go ahead and install it.

$ sudo apt install systemd-resolved
$ sudo systemctl enable --now systemd-resolved.service

Create the following configuration file under /etc/systemd/resolved.conf.d/99-split.conf:

[Resolve]
DNS=1.1.1.1#cloudflare-dns.com 1.0.0.1#cloudflare-dns.com 2606:4700:4700::1111#cloudflare-dns.com 2606:4700:4700::1001#cloudflare-dns.com

Domains=~.
DNSOverTLS=opportunistic

Domains=~. gives priority to the global DNS (1.1.1.1 in our case) over the link-local DNS configurations pushed through DHCP (like internal DNS servers).

DNSOverTLS=opportunistic defaults to DNS over TLS but allows fallback to regular DNS. This is useful when corporate DNS doesn’t support DNS over TLS and you still want to resolve corporate internal domains.

Restart systemd-resolved to reload the configuration:

$ sudo systemctl restart systemd-resolved

The final step is to redirect programs relying on /etc/resolv.conf (possibly through the glibc API) to the systemd-resolved resolver. The recommended way, according to the systemd-resolved man page, is to symlink it to /run/systemd/resolve/stub-resolv.conf.

$ sudo ln -rsf /run/systemd/resolve/stub-resolv.conf /etc/resolv.conf

F5 VPN

F5 VPN doesn’t play well with the above configuration. First, F5 VPN tries to overwrite the DNS configuration in /etc/resolv.conf by removing the existing file and replacing it with its own (pushing corporate DNS server configuration through it). The solution is to prevent F5 VPN from deleting /etc/resolv.conf by setting it to immutable. However, we cannot chattr +i a symlink. We have to resort to copying the configuration statically and then protecting it.

$ sudo cp /run/systemd/resolve/stub-resolv.conf /etc/resolv.conf
$ sudo chattr +i /etc/resolv.conf

Finally, because F5 VPN can’t update the DNS configuration, we have to manually configure the corporate DNS servers and the search domains.

$ sudo resolvectl dns tun0 192.168.100.20 192.168.100.22
$ sudo resolvectl domain tun0 ~example.corp ~example.local

Update: See Automating DNS Configurations for F5 VPN Tunnel using Systemd-resolved and NetworkManager-dispatcher for a script that automates the configuration.

Symbolized stack traces for a package in Debian

By default, Debian packages aren’t symbolized, resulting in unreadable stack traces:

#0  0x00007fb9d7a3a774 in ?? ()
#1  0x00005574a4450ea0 in ?? ()
#2  0x00005574a42cea60 in ?? ()
#3  0x00005574a3f8bd20 in ?? ()
#4  0x00007ffe0d782200 in ?? ()

The first step is to determine the right package containing the debug symbols for your binary. This can be done using find-dbgsym-packages from the debian-goodies package:

$ find-dbgsym-packages /usr/bin/gnome-control-center

Install the relevant *-dbgsym packages, for example:

$ sudo apt install gnome-control-center-dbgsym libglib2.0-0-dbgsym

And now you can have a symbolized stack trace:

#0  0x00007fb9d7a3a774 in g_type_check_instance_cast (type_instance=0x100000002, iface_type=93959409689392) at ../../../gobject/gtype.c:4122
#1  0x00005574a0d1382f in private_key_picker_helper (self=self@entry=0x5574a3f8bd20, filename=filename@entry=0x5574a42cea60 "*******************************************", changed=changed@entry=1)
    at ../panels/network/wireless-security/eap-method-tls.c:252
#2  0x00005574a0d13a34 in private_key_picker_file_set_cb (chooser=<optimized out>, user_data=0x5574a3f8bd20) at ../panels/network/wireless-security/eap-method-tls.c:297
#3  0x00007fb9d7a173b0 in g_closure_invoke (closure=0x5574a4302fb0, return_value=return_value@entry=0x0, n_param_values=2, param_values=param_values@entry=0x7ffe0d782200, invocation_hint=invocation_hint@entry=0x7ffe0d782180)
    at ../../../gobject/gclosure.c:832
#4  0x00007fb9d7a2a076 in signal_emit_unlocked_R (node=node@entry=0x5574a1335ba0, detail=detail@entry=1327, instance=instance@entry=0x5574a3f91350, emission_return=emission_return@entry=0x0, instance_and_params=instance_and_params@entry=0x7ffe0d782200)
    at ../../../gobject/gsignal.c:3796
#5  0x00007fb9d7a30bf5 in g_signal_emit_valist (instance=<optimized out>, signal_id=<optimized out>, detail=<optimized out>, var_args=var_args@entry=0x7ffe0d7823a0) at ../../../gobject/gsignal.c:3549

Further notes

By default, Debian doesn’t create core dumps. This can be changed (for the current terminal session) with:

$ ulimit -c unlimited

You can create more sensible core dump names using:

sudo sysctl -w kernel.core_pattern=/tmp/core-%e.%p.%h.%t