Regenerate Dracut initramfs images from a live USB

Thanks to an incompatability between dracut and systemd version 256, I was left with an unbootable system. I booted a live USB system, downgraded systemd back to 255, but I had to regenerate the initramfs images to be compatible with my old systemd.

While using systemd-nspawn is convenient for modifying the system from a live usb, generating dracut initramfs images through it, resulted in missing hardware support (for example, no nvme module) due to the abstracted hardware in the container. The solution is to revert to the traditional chroot and generate the images from there:

$ sudo /usr/lib/systemd/systemd-cryptsetup attach root /dev/nvme0n1p3
$ sudo mount /dev/mapper/root /mnt/
$ sudo mount /dev/nvme0n1p2 /mnt/boot/
$ sudo mount --bind /dev/ /mnt/dev/
$ sudo mount --bind /proc/ /mnt/proc/
$ sudo mount --bind /sys/ /mnt/sys/
$ sudo chroot /mnt/
# dracut -f --regenerate-all

Automating DNS Configurations for F5 VPN Tunnel using Systemd-resolved and NetworkManager-dispatcher

F5 VPN does not play well with split DNS configuration using systemd-resolved because it insists on trying to rewrite /etc/resolv.conf. The workaround is to make resolv.conf immutable, and configure the DNS settings for the tunnel manually. systemd-resolved does not have a mechanism for persistant per-interface configuration, and it relies on NetworkManager to configure each connection correctly. F5 VPN is not compatible with NetworkManager, and does not make it easy to configure it this way.

NetworkManager-dispatcher allows you to run scripts based on network events. In our case, we will use it to automatically add DNS configurations when the F5 VPN tunnel tun0 is up, and thus provide persistent configuration.

Here is the script:



case "$STATUS" in
        if [ "$INTERFACE" = "tun0" ]; then
            # Add your search domains here
            SEARCH_DOMAINS="~example.corp ~example.local"

            resolvectl domain "$INTERFACE" $SEARCH_DOMAINS
            resolvectl dns $INTERFACE
            resolvectl dnsovertls tun0 no

The script checks if the interface is tun0 and if the current action is up. If so, it uses resolvectl to configure search domains and local DNS servers. Lastly, DNS over TLS is disabled, as the corporate DNS servers do not support them.

To make this script work, install in the /etc/NetworkManager/dispatcher.d/ directory with the name f5-vpn. Make sure it’s executable and only writable by root. NetworkManager-dispatcher will run this script whenever a network interface goes up, automatically setting the DNS configurations for F5 VPN tunnel.

Nginx: Block access to sensitive files

As more websites and applications are being hosted on the web, the necessity for securing these resources is skyrocketing. Among various web servers, Nginx has gained substantial popularity due to its performance, flexibility, and stability. Despite its many merits, securing your applications can always be a tricky business. But fret not, securing sensitive files in Nginx is achievable by configuring the Nginx server to block attempts directed at accessing specific files or file types.

This blog post will guide you through a way to enhance the Nginx server’s security by blocking access to sensitive files. As seen in the code snippet in the block_bad.conf file, we will deny access to all dotfiles, files ending with ~ or .bak, and the dump.sql file which often contains database dumps.

Start by saving the following code snippet in the /etc/nginx/snippets/block_bad.conf:

# deny access to all dotfiles
location ~ /\. {
	deny all;
	log_not_found off;
	access_log off;
	return 404;

# Allow access to the ".well-known" directory
location ^~ /.well-known {
        allow all;

# deny access to files ending with ~ or .bak
location ~ ~$ {
	deny all;
	log_not_found off;
	access_log off;
	return 404;
location ~ \.bak$ {
	deny all;
	log_not_found off;
	access_log off;
	return 404;

location ~ /dump.sql$ {
	deny all;
	log_not_found off;
	access_log off;
	return 404;

By using the deny all; directive within the location blocks, Nginx will disallow access to the specified locations. The access_log off; and log_not_found off; directives prevent logging access to these blocked file types, and the return 404; directive makes Nginx return a 404 error when an attempt is made to access these files.

Specifically, the location ~ /\. clause will block access to all dotfiles. These files usually store configuration for individual programs, revealing them might cause security vulnerabilities. The next location block with the pattern ~ ^/.well-known overrides the previous one by specifically allowing access to URIs startings with .well-known (which is used by certbot).

The next set of directives beginning with location ~ ~$ and location ~ \.bak$ blocks access to files ending with ~ or .bak. These files are often temporary files or backup copies which may accidentally leak sensitive information if accessed by malicious actors.

Lastly, the location ~ /dump.sql$ rule denies access to dump.sql files. These files may contain database dumps, which can expose sensitive database information.

After saving this file, you need to include it in your server block configuration using the include directive. Open the Nginx server block configuration. Then, to load the block_bad.conf, add include snippets/block_bad.conf; in each server block that you want to apply these security rules. Restart the Nginx service after including this rule to apply the changes.

By taking the steps outlined in this guide, you can enhance the security of your Nginx server by preventing unauthorized access to sensitive files and thus reducing potentials for information breach. Remember, securing your servers is a continual process, and consistently updating your configurations to counter evolving threats is a good practice.

Changing the webcam’s default power-line frequency

The default powerline frequency for the Logitech C270 webcam is 60Hz, which causes flickering. It can be changed manually via v4l2-ctl or cameractrls, but the change isn’t permanent. To persist the change, we need to create a udev rule. Put the following lines in /etc/udev/rules.d/99-logitech-default-powerline.rules:

# 046d:0825 Logitech, Inc. Webcam C270
SUBSYSTEM=="video4linux", KERNEL=="video[0-9]*", ATTR{index}=="0", ATTRS{idVendor}=="046d", ATTRS{idProduct}=="0825", RUN+="/usr/bin/v4l2-ctl -d $devnode --set-ctrl=power_line_frequency=1"

power_line_frequency value 1 corresponds to 50Hz, 2 to 60Hz and 0 disables the anti-flickering algorithm.

How to Display Battery Percentage for Bluetooth Headphones in GNOME

By default, the Power tab in GNOME’s Settings does not show the battery percentage for Bluetooth headphones like the Sony WH-1000XM3. However, you can enable this feature by activating the DBUS interface of Bluez, the Linux Bluetooth protocol stack. The DBUS interface is hidden behind the --experimental flag for the Bluez service. To enable it, follow these steps:

  1. Create an override file for the bluetooth service:
$ sudo systemctl edit bluetooth

This command will create the file /etc/systemd/system/bluetooth.service.d/override.conf.

  1. Add the following lines to the file:
ExecStart=/usr/libexec/bluetooth/bluetoothd --experimental

Note that both ExecStart= lines are required.

  1. Restart the Bluetooth service.
Battery percentage for Sony WH-1000XM3 under Settings->Power

Moving Debian to a New Computer

These are the steps I took to migrate a Debian installation from an old computer to a new one. I took out the old SSD, and connected it via an external enclosure to the new computer, and booted via a live USB.

The next step is to copy over the entire disk from the old SSD to the new one. Because we will copy everything, even the partitions’ UUIDs will remain the same and no extra steps should be necessary apart from adjusting some partition sizes. Be very careful with the output and input devices. In my case the old SSD is connected as the external drive /dev/sdb and the new one is /dev/nvme0n1.

$ sudo dd if=/dev/sdb of=/dev/nvme0n1 bs=4K status=progress

Refresh the partition table:

$ sudo partprobe

Grow /dev/nvme0n1p3to fill the entire partition using gparted.

$ sudo cryptsetup --token-only open /dev/nvme0n1p3 new-root
$ sudo cryptsetup resize --token-only new-root 

(you can omit --token-only if you don’t use a Yubikey to unlock the drive).

Mount the btrfs root file system and resize it:

$ sudo mount -t btrfs /dev/mapper/new-root /mnt
$ sudo btrfs filesystem resize max /mnt	

Now you are ready to reboot into the new system.

Reencrypt the LUKS partition

Moving to a new SSD is also a good opportunity to rotate the master key of the LUKS encrypted root partition. This can be done while the disk is online and mounted, and takes some time.

The reencryption implementation doesn’t properly support FIDO2 keys for unlocking. We would have to delete those and re-register the keys afterwards. Select a key slot with a passphrase and pass it using the --key-slot parameter. You can check which key-slot is in use using cryptsetup luksDump

$ sudo cryptsetup reencrypt /dev/nvme0n1p3 --key-slot 1

Once done, re-enroll any FIDO2 keys you have by running the following command for each key:

$ sudo systemd-cryptenroll /dev/nvme0n1p3 --fido2-device=auto  --fido2-with-client-pin=yes

Enabling Secure Boot

Initially, I had problems with Secure Boot refusing to boot the new installation. They were resolved by reinstalling shim-signed and grub-efi-amd64-signed. Additionally, I had to enable “Allow Microsoft 3rd Party UEFI CA” in the Secure Boot settings of the UEFI:

Lenovo T14 Gen 4 Secure Boot settings

Creating a WireGuard profile for Cloudflare Warp

Connecting to Cloudflare Warp directly via wg can have advantages in flexibility or specific scenarios. For example, the Warp client, warp-cli would refuse to establish connection if it can’t override /etc/resolve.conf. By connecting directly using WireGuard, you get control over all that.

The first step is to install warp-cli and register using warp-cli register. This will create the WireGuard private-key used for the connection and register it with Cloudflare. The private key can be found in /var/lib/cloudflare-warp/reg.json. The endpoint data and Cloudflare’s public key should be constant. Alternative endpoints are listed in /var/lib/cloudflare-warp/conf.json.

An easy way to read the json configuration files is using jq:

$ sudo jq . /var/lib/cloudflare-warp/conf.json

Adjust the following template accordingly, and put in int /etc/wireguard/warp.conf:

Address =
Address = 2606:4700:110:892f:607d:85a6:5e07:70cf/128
PublicKey = bmXOC+F1FxEMF9dyiK2H5/1SUtzH0JuVo51h2wPfgyo=
AllowedIPs =, ::/0
Endpoint =

You can start the tunnel using

$ sudo wg-quick up warp`

Alternatively, you can import it to NetworkManager and be able to easily start it from the Gnome Quick Settings.

$ sudo nmcli connection import type wireguard file /etc/wireguard/warp.conf

You can easily check that the tunnel works, by visiting and looking for the line that says warp=on.

Sometimes, IPv4 won’t work while IPv6 works. Restarting the VPN several times can resolve the issue.

while ! ping -w1 -c1; do wg-quick down wgcf-profile; wg-quick up wgcf-profile; done

or using nmcli:

while ! ping -w1 -c1; do nmcli connection down wgcf-profile; nmcli connection up wgcf-profile; done

Disabling the Cloudflare client

The Cloudflare client might interfere with the Wireguard profile. It’s best to didable it:

$ sudo systemctl disable --now  warp-svc.service
$ systemctl --user disable --now warp-taskbar.service

Install GNOME 44 on Debian Unstable

GNOME 44 recently got released. Because Debian Bookworm is undergoing a freeze, GNOME 44 is not yet available in Debian Unstable. It’s very similar to the GNOME 40 situation 2 years ago. While the wait for the freeze to be over can take a long time, Debian already has (most) of the updated pacakges in the experimental, and we can update to GNOME 44 through it:

$ sudo apt install -t experimental baobab eog evince gdm3 gjs gnome-backgrounds gnome-calculator gnome-characters gnome-contacts gnome-control-center gnome-disk-utility gnome-font-viewer gnome-keyring gnome-logs gnome-menus gnome-online-accounts gnome-remote-desktop gnome-session gnome-settings-daemon gnome-shell gnome-shell-extensions gnome-software gnome-system-monitor gnome-text-editor gnome-user-docs mutter gnome-desktop3-data
$ sudo apt-mark auto baobab eog evince gdm3 gjs gnome-backgrounds gnome-calculator gnome-characters gnome-contacts gnome-control-center gnome-disk-utility gnome-font-viewer gnome-keyring gnome-logs gnome-menus gnome-online-accounts gnome-remote-desktop gnome-session gnome-settings-daemon gnome-shell gnome-shell-extensions gnome-software gnome-system-monitor gnome-text-editor gnome-user-docs mutter gnome-desktop3-data

Protecting OpenSSL private keys using PKCS#8

When generating private keys with OpenSSL, by default they are unprotected by a passphrase. For example:

$ openssl genrsa

generates a plaintext private key. By using the -aes256 parameters, the generated key is protected according to PKCS#8:

$ openssl genrsa -aes256
Enter PEM pass phrase:
Verifying - Enter PEM pass phrase:

However, is it actually secure?

The default protection used by genrsa is PBKDF2 with 2048 (0x800) iterations of HMAC-SHA256. This can be verified by examining the PKCS#8 key, which is ASN.1 encoded.

$ openssl asn1parse -in key.priv 
    0:d=0  hl=4 l=1325 cons: SEQUENCE          
    4:d=1  hl=2 l=  87 cons: SEQUENCE          
    6:d=2  hl=2 l=   9 prim: OBJECT            :PBES2
   17:d=2  hl=2 l=  74 cons: SEQUENCE          
   19:d=3  hl=2 l=  41 cons: SEQUENCE          
   21:d=4  hl=2 l=   9 prim: OBJECT            :PBKDF2
   32:d=4  hl=2 l=  28 cons: SEQUENCE          
   34:d=5  hl=2 l=   8 prim: OCTET STRING      [HEX DUMP]:EAD8B97C8EAD1414
   44:d=5  hl=2 l=   2 prim: INTEGER           :0800
   48:d=5  hl=2 l=  12 cons: SEQUENCE          
   50:d=6  hl=2 l=   8 prim: OBJECT            :hmacWithSHA256
   60:d=6  hl=2 l=   0 prim: NULL              
   62:d=3  hl=2 l=  29 cons: SEQUENCE          
   64:d=4  hl=2 l=   9 prim: OBJECT            :aes-256-cbc
   75:d=4  hl=2 l=  16 prim: OCTET STRING      [HEX DUMP]:290DB8F1786A9C8667829A4A394A8FB7
   93:d=1  hl=4 l=1232 prim: OCTET STRING      [HEX DUMP]:D828C11F170D22B82801A567A9136AA293D630692B88C1F07987ED256C29DC45C4625709A0D36039B22E5A8BCA2B400C590C2560CED3629D1F8C5D77AD...CC5F8027A3A5ED533A00D626E7DDC4ABC85547

A meager 2048 SHA256 iterations provide very little added protection on top of the password intrinsic entropy.

Better protection can be achieved by using scrypt to protect the PKCS#8 encoded key. You can upgrade an existing key’s protection to scrypt:

openssl pkcs8 -in key.priv -topk8 -scrypt -out key-scrypt.priv

The command will prompt for the key’s passphrase and re-encrypt it using a key derived from the same passphrase using scrypt.

openssl pkcs8 -in key3.priv -topk8 -scrypt -out key5priv

The default openssl scrypt parameters, N=0x4000=2^14, r=8, p=1, are simply weak. They are what scrypt’s author recommended to achieve 100ms per iteration on his 2002-era laptop. While better than the PBKDF2 protection offered by OpenSSL, you might want stronger protection.

Choosing better scrypt parameters

Before we dive into the parameters, let’s understand the scrypt algorithm briefly. Scrypt takes three input parameters: password, salt, and cost parameters. The password and salt are user-provided inputs, whereas the cost parameters are fixed for a given implementation.

The cost parameters determine the computational cost of the algorithm. Higher cost parameters result in increased computational requirements, making it harder to brute-force the derived key. The cost parameters are as follows:

  • N: This is the CPU/memory cost parameter. It determines the number of iterations the algorithm performs. The value of N must be a power of two.
  • r: This is the block size parameter. It determines the size of the blocks of memory used during the algorithm’s execution.
  • p: This is the parallelization parameter. Doesn’t affect the memory consumption of the algorithm. However, despite it’s name, OpenSSL doesn’t perform any parallelization, instead opting to run the algorithm serially.

The original article recommends r=8 and p=1 as a good ratio between memory and CPU costs. This allows us to scale N almost arbitrary, increasing both CPU and memory costs. However, OpenSSL, by default, limits it’s memory usage.

$ openssl pkcs8 -in key.priv -topk8 -scrypt -scrypt_N 0x8000 -out key-scrypt2.priv
Enter pass phrase for key.priv:
Error setting PBE algorithm
40E778DD277F0000:error:030000AC:digital envelope routines:scrypt_alg:memory limit exceeded:../providers/implementations/kdfs/scrypt.c:482:
40E778DD277F0000:error:068000E3:asn1 encoding routines:PKCS5_pbe2_set_scrypt:invalid scrypt parameters:../crypto/asn1/p5_scrypt.c:59:

This limitation severely our parameters choice. The only option to increase security, is through the CPU cost parameter (p). We can set it (almost) arbitrary large, getting added security. For example:

$ openssl pkcs8 -in MOK.priv -topk8 -scrypt -scrypt_p 32 -passin pass:1234 -passout pass:1234 -out key-scrypt3.priv