Moving Debian to a New Computer

These are the steps I took to migrate a Debian installation from an old computer to a new one. I took out the old SSD, and connected it via an external enclosure to the new computer, and booted via a live USB.

The next step is to copy over the entire disk from the old SSD to the new one. Because we will copy everything, even the partitions’ UUIDs will remain the same and no extra steps should be necessary apart from adjusting some partition sizes. Be very careful with the output and input devices. In my case the old SSD is connected as the external drive /dev/sdb and the new one is /dev/nvme0n1.

$ sudo dd if=/dev/sdb of=/dev/nvme0n1 bs=4K status=progress

Refresh the partition table:

$ sudo partprobe

Grow /dev/nvme0n1p3to fill the entire partition using gparted.

$ sudo cryptsetup --token-only open /dev/nvme0n1p3 new-root
$ sudo cryptsetup resize --token-only new-root 

(you can omit --token-only if you don’t use a Yubikey to unlock the drive).

Mount the btrfs root file system and resize it:

$ sudo mount -t btrfs /dev/mapper/new-root /mnt
$ sudo btrfs filesystem resize max /mnt	

Now you are ready to reboot into the new system.

Reencrypt the LUKS partition

Moving to a new SSD is also a good opportunity to rotate the master key of the LUKS encrypted root partition. This can be done while the disk is online and mounted, and takes some time.

The reencryption implementation doesn’t properly support FIDO2 keys for unlocking. We would have to delete those and re-register the keys afterwards. Select a key slot with a passphrase and pass it using the --key-slot parameter. You can check which key-slot is in use using cryptsetup luksDump

$ sudo cryptsetup reencrypt /dev/nvme0n1p3 --key-slot 1

Once done, re-enroll any FIDO2 keys you have by running the following command for each key:

$ sudo systemd-cryptenroll /dev/nvme0n1p3 --fido2-device=auto  --fido2-with-client-pin=yes

Enabling Secure Boot

Initially, I had problems with Secure Boot refusing to boot the new installation. They were resolved by reinstalling shim-signed and grub-efi-amd64-signed. Additionally, I had to enable “Allow Microsoft 3rd Party UEFI CA” in the Secure Boot settings of the UEFI:

Lenovo T14 Gen 4 Secure Boot settings

Install GNOME 44 on Debian Unstable

GNOME 44 recently got released. Because Debian Bookworm is undergoing a freeze, GNOME 44 is not yet available in Debian Unstable. It’s very similar to the GNOME 40 situation 2 years ago. While the wait for the freeze to be over can take a long time, Debian already has (most) of the updated pacakges in the experimental, and we can update to GNOME 44 through it:

$ sudo apt install -t experimental baobab eog evince gdm3 gjs gnome-backgrounds gnome-calculator gnome-characters gnome-contacts gnome-control-center gnome-disk-utility gnome-font-viewer gnome-keyring gnome-logs gnome-menus gnome-online-accounts gnome-remote-desktop gnome-session gnome-settings-daemon gnome-shell gnome-shell-extensions gnome-software gnome-system-monitor gnome-text-editor gnome-user-docs mutter gnome-desktop3-data
$ sudo apt-mark auto baobab eog evince gdm3 gjs gnome-backgrounds gnome-calculator gnome-characters gnome-contacts gnome-control-center gnome-disk-utility gnome-font-viewer gnome-keyring gnome-logs gnome-menus gnome-online-accounts gnome-remote-desktop gnome-session gnome-settings-daemon gnome-shell gnome-shell-extensions gnome-software gnome-system-monitor gnome-text-editor gnome-user-docs mutter gnome-desktop3-data

Split DNS using systemd-resolved

Many corporate environments have internal DNS servers that are required to resolve internal resources. However, you might prefer a different DNS server for external resources, for example 1.1.1.1 or 8.8.8.8. This allows you to use more secure DNS features like DNS over TLS (DoT). The solution is to set up systemd-resolved as your DNS resolver, and configure it for split DNS resolving.

Starting with systemd 251, Debian ships systemd-resolved as a separate package. If it isn’t installed, go ahead and install it.

$ sudo apt install systemd-resolved
$ sudo systemctl enable --now systemd-resolved.service

Create the following configuration file under /etc/systemd/resolved.conf.d/99-split.conf:

[Resolve]
DNS=1.1.1.1#cloudflare-dns.com 1.0.0.1#cloudflare-dns.com 2606:4700:4700::1111#cloudflare-dns.com 2606:4700:4700::1001#cloudflare-dns.com

Domains=~.
DNSOverTLS=opportunistic

Domains=~. gives priority to the global DNS (1.1.1.1 in our case) over the link-local DNS configurations which are pushed through DHCP (like internal DNS servers).

DNSOverTLS=opportunistic defaults to DNS over TLS but allows fallback to regular DNS. This is useful when corporate DNS doesn’t support DNS over TLS and you still want to resolve corporate internal domains.

Restart systemd-resolved to reload the configuration:

$ sudo systemctl restart systemd-resolved

The final step is to redirect programs relying on /etc/resolv.conf (possibly through the glibc API) to the systemd-resolved resolver. The recommended way according to the systemd-resolved man page is to symlink it to /run/systemd/resolv/stup-resolv.conf.

$ sudo ln -rsf /run/systemd/resolve/stub-resolv.conf /etc/resolv.conf

F5 VPN

F5 VPN doesn’t play well with the above configuration. First, F5 VPN tries to overwrite the DNS configuration in /etc/resolv.conf, by removing the existing file and replacing it with its own (pushing corporate DNS server configuration through it). The solution is to prevent F5 VPN from deleting the /etc/resolv.conf, by setting it to immutable. However, we cannot chattr +i a symlink. We have to resort to copying the configuration statically, and then protect it.

$ sudo cp /run/systemd/resolve/stub-resolv.conf /etc/resolv.conf
$ sudo chattr +i /etc/resolv.conf

Finally, because now F5 VPN can’t update the DNS configuration, we would have to manually configure the corporate DNS servers and the search domains.

$ sudo resolvectl dns tun0 192.168.100.20 192.168.100.22
$ sudo resolvectl domain tun0 ~example.corp ~example.local

Symbolized stacktraces for a package in Debian

By default, Debian packages aren’t symbolized, resulting in unreadable stacktraces:

#0  0x00007fb9d7a3a774 in ?? ()
#1  0x00005574a4450ea0 in ?? ()
#2  0x00005574a42cea60 in ?? ()
#3  0x00005574a3f8bd20 in ?? ()
#4  0x00007ffe0d782200 in ?? ()

The first step is to determine the right package containing the debug symbols for your binary. This can be done using find-dbgsym-packages from the debian-goodies packages:

$ find-dbgsym-packages /usr/bin/gnome-control-center

Install the relevant *-dbgsym packages, for example:

$ sudo apt install gnome-control-center-dbgsym libglib2.0-0-dbgsym

And now you can have symbolized stacktrace:

#0  0x00007fb9d7a3a774 in g_type_check_instance_cast (type_instance=0x100000002, iface_type=93959409689392) at ../../../gobject/gtype.c:4122
#1  0x00005574a0d1382f in private_key_picker_helper (self=self@entry=0x5574a3f8bd20, filename=filename@entry=0x5574a42cea60 "*******************************************", changed=changed@entry=1)
    at ../panels/network/wireless-security/eap-method-tls.c:252
#2  0x00005574a0d13a34 in private_key_picker_file_set_cb (chooser=<optimized out>, user_data=0x5574a3f8bd20) at ../panels/network/wireless-security/eap-method-tls.c:297
#3  0x00007fb9d7a173b0 in g_closure_invoke (closure=0x5574a4302fb0, return_value=return_value@entry=0x0, n_param_values=2, param_values=param_values@entry=0x7ffe0d782200, invocation_hint=invocation_hint@entry=0x7ffe0d782180)
    at ../../../gobject/gclosure.c:832
#4  0x00007fb9d7a2a076 in signal_emit_unlocked_R (node=node@entry=0x5574a1335ba0, detail=detail@entry=1327, instance=instance@entry=0x5574a3f91350, emission_return=emission_return@entry=0x0, instance_and_params=instance_and_params@entry=0x7ffe0d782200)
    at ../../../gobject/gsignal.c:3796
#5  0x00007fb9d7a30bf5 in g_signal_emit_valist (instance=<optimized out>, signal_id=<optimized out>, detail=<optimized out>, var_args=var_args@entry=0x7ffe0d7823a0) at ../../../gobject/gsignal.c:3549

Further notes

By default, Debian doesn’t create core dumps. This can be changed (for the current running terminal session) with

$ ulimit -c unlimited

You can create more sensible core dump names using:

sudo sysctl -w kernel.core_pattern=/tmp/core-%e.%p.%h.%t

Display reboot required message on Debian

You can use MOTD (message of the day) to let you know if a Debian server requires reboot and why upon login.

Create a new file named /etc/update-motd.d/98-reboot-required and add to it the following lines:

#!/bin/sh -e
#
# helper for update-motd

if [ -f /var/run/reboot-required ]; then
	echo "*** System restart required ***"
        cat /var/run/reboot-required.pkgs
fi

Make the file executable:

$ sudo chmod +x /etc/update-motd.d/98-reboot-required

Now, you can test the new MOTD script using:

$ run-parts --lsbsysinit /etc/update-motd.d

If you have any installed updates that require reboot, you will get a message stating so, with a list of the packages that require the reboot.

*** System restart required ***
linux-image-5.10.0-19-cloud-amd64

Signing kernel modules for Secure Boot

Some time ago, I needed to use the v4l2loopback module. It can be installed via:

$ sudo apt install v4l2loopback-dkms

Normally, after installing a module, you can just modprobe it, and it will load. However, due to Secure Boot, it will fail.

$ sudo modprobe v4l2loopback 
modprobe: ERROR: could not insert 'v4l2loopback': Operation not permitted

The problem is that the v4l2loopback isn’t signed. For example, compare the output of:

$ /usr/sbin/modinfo -F signer v4l2loopback

which is empty, versus

$ /usr/sbin/modinfo -F signer xor
Debian Secure Boot CA

The solution would be to sign the v4l2loopback module ourselves.

Creating a key

The update-secureboot-policy script available in Ubuntu’s shim-signed package is able to generate Machine Owner Keys (MOK) by itself. However, the currently available in Debian Unstable doesn’t have the key generation functionality. We can either fetch the Ubuntu version or generate the keys ourselves.

$ wget https://git.launchpad.net/ubuntu/+source/shim-signed/plain/update-secureboot-policy
$ chmod +x ./update-secureboot-policy
$ sudo ./update-secureboot-policy --new-key

Or through generating the keys ourselves:

$ sudo mkdir -p /var/lib/shim-signed/mok
$ cd /var/lib/shim-signed/mok/
$ sudo openssl genrsa -aes256 -out MOK.priv
$ sudo openssl req \
        -subj "/CN=`hostname -s | cut -b1-31` Secure Boot Module Signature key" \
        -new -x509 -nodes -days 36500 -outform DER \
        -key MOK.priv \
        -out MOK.der

Write down the passphrase for your private key. You will need it whenever you want to sign drivers.

Now we enroll the newly created key:

$ sudo mokutil --import MOK.der

You will be prompted for a password. This password will be required after reboot in order to complete the key enrollment, you will not need it afterwards.

After reboot, check that your key was indeed enrolled:

$ mokutil --list-enrolled

Signing the module

We need to put the passphrase for the private key in the KBUILD_SIGN_PIN env variable:

$ read -s KBUILD_SIGN_PIN
$ export KBUILD_SIGN_PIN

Now we can do the actual signing:

$ cd /usr/lib/modules/$(uname -r)/updates/dkms
$ sudo --preserve-env=KBUILD_SIGN_PIN /usr/lib/linux-kbuild-$(uname -r | cut -d. -f1-2)/scripts/sign-file sha256 /var/lib/shim-signed/mok/MOK{.priv,.der} v4l2loopback.ko

You will need to repeat this step for every new kernel that you install.

Setting up WireGuard on Debian

WireGuard is a modern VPN solution, which is much easier to configure than OpenVPN and its likes. In this tutorial, we assume a simple setup where we want to route all clients network traffic through the VPN, exiting through the server.

Server configuration

$ sudo apt install wireguard
$ PRIVATE_KEY=$(wg genkey)

Now we create the configuration file for our tunnel (wg0).

$ cat <<EOF | sudo tee /etc/wireguard/wg0.conf
[Interface]
PrivateKey = $PRIVATE_KEY
Address = 10.8.0.1/24, fd0d:d325:45c0::1/64
ListenPort = 51820
SaveConfig = true

PostUp = iptables -t nat -I POSTROUTING -o eth0 -j MASQUERADE
PostUp = ip6tables -t nat -I POSTROUTING -o eth0 -j MASQUERADE
PreDown = iptables -t nat -D POSTROUTING -o eth0 -j MASQUERADE
PreDown = ip6tables -t nat -D POSTROUTING -o eth0 -j MASQUERADE
EOF

If you have a firewall, you’ll need to open up UDP port 51820 (or whatever configured as the ListenPort).

Enable IPv4 and IPv6 forwarding, so we can route all the client traffic through the server (and reload sysctl configuration)

$ echo net.ipv4.ip_forward=1 | sudo tee /etc/sysctl.d/40-wireguard.conf
$ echo net.ipv6.conf.all.forwarding=1 | sudo tee -a /etc/sysctl.d/40-wireguard.conf
$ sudo sysctl --system 

Enable and start the WireGuard service:

$ sudo systemctl enable --now wg-quick@wg0.service

Finally, take note of the server public key (it will be a short base64 encoded string):

$ echo $PRIVATE_KEY | wg pubkey

You’ll need it in the next step.

Peer configuration

$ sudo apt install wireguard
$ PRIVATE_KEY=$(wg genkey)
$ PUBLIC_KEY=$(echo $PRIVATE_KEY | wg pubkey)
$ SERVER_PUBLIC_KEY=6TDw+U2WFhkaKUy/xXrCRtuZvB2m2SFN7URZA5AkGis=
$ SERVER_IP=8.8.8.8

Replace the value of SERVER_PUBLIC_KEY with the public key of your server, and SERVER_IP with the correct IP address of your server.

Edit /etc/wireguard/wg0.conf

$ cat <<EOF | sudo tee /etc/wireguard/wg0.conf
[Interface]
PrivateKey = $PRIVATE_KEY
Address = 10.8.0.2/24, fd0d:d325:45c0::2/64

[Peer]
PublicKey = $SERVER_PUBLIC_KEY
AllowedIPs = 0.0.0.0/0, ::/0
Endpoint = $SERVER_IP:51820
EOF

Setting AllowedIPs = 0.0.0.0/0, ::/0 will route all traffic through the VPN connection. If you don’t want to do that, edit the configuration file, and set AllowedIPs = 10.8.0.0/24, fd0d:d325:45c0::0/64.

We need to make the server aware of the peer. The following command should be executed on the server.

$ sudo wg set wg0 peer $PUBLIC_KEY allowed-ips 10.8.0.2,fd0d:d325:45c0::2

where PUBLIC_KEY is the value of the client’s public key (stored in the $PUBLIC_KEY environment variable).

The /etc/wireguard/wg0.conf will be updated according and make the added peer configuration persistent due to the SaveConfig = true. The configuration update will take place the next time the WireGuard interface goes down.

You can also import the profile into NetworkManager and easily enable/disable the new interface via GNOME:

$ sudo nmcli connection import type wireguard file /etc/wireguard/wg0.conf

Android Peer

It is also useful to have WireGuard on the phone. WireGuard supports both iOS and Android, and the setup should be similar in both cases. Start by installing WireGuard from the Play Store. The next step is to generate the required configuration. It can be done directly on the phone, or by creating a configuration file on your computer and transferring it, which I find simpler.

$ PRIVATE_KEY=$(wg genkey)
$ PUBLIC_KEY=$(echo $PRIVATE_KEY | wg pubkey)
$ SERVER_PUBLIC_KEY=6TDw+U2WFhkaKUy/xXrCRtuZvB2m2SFN7URZA5AkGis=
$ SERVER_IP=8.8.8.8
$ cat <<EOF > wg0.conf
[Interface]
PrivateKey = $PRIVATE_KEY
Address = 10.8.0.3/24, fd0d:d325:45c0::3/64

[Peer]
PublicKey = $SERVER_PUBLIC_KEY
AllowedIPs = 0.0.0.0/0, ::/0
Endpoint = $SERVER_IP:51820
EOF

Again, make the server aware of the client by running the following command on the server:

$ sudo wg set wg0 peer $PUBLIC_KEY allowed-ips 10.8.0.3,fd0d:d325:45c0::3

Transfer the configuration file wg0.conf to the phone and load it using the WireGuard app.

Update 2023-11-30: Added IPv6 configuration.