Quickly generate lots of random data

What is the quickest way to generate lots of random data on the command line? Usually when I had to wipe hard-drives I would simply use dd to copy from /dev/urandom over the device. However, `/dev/urandom is quite slow and wiping hard-disks can take a long time that way. So, I decided to benchmark a few methods to generate long random streams that are usable in such scenarios.

The benchmark is based on the dd command. For example:

$ dd if=/dev/urandom of=/dev/null bs=4k count=1M

This command will copy a 4GB of random bytes from /dev/urandom over /dev/null. This is probably the simplest method to create a large stream of random bytes, and as it turns out, also the slowest.

The second construct I tried is to use OpenSSL to create a stream of random data which I can read with dd and then write to the target. For example the following would use AES-128 with a random key:

$ openssl rand -hex 32 | openssl enc -aes-128-ctr -in /dev/zero -pass stdin -nosalt | dd if=/dev/stdin of=/dev/null bs=4k count=1M

Let’s breakup this command: openssl rand -hex 32 will generate a random encryption key to be used by the AES encryption. openssl enc -aes-128-ctr -in /dev/zero -pass stdin -nosalt does the actual encryption. It reads the (random) key from stdin and then uses it to encrypt /dev/zero using AES-128 in counter mode. As /dev/zero in an endless stream of zeros, it will simply output an endless stream of (pseudo-)random data. We can also repeat the same command only swapping aes-128-ctr with aes-256-ctr. For most (all?) usage scenarios it doesn’t provided any added security benefits but does have a (small) performance penalty.

Apart from AES, which is a block cipher, we can also try to use actual stream ciphers like the old rc4 and the modern chacha20.

Additionally, many new CPUs come with AES-NI extension which speeds up AES operations considerably. We can repeat the benchmark while disabling AES-NI to see how the different methods will perform if used a CPU that doesn’t support AES-NI.

Finally, I’ve repeated the test with /dev/zero as input, just to have an upper-limit in terms of performance to compare against.

Benchmark results
AES-NINo AES-NI
/dev/zero0.42609
/dev/urandom18.8967
chacha202.693063.79217
aes-128-ctr2.0410614.9022
aes-256-ctr2.2475618.9014
rc47.77392
Benchmark results, time (in seconds) to create 4GB of random data

Conclusions

The results clearly show that you should avoid /dev/urandom. It’s simply not suitable for this task and doesn’t perform well. The various methods of using OpenSSL perform much better. The best performance is achieved by the two AES variants, with aes-128-ctr being the fastest. However, if AES-NI is not supported by the CPU, AES takes a huge performance hit, and is even slower than the (not-so-)good and old RC4. However, ChaCha20 (a modern stream cipher) performs within 30% of AES if AES-NI is available, but if AES-NI is not supported ChaCha20 outperforms the AES variants. So, unless you know AES-NI is supported ChaCha20 is the safe choice.

Fix missing icons in digiKam

It looks like digiKam installed on a default Gnome environment has missing icons. For example the "pick" icons (the little flags for Rejected/Pending/Accepted) are missing. The reason is that the default Gnome icon pack, Adwaita is missing some of the icons used by digiKam.

The solution is to install the Breeze icon theme and then select it in digiKam:

$ sudo apt install breeze-icon-theme

and then in digikam Settings -> Configure digiKam -> Miscellaneous -> Appearance -> Icon theme and select "Breeze". Actually you can leave it as "Use Icon Theme From System" and it will use Adwaita and only fall back to Breeze for missing icons. However, I do find it more pleasant to have a consistent icon theme.

Short cryptsetup/LUKS tutorial

This short tutorial will guide you in encrypting a drive with cryptsetup and LUKS scheme.

Before starting, if the device had previous data on it, it’s best to delete any filesystem signatures that may be on it. Assuming that the drive we operate is /dev/sda you can use the following command to remove the signatures:

$ sudo wipefs --all /dev/sda --no-act

Remove the --no-act flag to actually modify the disk.

The next step is to actually format the drive using LUKS. This is done using the cryptsetup utility.

$ sudo cryptsetup luksFormat --type=luks2 /dev/sda

WARNING!
========
This will overwrite data on /dev/sda irrevocably.

Are you sure? (Type 'yes' in capital letters): YES
Enter passphrase for /dev/sda: 
Verify passphrase: 

The command will prompt you to enter a passphrase for the encryption and should take a few seconds to complete.

The next step is to add an appropriate entry to crypttab which will simplify starting the dm-crypt mapping later. Add the following line to /etc/crypttab:

archive_crypt UUID=114d42e5-6aeb-4af0-8758-b4cc79dd1ba0 none luks,discard,noauto

where the UUID is obtained through lsblk /dev/sda -o UUID or a similar command. The archive_crypt is the name for the mapped device. It will appear as /dev/mapper/archive_crypt when the device is mapped. The none parameter specifies that no keyfile is used and the system should prompt for an encryption passphrase instead. The noauto, means not to attempt to load the device automatically upon boot. discard should be used if the underlying device is an SSD.

You can test everything works so far by opening and loading the LUKS device:

$ sudo cryptdisks_start archive_crypt

While the device is now encrypted, there is a possible leakage of metadata such as used blocks as an attacker can discern used vs unused blocks by examining the physical drive. This and other side-channel leaks can be mitigated by simply wiping the contents of the encrypted device.

$ openssl rand -hex 32 | openssl enc -chacha20 -in /dev/zero -pass stdin -nosalt | sudo dd if=/dev/stdin of=/dev/mapper/sda_crypt bs=4096 status=progress

We could also have used /dev/urandom but the above technique is much faster.

Now we can create the actual filesystem.

$ sudo mkfs.btrfs --label archive /dev/mapper/archive_crypt

At this point we’re actually pretty much done. You can add and entry to /etc/fstab to easily mount the filesystem and you’re done.

/dev/mapper/archive_crypt /home/guyru/archive btrfs noauto,user 0 0

Gave up waiting for suspend/resume device

My boot process was pretty slow in a new setup I had. It would stop for about 30 seconds and then give the following error:

Gave up waiting for suspend/resume device

Turns out I had a resumable device listed in /etc/initramfs-tools/conf.d/resume even though my swap is both encrypted with random keys and too small. Editing that file and setting RESUME=none and running sudo update-initramfs -u fixed the issue.

Fixing Zoom’s Screen Sharing on Debian Unstable

Zoom has a native Linux client which supports screen sharing in Wayland, at least on some platforms. Today when I tried to start a Share Screen I encountered the following error:

Error when trying to start Share Screen

Can not start share, we only support Wayland on GNOME with Ubuntu 17 and above, Fedora 25 and above, Debian 9 and above, CentOS 8 and above, OpenSUSE Leap 15 and above, Oracle Linux 8 and above, Arch Linux, AnterGos, Manjaro. If your OS is not on the list, please use x11 instead.

The feature works for me when I’m using Debian Stable (Buster), and also worked for the short while I’ve used Debian Testing (Bullseye). So, I guessed that the feature is broken due to wrong OS version detection. The fix is easy: Remove /etc/os-release (which is by default a symlink to /usr/lib/os-release) and append to the original contents the following lines:

VERSION_ID="99"
VERSION="99 (sid)"
VERSION_CODENAME="sid"

So the entire file should look like:

PRETTY_NAME="Debian GNU/Linux bullseye/sid"
NAME="Debian GNU/Linux"
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
VERSION_ID="99"
VERSION="99 (sid)"
VERSION_CODENAME="sid"

Why it works?

When I first encountered the error, I guessed Zoom doesn’t actually attempt the Share Screen, but relies on a pre-configured list of supported distros and (minimal) versions. It worked for me with Debian Stable (10) and Testing (11), but what version number is Unstable? Debian Unstable doesn’t have a version number associated with it, so it must be the problem.

A quick strace revealed how Zoom retrieves the current distro name and version:

$ strace -f -tt zoom
...
107498 11:20:58.943764 openat(AT_FDCWD, "/etc/os-release", O_RDONLY|O_CLOEXEC
107513 11:20:58.943771 futex(0x565257f88760, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, {tv_sec=1592727659, tv_nsec=303765000}, FUTEX_BITSET_MATCH_ANY
107498 11:20:58.943779 <… openat resumed>) = 17
107498 11:20:58.943787 fcntl(17, F_SETFD, FD_CLOEXEC) = 0
107498 11:20:58.943802 fstat(17, {st_mode=S_IFREG|0644, st_size=200, …}) = 0
107498 11:20:58.943817 read(17, "PRETTY_NAME=\"Debian GNU/Linux bu"…, 200) = 200
...

As you can see Zoom reads (and probably later parses) the entire /etc/os-release file. This file contains identification data for the current running distro including name and version. Because Debian Sid doesn’t have the version variables set, Zoom erroneously misinterpret it as an old version instead of the newest. Thus, it refuses to enable the Share Screen feature. Adding the relevant version variables solves this issue.

See also: man os-release (5)

Prevent applications from increasing output volume in PulseAudio

By default, PulseAudio allows an application to change the max volume output to be louder than the one set by the user. I find it annoying that some apps tend to set volume to 100% which ends up increasing the system volume to unreasonable levels. You can prevent it by setting flat-volumes to no in ~/.config/pulse/daemon.conf.

$ echo "flat-volumes = no" >> ~/.config/pulse/daemon.conf 
$ pulseaudio --kill
$ pulseaudio --start

With flat-volumes = no PulseAudio uses relative volumes. Each applications sets its own volume relative to the system output.

References:

Fast bytes concatenation in Python

So what is the fastest way to concatenate bytes in Python? I decided to benchmark and compare few common patterns to see how they hold up. The scenario I tested is iterative concatenation of a block of 1024 bytes until we get 1MB of data. This is very similar to what one might do when reading a large file to memory, so this test is pretty realistic.

The first implementation is the naive one.

def f():
    ret = b''
    for i in range(2**10):
        ret += b'a' * 2**10
    return ret

It is known that the naive implementation is very slow, as bytes in Python are immutable type, hence we need to realloc the bytes and copy them after each concatenation. Just how slow is it? about 330 times slower than the append-and-join pattern. The append-and-join pattern was a popular (and efficient) way to concatenate strings in old Python versions

def g():
    ret = list()
    for i in range(2**10):
        ret.append(b'a' * 2**10)
    return b''.join(ret)

It relies on the fact that appending to lists is efficient and then ''.join can preallocate the entire needed memory and perform the copy efficiently. As you can see below it is much more efficient than the naive implementation.

Python 2.6 introduced the bytearray as an efficient mutable bytes sequence. Being mutable allows one to "naively" concatenate the bytearray and achieve great performance more than 30% faster than the join pattern above.

def h():
    ret = bytearray()
    for i in range(2**10):
        ret += b'a' * 2**10
Comparing the naive, join and bytearray implementation. Time is for 64 iterations.
Comparing the join, bytearray, preallocated bytearray and memoryview implementation. Time is for 8196 iterations.

What about perallocating the memory?

def j():
    ret = bytearray(2**20)
    for i in range(2**10):
        ret[i*2**10:(i+1)*2**10] = b'a' * 2**10
    return ret

While this sounds like a good idea, Pythons copy semantics turn out to be very slow. This resulted in 5 times slower run times. Python also offers memeoryview:

memoryview objects allow Python code to access the internal data of an object that supports the buffer protocol without copying.

The idea of access to the internal data without unnecessary copying sounds great.

def k():
    ret = memoryview(bytearray(2**20))
    for i in range(2**10):
        ret[i*2**10:(i+1)*2**10] = b'a' * 2**10
    return ret

And it does run almost twice as fast as preallocated bytearray implementation, but still about 2.5 times slower than the simple bytearray implementation.

I ran the benchmark using the timeit module, taking the best run out of five for each. CPU was Intel i7-8550U.

import timeit

for m in [f, g, h]:
    print(m, min(timeit.repeat(m, repeat=5, number=2**6)))

for m in [g, h, j, k]:
    print(m, min(timeit.repeat(m, repeat=5, number=2**13)))

Conclusion

The simple bytearray implementation was the fastest method, and also as simple as the naive implementation. Also preallocating doesn’t help, because python it looks like python can’t copy efficiently.

`xdg-open` fails when using Firefox under Wayland

Recently I noticed xdg-open started failing opening links in Firefox. Giving me the following error:

Firefox is already running, but is not responding. To open a new window, you must first close the existing Firefox process, or restart your system.

Firefox is already running, but is not responding. To open a new window, you must first close the existing Firefox process, or restart your system.

It happened while I had Firefox running and responding to everything else. I’m running the latest stable Firefox (74 as I’m writing this) on Wayland. Wayland brings a lot of good things, but also a lot of interoperability problems, so I suspected it had something to do with it. Thanks to Martin Stransky I found out that the solution is to set the MOZ_DBUS_REMOTE environment variable prior to launching Firefox. If you are using a desktop file to launch Firefox, you can set the variable in the Exec line like this:

[Desktop Entry]
Type=Application
Name=Firefox
Exec=env MOZ_DBUS_REMOTE=1 MOZ_ENABLE_WAYLAND=1 /home/guyru/.local/firefox/firefox %u
X-MultipleArgs=false
Icon=firefox-esr
Categories=Network;WebBrowser;
Terminal=false
MimeType=text/html;text/xml;application/xhtml+xml;application/xml;application/vnd.mozilla.xul+xml;application/rss+xml;application/rdf+xml;image/gif;image/jpeg;image/png;x-scheme-handler/http;x-scheme-handler/https;

You will need to restart Firefox before the fix will take affect.

TP-Link Archer T4Uv2 support for Debian Buster

This post outlines how I backported rtl8812au-dkms from Ubuntu Focal for Debian Buster and added support for the TP-Link Archer T4U v2 card to it. If you are only interested in the resulting .deb file skip to the end.

The TP-Link Archer T4U v2 is an AC1300 WiFi USB adapter. TP-Link provides drivers but they are built only for old kernel versions (<=3.19) and do not supoort DKMS, which makes upgrading a hassle.

Ubuntu provides the rtl8812au-dkms package which support the chipset in the Archer T4Uv2, but it doesn’t recognize the TP-Link product. So I set out to backport it to Debian Buster and make it support the Archer T4Uv2.

We start by fetching the rtl8812au source package from Ubuntu.

$ dget --allow-unauthenticated http://archive.ubuntu.com/ubuntu/pool/universe/r/rtl8812au/rtl8812au_4.3.8.12175.20140902+dfsg-0ubuntu12.dsc
$ cd rtl8812au-4.3.8.12175.20140902+dfsg/
$ sed -i s/dh-modaliases// debian/control
$ sed -i s/,modaliases// debian/rules
$ mk-build-deps ./debian/control --install --root-cmd sudo --remove

The sed lines remove reference to the dh-modaliases build dependency which Debian doesn’t have. I’m not really sure why they needed it for this package, but removing it didn’t hurt.

Next we add a new patch using quilt to support the Archer T4Uv2. We extract the 2357:010d USB vid:pid pair of the adapter using lsusb.

$ quilt push -a
$ quilt new add_archer_t4uv2.patch
$ quilt add os_dep/linux/usb_intf.c
$ vim os_dep/linux/usb_intf.c

The change we’ll be making to os_dep/linux/usb_intf.c is outlined by the following patch:

--- rtl8812au-4.3.8.12175.20140902+dfsg.orig/os_dep/linux/usb_intf.c
+++ rtl8812au-4.3.8.12175.20140902+dfsg/os_dep/linux/usb_intf.c
@@ -303,6 +303,7 @@ static struct usb_device_id rtw_usb_id_t
 	{USB_DEVICE(0x20f4, 0x805b),.driver_info = RTL8812}, /* TRENDnet - */
 	{USB_DEVICE(0x2357, 0x0101),.driver_info = RTL8812}, /* TP-Link - Archer T4U */
 	{USB_DEVICE(0x2357, 0x0103),.driver_info = RTL8812}, /* TP-Link - Archer T4UH */
+	{USB_DEVICE(0x2357, 0x010d),.driver_info = RTL8812}, /* TP-Link - Archer T4Uv2 */
 	{USB_DEVICE(0x0411, 0x025d),.driver_info = RTL8812}, /* Buffalo - WI-U3-866D */
 #endif

Finish up adding the patch:

$ quilt header --dep3 -e
$ quilt refresh
$ quilt pop -a

And build the package:

$ DEBEMAIL="Guy Rutenberg <guyrutenberg@gmail.com>" debchange --bpo
$ debuild -us -uc

Now we can install the newly created deb package:

$ cd ../
$ sudo apt install ./rtl8812au-dkms_4.3.8.12175.20140902+dfsg-0ubuntu12~bpo10+1_all.deb

If you came here only for the actual binary package you can find it in my deb repository: https://guyrutenberg.com/debian/buster/

Install JetBrains Mono in Debian/Ubuntu

JetBrains Mono is a new monospace typeface designed to be comfortable to read. It has clear distinction between the different letters and relatively high x-height.

The installation instruction were tested on Debian, but should work on every Linux.

Download and unzip the font:

$ wget https://download.jetbrains.com/fonts/JetBrainsMono-1.0.0.zip
$ unzip JetBrainsMono-1.0.0.zip

Install the font to either the user’s font directory

$ mv JetBrainsMono-*.ttf ~/.local/share/fonts/

or the system-wide one:

$ sudo mv JetBrainsMono-*.ttf /usr/share/fonts/

To use the font in gVim set guifont accordingly in ~/.vimrc:

:set guifont=JetBrains\ Mono\ 13