Quickly generate lots of random data

What is the quickest way to generate lots of random data on the command line? Usually when I had to wipe hard-drives I would simply use dd to copy from /dev/urandom over the device. However, `/dev/urandom is quite slow and wiping hard-disks can take a long time that way. So, I decided to benchmark a few methods to generate long random streams that are usable in such scenarios.

The benchmark is based on the dd command. For example:

$ dd if=/dev/urandom of=/dev/null bs=4k count=1M

This command will copy a 4GB of random bytes from /dev/urandom over /dev/null. This is probably the simplest method to create a large stream of random bytes, and as it turns out, also the slowest.

The second construct I tried is to use OpenSSL to create a stream of random data which I can read with dd and then write to the target. For example the following would use AES-128 with a random key:

$ openssl rand -hex 32 | openssl enc -aes-128-ctr -in /dev/zero -pass stdin -nosalt | dd if=/dev/stdin of=/dev/null bs=4k count=1M

Let’s breakup this command: openssl rand -hex 32 will generate a random encryption key to be used by the AES encryption. openssl enc -aes-128-ctr -in /dev/zero -pass stdin -nosalt does the actual encryption. It reads the (random) key from stdin and then uses it to encrypt /dev/zero using AES-128 in counter mode. As /dev/zero in an endless stream of zeros, it will simply output an endless stream of (pseudo-)random data. We can also repeat the same command only swapping aes-128-ctr with aes-256-ctr. For most (all?) usage scenarios it doesn’t provided any added security benefits but does have a (small) performance penalty.

Apart from AES, which is a block cipher, we can also try to use actual stream ciphers like the old rc4 and the modern chacha20.

Additionally, many new CPUs come with AES-NI extension which speeds up AES operations considerably. We can repeat the benchmark while disabling AES-NI to see how the different methods will perform if used a CPU that doesn’t support AES-NI.

Finally, I’ve repeated the test with /dev/zero as input, just to have an upper-limit in terms of performance to compare against.

Benchmark results
AES-NINo AES-NI
/dev/zero0.42609
/dev/urandom18.8967
chacha202.693063.79217
aes-128-ctr2.0410614.9022
aes-256-ctr2.2475618.9014
rc47.77392
Benchmark results, time (in seconds) to create 4GB of random data

Conclusions

The results clearly show that you should avoid /dev/urandom. It’s simply not suitable for this task and doesn’t perform well. The various methods of using OpenSSL perform much better. The best performance is achieved by the two AES variants, with aes-128-ctr being the fastest. However, if AES-NI is not supported by the CPU, AES takes a huge performance hit, and is even slower than the (not-so-)good and old RC4. However, ChaCha20 (a modern stream cipher) performs within 30% of AES if AES-NI is available, but if AES-NI is not supported ChaCha20 outperforms the AES variants. So, unless you know AES-NI is supported ChaCha20 is the safe choice.

Short cryptsetup/LUKS tutorial

This short tutorial will guide you in encrypting a drive with cryptsetup and LUKS scheme.

Before starting, if the device had previous data on it, it’s best to delete any filesystem signatures that may be on it. Assuming that the drive we operate is /dev/sda you can use the following command to remove the signatures:

$ sudo wipefs --all /dev/sda --no-act

Remove the --no-act flag to actually modify the disk.

The next step is to actually format the drive using LUKS. This is done using the cryptsetup utility.

$ sudo cryptsetup luksFormat --type=luks2 /dev/sda

WARNING!
========
This will overwrite data on /dev/sda irrevocably.

Are you sure? (Type 'yes' in capital letters): YES
Enter passphrase for /dev/sda: 
Verify passphrase: 

The command will prompt you to enter a passphrase for the encryption and should take a few seconds to complete.

The next step is to add an appropriate entry to crypttab which will simplify starting the dm-crypt mapping later. Add the following line to /etc/crypttab:

archive_crypt UUID=114d42e5-6aeb-4af0-8758-b4cc79dd1ba0 none luks,discard,noauto

where the UUID is obtained through lsblk /dev/sda -o UUID or a similar command. The archive_crypt is the name for the mapped device. It will appear as /dev/mapper/archive_crypt when the device is mapped. The none parameter specifies that no keyfile is used and the system should prompt for an encryption passphrase instead. The noauto, means not to attempt to load the device automatically upon boot. discard should be used if the underlying device is an SSD.

You can test everything works so far by opening and loading the LUKS device:

$ sudo cryptdisks_start archive_crypt

While the device is now encrypted, there is a possible leakage of metadata such as used blocks as an attacker can discern used vs unused blocks by examining the physical drive. This and other side-channel leaks can be mitigated by simply wiping the contents of the encrypted device.

$ openssl rand -hex 32 | openssl enc -chacha20 -in /dev/zero -pass stdin -nosalt | sudo dd if=/dev/stdin of=/dev/mapper/sda_crypt bs=4096 status=progress

We could also have used /dev/urandom but the above technique is much faster.

Now we can create the actual filesystem.

$ sudo mkfs.btrfs --label archive /dev/mapper/archive_crypt

At this point we’re actually pretty much done. You can add and entry to /etc/fstab to easily mount the filesystem and you’re done.

/dev/mapper/archive_crypt /home/guyru/archive btrfs noauto,user 0 0

Gave up waiting for suspend/resume device

My boot process was pretty slow in a new setup I had. It would stop for about 30 seconds and then give the following error:

Gave up waiting for suspend/resume device

Turns out I had a resumable device listed in /etc/initramfs-tools/conf.d/resume even though my swap is both encrypted with random keys and too small. Editing that file and setting RESUME=none and running sudo update-initramfs -u fixed the issue.

TP-Link Archer T4Uv2 support for Debian Buster

This post outlines how I backported rtl8812au-dkms from Ubuntu Focal for Debian Buster and added support for the TP-Link Archer T4U v2 card to it. If you are only interested in the resulting .deb file skip to the end.

The TP-Link Archer T4U v2 is an AC1300 WiFi USB adapter. TP-Link provides drivers but they are built only for old kernel versions (<=3.19) and do not supoort DKMS, which makes upgrading a hassle.

Ubuntu provides the rtl8812au-dkms package which support the chipset in the Archer T4Uv2, but it doesn’t recognize the TP-Link product. So I set out to backport it to Debian Buster and make it support the Archer T4Uv2.

We start by fetching the rtl8812au source package from Ubuntu.

$ dget --allow-unauthenticated http://archive.ubuntu.com/ubuntu/pool/universe/r/rtl8812au/rtl8812au_4.3.8.12175.20140902+dfsg-0ubuntu12.dsc
$ cd rtl8812au-4.3.8.12175.20140902+dfsg/
$ sed -i s/dh-modaliases// debian/control
$ sed -i s/,modaliases// debian/rules
$ mk-build-deps ./debian/control --install --root-cmd sudo --remove

The sed lines remove reference to the dh-modaliases build dependency which Debian doesn’t have. I’m not really sure why they needed it for this package, but removing it didn’t hurt.

Next we add a new patch using quilt to support the Archer T4Uv2. We extract the 2357:010d USB vid:pid pair of the adapter using lsusb.

$ quilt push -a
$ quilt new add_archer_t4uv2.patch
$ quilt add os_dep/linux/usb_intf.c
$ vim os_dep/linux/usb_intf.c

The change we’ll be making to os_dep/linux/usb_intf.c is outlined by the following patch:

--- rtl8812au-4.3.8.12175.20140902+dfsg.orig/os_dep/linux/usb_intf.c
+++ rtl8812au-4.3.8.12175.20140902+dfsg/os_dep/linux/usb_intf.c
@@ -303,6 +303,7 @@ static struct usb_device_id rtw_usb_id_t
 	{USB_DEVICE(0x20f4, 0x805b),.driver_info = RTL8812}, /* TRENDnet - */
 	{USB_DEVICE(0x2357, 0x0101),.driver_info = RTL8812}, /* TP-Link - Archer T4U */
 	{USB_DEVICE(0x2357, 0x0103),.driver_info = RTL8812}, /* TP-Link - Archer T4UH */
+	{USB_DEVICE(0x2357, 0x010d),.driver_info = RTL8812}, /* TP-Link - Archer T4Uv2 */
 	{USB_DEVICE(0x0411, 0x025d),.driver_info = RTL8812}, /* Buffalo - WI-U3-866D */
 #endif

Finish up adding the patch:

$ quilt header --dep3 -e
$ quilt refresh
$ quilt pop -a

And build the package:

$ DEBEMAIL="Guy Rutenberg <guyrutenberg@gmail.com>" debchange --bpo
$ debuild -us -uc

Now we can install the newly created deb package:

$ cd ../
$ sudo apt install ./rtl8812au-dkms_4.3.8.12175.20140902+dfsg-0ubuntu12~bpo10+1_all.deb

If you came here only for the actual binary package you can find it in my deb repository: https://guyrutenberg.com/debian/buster/

Fixing HTML Rendering in Wine on Debian Jessie

Some application rely on Internet Explorer to provide HTML rendering capabilities. Wine implements the same functionality based on a custom version of Mozilla’s Gecko rendering engine (the same engine used in Firefox). In Debian Jessie you have a package called libwine-gecko-2.24 (the version is part of the name) which provides this rendering engine for Wine. However, different versions of Wine require different versions of wine-gecko. The package provided in Debian Jessie, matches the Wine version provided by wine-development from the main Jessie repository (1.7.29). Unfortunately wine-development from the jessie-backports if of version 1.9.8 and requires wine-gecko of version 2.44 which is not provided by any Debian repository. This will lead to errors like

Could not load wine-gecko. HTML rendering will be disabled.

and blank spaces where HTML content would be rendered in many applications.

The solution would be to manually install the required version of wine-gecko. We start by downloading the MSI binaries provided by Wine

$ wget https://dl.winehq.org/wine/wine-gecko/2.44/wine_gecko-2.44-x86.msi
$ wget https://dl.winehq.org/wine/wine-gecko/2.44/wine_gecko-2.44-x86_64.msi

Now install the required one, based on whether you are using 32bit or 64bit wine environment:

wine-development msiexec /i wine_gecko-2.44-x86.msi

(be sure the setup the correct $WINEPREFIX if needed).

en_IL: English locale for Israel

Update: The new locale was committed to glibc and should be part of glibc-2.24.

Most Israelis are literate in English, and for a large percentage of them, English is also the preferred language when it comes to computers. They prefer English, as it solves right-to-left issues and general inconsistencies (it might be annoying when some programs are translated ands some not). The downside is, that currently, the existing English locales are not suitable for Israel, as there are cultural differences:

  • American English spelling is more common in Israel.
  • The metric system is used, along with the relevant paper sizes (“A4” instead of Letter).
  • Dates are written in dd/mm/YYYY format, unlike in the USA.
  • The first day of week, and also the first workday is Sunday.
  • The currency used is ILS (₪).

So, up until now users had to choose locales such as en_US or en_GB and compromise on some stuff. To solve this issue, and create a truly suitable English locale for Israel, I wrote a localedef file for the en_IL locale.

To install the new locale, copy the en_IL file from the gist below and place under /usr/share/i18n/locales/en_IL (no extension). Next

# echo "en_IL.UTF-8 UTF-8" >> /usr/local/share/i18n/SUPPORTED

Now, complete the installation by running dpkg-reconfigure locales and enable en_IL.UTF-8 from the list, and set it as the default locale.


comment_char %
escape_char /
% This file is part of the GNU C Library and contains locale data.
% The Free Software Foundation does not claim any copyright interest
% in the locale data contained in this file. The foregoing does not
% affect the license of the GNU C Library as a whole. It does not
% exempt you from the conditions of the license if your use would
% otherwise be governed by that license.
% Locale for English locale in Israel
% Contributed by Guy Rutenberg <guyrutenberg@gmail.com>, 2016
LC_IDENTIFICATION
title "English locale for Israel"
source "Free Software Foundation, Inc."
address "http:////www.gnu.org//software//libc//"
contact ""
email "bug-glibc-locales@gnu.org"
tel ""
fax ""
language "English"
territory "Israel"
revision "1.1"
date "2016-04-19"
%
category "i18n:2012";LC_IDENTIFICATION
category "i18n:2012";LC_CTYPE
category "i18n:2012";LC_COLLATE
category "i18n:2012";LC_TIME
category "i18n:2012";LC_NUMERIC
category "i18n:2012";LC_MONETARY
category "i18n:2012";LC_MESSAGES
category "i18n:2012";LC_PAPER
category "i18n:2012";LC_NAME
category "i18n:2012";LC_ADDRESS
category "i18n:2012";LC_TELEPHONE
category "i18n:2012";LC_MEASUREMENT
END LC_IDENTIFICATION
LC_CTYPE
copy "en_GB"
END LC_CTYPE
LC_COLLATE
copy "en_GB"
END LC_COLLATE
LC_MONETARY
copy "he_IL"
END LC_MONETARY
LC_NUMERIC
copy "he_IL"
END LC_NUMERIC
LC_TIME
abday "<U0053><U0075><U006E>";"<U004D><U006F><U006E>";/
"<U0054><U0075><U0065>";"<U0057><U0065><U0064>";/
"<U0054><U0068><U0075>";"<U0046><U0072><U0069>";/
"<U0053><U0061><U0074>"
day "<U0053><U0075><U006E><U0064><U0061><U0079>";/
"<U004D><U006F><U006E><U0064><U0061><U0079>";/
"<U0054><U0075><U0065><U0073><U0064><U0061><U0079>";/
"<U0057><U0065><U0064><U006E><U0065><U0073><U0064><U0061><U0079>";/
"<U0054><U0068><U0075><U0072><U0073><U0064><U0061><U0079>";/
"<U0046><U0072><U0069><U0064><U0061><U0079>";/
"<U0053><U0061><U0074><U0075><U0072><U0064><U0061><U0079>"
abmon "<U004A><U0061><U006E>";"<U0046><U0065><U0062>";/
"<U004D><U0061><U0072>";"<U0041><U0070><U0072>";/
"<U004D><U0061><U0079>";"<U004A><U0075><U006E>";/
"<U004A><U0075><U006C>";"<U0041><U0075><U0067>";/
"<U0053><U0065><U0070>";"<U004F><U0063><U0074>";/
"<U004E><U006F><U0076>";"<U0044><U0065><U0063>"
mon "<U004A><U0061><U006E><U0075><U0061><U0072><U0079>";/
"<U0046><U0065><U0062><U0072><U0075><U0061><U0072><U0079>";/
"<U004D><U0061><U0072><U0063><U0068>";/
"<U0041><U0070><U0072><U0069><U006C>";/
"<U004D><U0061><U0079>";/
"<U004A><U0075><U006E><U0065>";/
"<U004A><U0075><U006C><U0079>";/
"<U0041><U0075><U0067><U0075><U0073><U0074>";/
"<U0053><U0065><U0070><U0074><U0065><U006D><U0062><U0065><U0072>";/
"<U004F><U0063><U0074><U006F><U0062><U0065><U0072>";/
"<U004E><U006F><U0076><U0065><U006D><U0062><U0065><U0072>";/
"<U0044><U0065><U0063><U0065><U006D><U0062><U0065><U0072>"
d_t_fmt "<U0025><U0061><U0020><U0025><U0064><U0020><U0025><U0062><U0020><U0025><U0059><U0020><U0025><U0054><U0020><U0025><U005A>"
d_fmt "<U0025><U0064><U002F><U0025><U006D><U002F><U0025><U0079>"
t_fmt "<U0025><U0054>"
am_pm "";"" % 24-hour clock is used
t_fmt_ampm ""
date_fmt "<U0025><U0061><U0020><U0025><U0065><U0020><U0025><U0062>/
<U0020><U0025><U0048><U003A><U0025><U004D><U003A><U0025><U0053><U0020>/
<U0025><U005A><U0020><U0025><U0059>"
week 7;19971130;1
% In Israel Sunday is a workday.
first_workday 1
END LC_TIME
LC_MESSAGES
copy "en_US"
END LC_MESSAGES
LC_PAPER
copy "he_IL"
END LC_PAPER
LC_NAME
copy "en_US"
END LC_NAME
LC_ADDRESS
postal_fmt "<U0025><U0066><U0025><U004E><U0025><U0061><U0025><U004E>/
<U0025><U0064><U0025><U004E><U0025><U0062><U0025><U004E><U0025><U0073>/
<U0020><U0025><U0068><U0020><U0025><U0065><U0020><U0025><U0072><U0025>/
<U004E><U0025><U007A><U0020><U0025><U0054><U0025>/
<U004E><U0025><U0063><U0025><U004E>"
country_name "<U0049><U0073><U0072><U0061><U0065><U006C>"
country_post "<U0049><U0053><U0052>"
country_ab2 "<U0049><U004C>"
country_ab3 "<U0049><U0053><U0052>"
country_num 376
country_car "<U0049><U004C>"
lang_name "<U0045><U006E><U0067><U006C><U0069><U0073><U0068>"
lang_ab "<U0065><U006E>"
lang_term "<U0065><U006E><U0067>"
lang_lib "<U0065><U006E><U0067>"
END LC_ADDRESS
LC_TELEPHONE
copy "he_IL"
END LC_TELEPHONE
LC_MEASUREMENT
copy "he_IL"
END LC_MEASUREMENT

view raw

en_IL

hosted with ❤ by GitHub

Installing Debian Unstable’s source Packages in Debian Jessie

Sometimes a package that you need is not available for Debian Jessie, but you can find it for Sid (unstable). You may be tempted to try to install it manually, by downloading the binary deb package, but it will most likely fail due to binary incompatibilities with different libraries’ versions you have. The better method will be to get the source package used to build the binary package, and build it yourself. Most of the time the process is not as hard as it sounds.

First, a short preliminary setup is needed Add the following lines to /etc/apt/sources.list:

deb http://http.debian.net/debian jessie-backports main
deb-src http://httpredir.debian.org/debian unstable main contrib

You can replace unstable with testing if you prefer to use packages from testing. Update the lists of packages

sudo apt-get update

Next you need, to get the build dependencies for your package. The example below uses the package lyx:

sudo apt-get build-dep lyx/unstable

Now you are ready to fetch and build the source package:

sudo apt-get source -b lyx/unstable

Finally, you will see in the current directory the resulting DEBs. Simply install them:

sudo gdebi lyx-common_2.1.4-2_all.deb
sudo gdebi lyx_2.1.4-2_amd64.deb

You can later mark the dependecies that you manually installed as automatic:

sudo apt-mark auto lyx-common