Fast bytes concatenation in Python

So what is the fastest way to concatenate bytes in Python? I decided to benchmark and compare few common patterns to see how they hold up. The scenario I tested is iterative concatenation of a block of 1024 bytes until we get 1MB of data. This is very similar to what one might do when reading a large file to memory, so this test is pretty realistic.

The first implementation is the naive one.

def f():
    ret = b''
    for i in range(2**10):
        ret += b'a' * 2**10
    return ret

It is known that the naive implementation is very slow, as bytes in Python are immutable type, hence we need to realloc the bytes and copy them after each concatenation. Just how slow is it? about 330 times slower than the append-and-join pattern. The append-and-join pattern was a popular (and efficient) way to concatenate strings in old Python versions

def g():
    ret = list()
    for i in range(2**10):
        ret.append(b'a' * 2**10)
    return b''.join(ret)

It relies on the fact that appending to lists is efficient and then ''.join can preallocate the entire needed memory and perform the copy efficiently. As you can see below it is much more efficient than the naive implementation.

Python 2.6 introduced the bytearray as an efficient mutable bytes sequence. Being mutable allows one to "naively" concatenate the bytearray and achieve great performance more than 30% faster than the join pattern above.

def h():
    ret = bytearray()
    for i in range(2**10):
        ret += b'a' * 2**10
Comparing the naive, join and bytearray implementation. Time is for 64 iterations.
Comparing the join, bytearray, preallocated bytearray and memoryview implementation. Time is for 8196 iterations.

What about perallocating the memory?

def j():
    ret = bytearray(2**20)
    for i in range(2**10):
        ret[i*2**10:(i+1)*2**10] = b'a' * 2**10
    return ret

While this sounds like a good idea, Pythons copy semantics turn out to be very slow. This resulted in 5 times slower run times. Python also offers memeoryview:

memoryview objects allow Python code to access the internal data of an object that supports the buffer protocol without copying.

The idea of access to the internal data without unnecessary copying sounds great.

def k():
    ret = memoryview(bytearray(2**20))
    for i in range(2**10):
        ret[i*2**10:(i+1)*2**10] = b'a' * 2**10
    return ret

And it does run almost twice as fast as preallocated bytearray implementation, but still about 2.5 times slower than the simple bytearray implementation.

I ran the benchmark using the timeit module, taking the best run out of five for each. CPU was Intel i7-8550U.

import timeit

for m in [f, g, h]:
    print(m, min(timeit.repeat(m, repeat=5, number=2**6)))

for m in [g, h, j, k]:
    print(m, min(timeit.repeat(m, repeat=5, number=2**13)))

Conclusion

The simple bytearray implementation was the fastest method, and also as simple as the naive implementation. Also preallocating doesn’t help, because python it looks like python can’t copy efficiently.

`xdg-open` fails when using Firefox under Wayland

Recently I noticed xdg-open started failing opening links in Firefox. Giving me the following error:

Firefox is already running, but is not responding. To open a new window, you must first close the existing Firefox process, or restart your system.

Firefox is already running, but is not responding. To open a new window, you must first close the existing Firefox process, or restart your system.

It happened while I had Firefox running and responding to everything else. I’m running the latest stable Firefox (74 as I’m writing this) on Wayland. Wayland brings a lot of good things, but also a lot of interoperability problems, so I suspected it had something to do with it. Thanks to Martin Stransky I found out that the solution is to set the MOZ_DBUS_REMOTE environment variable prior to launching Firefox. If you are using a desktop file to launch Firefox, you can set the variable in the Exec line like this:

[Desktop Entry]
Type=Application
Name=Firefox
Exec=env MOZ_DBUS_REMOTE=1 MOZ_ENABLE_WAYLAND=1 /home/guyru/.local/firefox/firefox %u
X-MultipleArgs=false
Icon=firefox-esr
Categories=Network;WebBrowser;
Terminal=false
MimeType=text/html;text/xml;application/xhtml+xml;application/xml;application/vnd.mozilla.xul+xml;application/rss+xml;application/rdf+xml;image/gif;image/jpeg;image/png;x-scheme-handler/http;x-scheme-handler/https;

You will need to restart Firefox before the fix will take affect.

TP-Link Archer T4Uv2 support for Debian Buster

This post outlines how I backported rtl8812au-dkms from Ubuntu Focal for Debian Buster and added support for the TP-Link Archer T4U v2 card to it. If you are only interested in the resulting .deb file skip to the end.

The TP-Link Archer T4U v2 is an AC1300 WiFi USB adapter. TP-Link provides drivers but they are built only for old kernel versions (<=3.19) and do not supoort DKMS, which makes upgrading a hassle.

Ubuntu provides the rtl8812au-dkms package which support the chipset in the Archer T4Uv2, but it doesn’t recognize the TP-Link product. So I set out to backport it to Debian Buster and make it support the Archer T4Uv2.

We start by fetching the rtl8812au source package from Ubuntu.

$ dget --allow-unauthenticated http://archive.ubuntu.com/ubuntu/pool/universe/r/rtl8812au/rtl8812au_4.3.8.12175.20140902+dfsg-0ubuntu12.dsc
$ cd rtl8812au-4.3.8.12175.20140902+dfsg/
$ sed -i s/dh-modaliases// debian/control
$ sed -i s/,modaliases// debian/rules
$ mk-build-deps ./debian/control --install --root-cmd sudo --remove

The sed lines remove reference to the dh-modaliases build dependency which Debian doesn’t have. I’m not really sure why they needed it for this package, but removing it didn’t hurt.

Next we add a new patch using quilt to support the Archer T4Uv2. We extract the 2357:010d USB vid:pid pair of the adapter using lsusb.

$ quilt push -a
$ quilt new add_archer_t4uv2.patch
$ quilt add os_dep/linux/usb_intf.c
$ vim os_dep/linux/usb_intf.c

The change we’ll be making to os_dep/linux/usb_intf.c is outlined by the following patch:

--- rtl8812au-4.3.8.12175.20140902+dfsg.orig/os_dep/linux/usb_intf.c
+++ rtl8812au-4.3.8.12175.20140902+dfsg/os_dep/linux/usb_intf.c
@@ -303,6 +303,7 @@ static struct usb_device_id rtw_usb_id_t
 	{USB_DEVICE(0x20f4, 0x805b),.driver_info = RTL8812}, /* TRENDnet - */
 	{USB_DEVICE(0x2357, 0x0101),.driver_info = RTL8812}, /* TP-Link - Archer T4U */
 	{USB_DEVICE(0x2357, 0x0103),.driver_info = RTL8812}, /* TP-Link - Archer T4UH */
+	{USB_DEVICE(0x2357, 0x010d),.driver_info = RTL8812}, /* TP-Link - Archer T4Uv2 */
 	{USB_DEVICE(0x0411, 0x025d),.driver_info = RTL8812}, /* Buffalo - WI-U3-866D */
 #endif

Finish up adding the patch:

$ quilt header --dep3 -e
$ quilt refresh
$ quilt pop -a

And build the package:

$ DEBEMAIL="Guy Rutenberg <guyrutenberg@gmail.com>" debchange --bpo
$ debuild -us -uc

Now we can install the newly created deb package:

$ cd ../
$ sudo apt install ./rtl8812au-dkms_4.3.8.12175.20140902+dfsg-0ubuntu12~bpo10+1_all.deb

If you came here only for the actual binary package you can find it in my deb repository: https://guyrutenberg.com/debian/buster/

JetBrains Mono

Install JetBrains Mono in Debian/Ubuntu

JetBrains Mono is a new monospace typeface designed to be comfortable to read. It has clear distinction between the different letters and relatively high x-height.

The installation instruction were tested on Debian, but should work on every Linux.

Download and unzip the font:

$ wget https://download.jetbrains.com/fonts/JetBrainsMono-1.0.0.zip
$ unzip JetBrainsMono-1.0.0.zip

Install the font to either the user’s font directory

$ mv JetBrainsMono-*.ttf ~/.local/share/fonts/

or the system-wide one:

$ sudo mv JetBrainsMono-*.ttf /usr/share/fonts/

To use the font in gVim set guifont accordingly in ~/.vimrc:

:set guifont=JetBrains\ Mono\ 13

Compiling lensfun-0.3.95 on Debian Buster

Lensfun provides lens distoration correction for Darktable and other raw processing applications. Version 0.3.95 provides ability to use the Adobe Camera Model, and hence use Adobe lens profiles (lcp files). However, lensfun 0.3.95 is not packaged for Debian Buster. Also Darktable won’t compile against the latest git version of Lensfun, so you must compile and install specifically version 0.3.95 to get ACM support.

We begin by downloading and extracting Lensfun 0.3.95. Lensfun 0.3.95 is not tagged in git, so we have to download the release directly from SourceForge. The release is not available from the GitHub repository.

$ wget https://sourceforge.net/projects/lensfun/files/0.3.95/lensfun-0.3.95.tar.gz
$ tar -xvf lensfun-0.3.95.tar.gz
$ cd lensfun-0.3.95/

Lensfun uses CMake for building and has also has CPack enabled. We can use it to build a deb package and install it. This allows easier integration and uninstallation in the future.

$ mkdir build
$ cd build
$ cmake -DCMAKE_BUILD_TYPE=release -DCPACK_BINARY_DEB=ON ../
$ make -j`nproc` && make package
$ sudo apt install ./liblensfun2_0.3.95.0_amd64.deb

libGL error: unable to load driver: radeonsi_dri.so

After playing with installing and removing the amdgpu and amdgpu-pro drivers, my system could not load the radeonsi. glxinfo returned the following error:

$ DRI_PRIME=1 glxinfo | grep OpenGL
libGL error: unable to load driver: radeonsi_dri.so
libGL error: driver pointer missing
libGL error: failed to load driver: radeonsi

It was solved by cleaning up leftover symlinks and reinstalling libgl-mesa-dri

$ sudo find /usr/lib/ -lname "/opt/amdgpu/*" -print -delete
/usr/lib/i386-linux-gnu/dri/kms_swrast_dri.so
/usr/lib/i386-linux-gnu/dri/r600_dri.so
/usr/lib/i386-linux-gnu/dri/r200_dri.so
/usr/lib/i386-linux-gnu/dri/vmwgfx_dri.so
/usr/lib/i386-linux-gnu/dri/swrast_dri.so
/usr/lib/i386-linux-gnu/dri/radeonsi_dri.so
/usr/lib/i386-linux-gnu/dri/r300_dri.so
/usr/lib/x86_64-linux-gnu/dri/kms_swrast_dri.so
/usr/lib/x86_64-linux-gnu/dri/r600_dri.so
/usr/lib/x86_64-linux-gnu/dri/r200_dri.so
/usr/lib/x86_64-linux-gnu/dri/vmwgfx_dri.so
/usr/lib/x86_64-linux-gnu/dri/swrast_dri.so
/usr/lib/x86_64-linux-gnu/dri/radeonsi_dri.so
/usr/lib/x86_64-linux-gnu/dri/r300_dri.so

$ sudo apt install --reinstall libgl1-mesa-dri

Enable periodic trim support

By default, trimming is disabled in Debian Buster. If you have an SSD, you can enable periodic trimming:

$ sudo systemctl enable fstrim.timer

The alternative is continuous trimming, which is enabled by passing discard option to the filesystem in /etc/fstab/.

Yubikey doesn’t work on Firefox installed via Snap

Installing Firefox via Snap is an easy way to get the latest Firefox version on your favorite distro, regardless of the version the distro ships with. However, due to Snap’s security model, Yubikeys, or any other FIDO tokens do not work out of the box. To enable U2F devices, like Yubikeys, you need to give the Firefox package the necessary permissions manually:

$ snap connect firefox:u2f-devices

Generating secure passphrases on the command line

The following snippet should work on every system that has coreutils.

$ shuf /usr/share/dict/words --repeat --random-source /dev/random -n 5

You can swap /usr/share/dict/words with any good wordlist, like EFF’s or Arnold Reinhold’s Diceware list.

You can also add it for ease of use to your ~/.bash_aliases

alias passphrase="shuf ~/dotfiles/misc/diceware8k.txt --repeat --random-source /dev/random -n"

And then you could easily use it in bash:

$ passphrase 5
notch
kane
to
drag
cater