Speeding up with parallel compression – pbzip2

Premise

Today I found myself in need of archiving some virtual machines, which is quite often rather large. The actual machine I was working on was a 4 core, 8 with HT, Xeon powerhouse and I was curious to see if there was any way to speed up compression times for this particular task.

Looking into things

Usually I always grab for my trusty old friend tar when creating archives and it does get the job done well. The thing about tar though, is that it is inherently single-threaded, so it doesn’t really matter how many CPU cores you throw at it.

After digging around a bit I found pbzip2. Description:

pbzip2 is a parallel implementation of the bzip2 block-sorting file compressor that uses pthreads and achieves near-linear speedup on SMP machines.

Sounds good right? I decided to try i out and measure the results. The size of the virtual machine was about 11G:

~$ du -hs *
11G     WinXP_32Bit

With just plain old tar it took about eight minutes:

~$ time tar zcvf winxp.tar.gz WinXP_32Bit
 
real    8m18.583s
user    6m47.089s
sys     0m15.129s

Not bad, but that is nothing compared to piping it through pbzip2:

~$ time tar -c WinXP_32Bit | pbzip2 -c > winxp.tar.bz2
 
real    4m54.942s
user    38m22.452s
sys     0m25.022s

Screenshots of htop to show the difference in cpu core utilisation:

Both resulting archives were of equal size, so the immediate benefit is purely speed:

~$ du -hs *
11G     WinXP_32Bit
6.2G    winxp.tar.bz2
6.2G    winxp.tar.gz

For good measure, I also timed the decompression speeds. Though there was still a gain in speed, it was not quite as significant as with compression:

~$ time tar zxvf winxp.tar.gz
 
real    5m8.636s
user    1m20.061s
sys     0m20.413s
~$ time pbzip2 -d winxp.tar.bz2
<
real    4m32.329s
user    13m15.814s
sys     0m19.057s

Some things might be said for a lot of other limiting factors such as disk read/write speed etc. Playing around with different settings of pbzip2 might also reveal greater performance boosts than this simple example, but by standard, it is now a welcome addition to my *nix toolkit.