compressing large files is a time-consuming task and when it turns out the compressed file size is not much smaller than the original, the question whether it's worth the effort or not can be difficult to answer, especially with nowadays ever-growing storage availability

lrzip to the rescue! we no longer need to trade time for space. oh well, not as much as we used to :)

the idea is to find long-distance redundancy first and only afterwords compressing with a traditional tool, increasing overall speed and compression ratio. another brilliant idea by Con Kolivas!

I ran some tests, comparing time and size of the same file (a 1.8GB VirtualBox snapshot) compressed by gzip, bzip2, lzma and lrzip. every tool has been tested with default values and with maximum compression settings. lrzip uses lzma by default, but can optionally be used with LZO/gzip/bzip2/ZPAQ as second stage compression. it can even be told to just do stage one, which yields interesting results on its own! unfortunately ZPAQ compression (albeit running for an hour before failing) didn't complete, so I can't represent this super-slow method side-by-side with the other ones

below is a brief report about my findings, enjoy!