To what compression ratios do the different compression settings translate?

[TomiReiman]

What compression ratios do the different levels of meta.compression.level and packet.compression.level effectively translate to with the different packet.compression and meta.compression values?

I.e. if we start from the situation where both meta.compression and packet.compression are set to "none", and we consider the storage requirement at this starting point at 100%, what are the effectively values for each meta.compression.level and packet.compression.level values for the different compression methods (gzip 1-9, bzip2 1-9, and lzma 1-9)?

I am in a position where I would like to avoid adding more storage to our log hybrids and to avoid adding an archiver. We will most likely reach a retention of about 9 months and we would like to stretch that all the way up to 24 months.

[RichardB]

There are some good benchmarks at Quick Benchmark: Gzip vs Bzip2 vs LZMA vs XZ vs LZ4 vs LZO - CatchChallenger wiki  and https://linuxaria.com/article/linux-compressors-comparison-on-centos-6-5-x86-64-lzo-vs-lz4-vs-gzip-vs-bzip2-vs-lzma 

[KevinArunski]

There's no way to predict exactly what the compression ratios will be.  It depends entirely on the data provided on input, and it can vary wildly depending on what data is coming in at any given moment. 

As a very broad estimate you could say log db (packetdb) compresses 9 to 1, and metadb compresses 5 to 1.  However, your data will vary.  You have to experiment with it to see what your effective compression is.

We typically only use gzip compression level 5.  Going higher than five doesn't produce hugely better compression, and compression level 5 is not much faster than compression level 1. 

Bzip2 and LZMA compressors are much much slower than gzip so if you try those you have to be very careful that you don't introduce a bottleneck.

[RichardB]

I think you meant to say "compression level 1 is not much faster than compression level 5". We came to the same conclusion and are also using gzip level 5. It is a shame NW only supports gzip/bzip2/lzma which are all compression (size) optimized algorithms. Providing a speed optimized algorithm like lz4 as well could benefit CPU constrained appliances.