Does VLC compress logs "at rest"

[TomiReiman]

To my understanding a VLC can compress logs before they are forwarded to a log decoder. At what point do the logs get compressed? I am asking because I want to know how much storage capacity to allocate for a VLC's "buffer" - should I simply calculate it using the size of the raw events collected by the VLC? I.e. if I collect 50 GB per day, and I wish to allocate a buffer for 24 hours, should I allocate 150 GB (recommended by sadocs) plus 50 GB for the actual buffer?

[KhaledGamal]

Hi Tomi,

VLCs use rabbitmq to compress logs and send them through the rabbitmq queues to the local collector on the log decoder.If everything is working correctly, the logs will be sent to the Local collector and the buffer won't get full at all. The buffer is useful in cases where there is a problem between the VLC and Local collector so the logs are stored in buffer till the issue is fixed and then rabbitmq starts processing the logs in the buffer.

Note that rabbitmq will raise an alarm and will stop working when 80% of the disk space for the rabbitmq mount point is reached.

Hope this helps!

Best regards

Khaled

[TomiReiman]

Hi Khaled,

Thank you for the response. A hypothetical situation where the local collector on the log decoder would be down is exactly something I am planning for here. What is the mount point used by RabbitMQ? I.e. where should I allocate more storage when increasing the buffer?

[KhaledGamal · Accepted solution]

Hi Tomi,

You are always welcome

The default mount point is "/var/lib/rabbitmq". You could check with the logical volume as well in case you have a different mount point. The below is how it should look like when doing "df -h"

/dev/mapper/VolGroup00-rabmq                      245G   48M  244G   1% /var/lib/rabbitmq

The bigger the mount point, the bigger the buffer for storing logs is.

Best regards

Khaled