The messages file fills up the /var/log partition and prevents services from starting on an RSA Security Analytics

[AnujShrivastava]

In our Project a completely virtual Security Analatics environment is deployed,

/var/log partition in VLC get full each time and whole thing gets stuck, not sure why.

I am new to Security Analytics and followed following two article but var/log still getting full Please Help...:)

1. 000032301 - The messages file fills up the /var/log partition and prevents services from starting on an RSA Security Ana… 

2. 000030086 - The /var/log partition becomes full on an RSA Security Analytics Log Collector due to rabbitmq log files not… 

for workaround each time i need to delete old message files.

what to do ? so that /var/log partition rotate logs and partition doesn't get full 

[NaushadKasu1]

So the /var/log partition is getting filled, but can you find out which files are taking up the most space so you can look to that service for troubleshooting?  Run the following:

du -sh /var/log/*

That should give you a listing of what is taking up the space so you can target your searches / troubleshooting directly to that service.

[AnujShrivastava]

HI Naushad,

Thanks For your Reply, It is basically the /var/log/message file which fills up the /var/log partition. i want it to be compress or delete old file when my partition fills to 90 or 95%.

following is the message which is flooding and i am working on this also. 

Mar 25 23:25:31 so-rsa-vlc NwLogCollector[1932]: [SyslogCollection] [warning] [syslog-tcp.tcp514] [processing] [Receiver WorkUnit] [processing] Unidentified content from 10.200.50.11 received on receiver: 'Rj?Er??[????lN? ???????H?[???0?]?Y??t???Q??O=??G???{zz?gY5? V????????N??d.%_?S}i??<R?1? ?????~m?Z?'?' 

[NathanGetty]

First off. 

Stop 10.200.50.11 from sending logs. Stop all hosts from sending non-RFC compliant syslog. 

This will clear up those errors.

Second of all, run:

du -ch /var/log

This will show the disk space, this partition should not fill up.

You are most likely sending many, many non-compliant syslog messages.

[JesseLyon]

I had the same issue. There are two routes to take here. The first is to follow the RSA guide on rotating the logs to ensure they are compressed after they hit 300MB: 000032301 - The messages file fills up the /var/log partition and prevents services from starting on an RSA Security Ana… 

On some of my really high throughput VLCs, the log rotate method didn't work for me however. By high throughput I mean 7500EPS sustained. To remediate, I ssh'd to the host, vi /etc/rsyslog.conf, and commented out the sections under rules except save boot messages and news errors (see attached screenshot). 

After a restart of the rsyslog service my issues haven't returned. I understand that there could potentially be an issue where I was left without host logs to troubleshoot an issue, however I feel that is an acceptable risk. Also, be sure to update your spool directory to point to the correct mount point on your VLC in the event of an loss of connectivity between the VLC and the LD.

[DavidPoirier]

Hi Jesse,

We should investigate why we are building so may messages, are we logging any bad syslog content? Do you see any logs coming in as unidentified messages?

This would be something to look at as well.

David

[JesseLyon]

There is some bad syslog content, and we are working to fix as many hosts as we can as rapidly as possible. Most of the content flagged as bad is still being parsed correctly however, so from a resource perspective a resolution is pretty low on our priority list.

We also have a usecase where we are forwarding traffic from VLC's to a 3rd party system. Once forwarding is turned on, it looks like all logs the VLC recieves are written to the messages forwarder, so in that case, it is totally necessary to disable writing logs to disk other than to the spooler.