This issue is most often seen in RSA NetWitness Hybrid and All-In-One (AIO) appliances and Virtual Log Collectors (VLCs) due to the volume of entries that the nwlogcollector service writes to /var/log/messages. VLCs often have a smaller /var/log volume (e.g. 3.9G) than physical appliances (e.g. 9.8G).
In order to detect the problem, log into the affected host using SSH and run the following commands. The outputs in the examples below were taken from a VLC
We have identified the issue now, /var/log/messages is causing /var/log to fill up quickly.
An alternative way of doing this would be to use the 'ls' command and sorting file size to examine the files in /var/log directory (Hint: Could add the -R switch as well to recurse into subdirectories, however, the -S switch only sorts files within each directory):
# ls -AhlSr /var/logExample Output:
-rw-------. 1 root root 9.7M Dec 16 00:01 messages-20181019.gz
-rw-------. 1 root root 22M Dec 16 16:57 cron
-rw-------. 1 root root 50M Dec 16 16:57 secure
-rw-------. 1 root root 123M Dec 15 20:01 maillog-20181019.gz
-rw-------. 1 root root 2.7G Dec 16 16:52 messages
Note: If the utilisation of 'df -hP' and 'du -ahx' don't match then this is likely due to a failure of logrotate when writing to a new file. Run the following command to check for deleted but not released log files:
# lsof -X /var/log 2>/dev/null | grep -E "(^COMMAND|\(deleted\))"Example Output:
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
rsyslogd 3104 root 1w REG 253,5 2516186901 58 /var/log/messages-20190212 (deleted)
To release the space being taken by the deleted file (but held by rsyslogd as it still has an open file handle), you will either need to reboot the OS or restart the syslog services
# service rsyslog restart
The logrotate service's configuration need to be adjusted by editing /etc/logrotate.d/syslog to allow the normal rotation of /var/log/messages.
BEFORE: The current configuration of logrotate for syslog services in 10.6.x is as follows:
In this way we are going to rotate /var/log/messages on a weekly basis (retaining 4 compressed logs) or when the file reaches the size of 250 MB (whichever comes first). The dateext means that the date of rotate will be appended to the filename e.g. messages-20190212
Test that the configuration is correct by running logrotate manually using the following command:
# logrotate --force -vd /etc/logrotate.d/syslog
If you are unsure of any of the steps above or experience any issues, contact RSA Customer Support and reference this article for further assistance.
If after applying the above steps logrotate is not working, then the syslog service may need to be restarted as shown below.
# service rsyslog restart
Note: Other non-standard packages installed on the host such as syslog-ng may also cause logrotate to fail due to additional file handles on /var/log/messages. RSA Support would recommend that these non-standard packages be removed. You may be able to find these processes using the following command: