The /var/log/messages log file grows to occupy all of the available space in the /var/log partition preventing services such as the nwlogcollector (and other services) from starting.
For the failure of logrotate for other logs refer to the following KBs:
#000030086 RabbitMQ in NetWitness 10.4.0.2 - The /var/log partition becomes full on an RSA Security Analytics Log Collector due to rabbitmq log files not rotating
#000037185 logstash in NetWitness 11.x - RSA NetWitness 11.x /var/log mount is full due to logstash directory
# df -hP
Example Output:
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 9.9G 827M 8.6G 9% /
tmpfs 7.8G 0 7.8G 0% /dev/shm
/dev/mapper/VolGroup00-usr 3.9G 1.3G 2.4G 36% /usr
/dev/mapper/VolGroup00-usrhome 2.0G 3.1M 1.9G 1% /home
/dev/mapper/VolGroup00-var 3.9G 278M 3.4G 8% /var
/dev/mapper/VolGroup00-log 3.9G 3.9G 0 100% /var/log
/dev/mapper/VolGroup00-tmp 5.8G 12M 5.5G 1% /tmp
/dev/mapper/VolGroup00-vartmp 2.0G 3.0M 1.9G 1% /var/tmp
/dev/mapper/VolGroup00-opt 3.9G 468M 3.2G 13% /opt
/dev/mapper/VolGroup00-rabmq 10G 38M 10G 1% /var/lib/rabbitmq
/dev/mapper/VolGroup00-nwhome 12G 858M 12G 7% /var/netwitness
/dev/mapper/VolGroup01-logcoll 104G 1.4G 103G 2% /var/netwitness/logcollector
As can be seen in the hi-lighted section in red above, /var/log volume has reached 100% utilization.# du -ahx /var/log | sort -h | tail Example Output: 96M /var/log/rabbitmq 123M /var/log/maillog-20181019.gz 169M /var/log/netwitness/logcollector/NwServerLog-000000055.log 251M /var/log/netwitness/logcollector/NwServerLog-000000052.log 251M /var/log/netwitness/logcollector/NwServerLog-000000053.log 251M /var/log/netwitness/logcollector/NwServerLog-000000054.log 940M /var/log/netwitness/logcollector 941M /var/log/netwitness 2.7G /var/log/messages 3.9G /var/logWe have identified the issue now, /var/log/messages is causing /var/log to fill up quickly.
# ls -AhlSr /var/log Example Output: -rw-------. 1 root root 9.7M Dec 16 00:01 messages-20181019.gz -rw-------. 1 root root 22M Dec 16 16:57 cron -rw-------. 1 root root 50M Dec 16 16:57 secure -rw-------. 1 root root 123M Dec 15 20:01 maillog-20181019.gz -rw-------. 1 root root 2.7G Dec 16 16:52 messages
# lsof -X /var/log 2>/dev/null | grep -E "(^COMMAND|\(deleted\))" Example Output: COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME rsyslogd 3104 root 1w REG 253,5 2516186901 58 /var/log/messages-20190212 (deleted)To release the space being taken by the deleted file (but held by rsyslogd as it still has an open file handle), you will either need to reboot the OS or restart the syslog services
# service rsyslog restart
The logrotate service's configuration need to be adjusted by editing /etc/logrotate.d/syslog to allow the normal rotation of /var/log/messages.
# cat /etc/logrotate.d/syslog /var/log/cron /var/log/maillog /var/log/messages /var/log/secure /var/log/spooler { sharedscripts postrotate /bin/kill -HUP `cat /var/run/syslogd.pid 2> /dev/null` 2> /dev/null || true endscript }
/var/log/cron /var/log/maillog /var/log/messages /var/log/secure /var/log/spooler { weekly rotate 4 maxsize 250M dateext notifempty sharedscripts postrotate /bin/kill -HUP `cat /var/run/syslogd.pid 2> /dev/null` 2> /dev/null || true endscript }
# logrotate --force -vd /etc/logrotate.d/syslog
# service rsyslog restart
# lsof +D /var/log | grep messages