Article Number
000001681
Applies To
RSA Product Set: RSA NetWitness Platform, NetWitness Logs & Network
RSA Product/Service Type: NetWitness Server/Admin Server
RSA Version/Condition: 11.x
Platform: CentOS
O/S Version: 7
Issue
After upgrading from NetWitness 10.x to NetWitness 11.x the /var/log mount is filling up, eventually becoming full.
Image description
# du -hx --max-depth=2 /var/log | sort -h | tail
11M /var/log/netwitness/investigate-server
11M /var/log/netwitness/orchestration-server
12M /var/log/nginx
36M /var/log/audit
56M /var/log/sa
74M /var/log/netwitness
222M /var/log/rabbitmq
613M /var/log/mongodb
6.4G /var/log/logstash
10G /var/log
In the above example, the /var/log/logstash directory is using 6.4 GB of the 10 GB that has been utilized of the /var/log mount point.
# awk "BEGIN { printf \"/var/log/logstash is using %3.2f%% of the /var/log mount\n\", $(du -ksx /var/log/logstash | awk '{print $1}')/$(df -k /var/log | tail -n1 | awk '{print $2}') * 100, 4 }"
Sample Output:
/var/log/logstash is using 64.06% of the /var/log mount
In the above example the /var/log/logstash directory is using 64.06% of the 10 GB /var/log mount. Size of the /var/log mount is derived from a command equivalent to:
# df -hP /var/log
Cause
In NetWitness 10.x there was a /etc/logrotate.d/logstash configuration file to manage the logstash log files.
This configuration file may not exist by default in some versions of NetWitness 11.x.
Resolution
- Obtain a console session on the problem host (using SSH, iDRAC console or local console)
- If the /var/log mountpoint is >90% utilized as seen in `df -h /var/log`, then move/remove a few of the oldest logstash-plain-*.log files under the /var/log/logstash directory, so that there is sufficient free disk space to compress the remaining files using gzip and for the logrotate command to be able to run.
- Create a /etc/logrotate.d/logstash file to have all the following lines:
/var/log/logstash/*.log {
maxsize 100M
weekly
rotate 7
dateext
dateformat -%Y-%m-%d
extension .log
copytruncate
compress
missingok
notifempty
su logstash logstash
}
- Make sure the permissions are correct on the logstash logrotate config file and also that the files in the logstash directory belong to the logstash group
# chmod 644 /etc/logrotate.d/logstash
# chown logstash:logstash /var/log/logstash/*
- Clean up the current directory and restart the logstash service
# find /var/log/logstash -name "logstash-plain-*.log" -type f -print0 | xargs -0 -I % --no-run-if-empty gzip -9 %
# sudo -u logstash systemctl restart logstash.service
- To test the newly created configuration file is correct, run the following command to force an immediate logrotate run:
# logrotate --force --verbose /etc/logrotate.d/logstash
Notes
What if the logrotate command doesn't tidy up the files in the /var/log/logstash directory?
If the logrotate command encounters an error condition, then it will stop without doing anything.
Check if any of the below is the cause for no action.
- The file /var/log/logstash/logstash-plain.log doesn't exist or is empty, then logrotate stops. ls -l /var/log/logstash/logstash-plain.log
-rw-r--r--. 1 logstash logstash 0 Mar 6 03:24 logstash-plain.log
Restart the logstash service to create the file, or enter some text into the file.
systemctl restart logstash
- A .gz file already exists for a particular .log file, then logrotate can't compress the .log file to overwrite the existing .gz file and it stops.
-rw-r--r--. 1 logstash logstash 6.3M Mar 4 23:19 logstash-plain-2019-03-03.log
-rw-r--r--. 1 logstash logstash 20 Mar 4 23:19 logstash-plain-2019-03-03.log.gz
Move or rename the existing .gz file. - Run the logrotate command with the debug (-d) switch. No action occurs with the debug switch selected, but the output shows what the logrotate command would do, and possibly show the error.
-d, --debug Turns on debug mode and implies -v. In debug mode, no changes will be made to the logs or to the logrotate state file.
logrotate -df /etc/logrotate.d/logstash
- Can just manually delete the oldest /var/log/logstash logs files, keeping the newest 7 files.