RSA NetWitness Platform - Error 'disk resource limit alarm has tripped' in sa@localhost.log due to file collection logs piled up in Collector

[#1]

Article Number

000001691

Applies To

RSA Product Set: NetWitness Logs & Network
RSA Product/Service Type: Collector
RSA Version/Condition: 10.6.X, 11.X
Platform: CentOS
O/S Version: 7
 

Issue

Rabbitmq server is not running due to 'disk resource limit alarm' as below.

/var/log/rabbitmq/sa@localhost.log:

=WARNING REPORT==== 29-Aug-2019::09:26:47 ===
disk resource limit alarm has tripped on node sa@localhost.  Collection will be blocked until this alarm clears!

/var/log/messages:

Aug 24 04:05:12 Collector1 NwLogCollector[23993]: [MessageBrokerLogReceiver] [info] info 2019-08-24T04.05.12Z Disk free space insufficient. Free bytes:104907436032 Limit:104908750000

Cause

This issue is due to /var/netwitness/logcollector has reached 80% of available storage as below.

[root@Collector1 ~]# df -h

Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-root
                       20G  4.5G   14G  25% /
tmpfs                  48G     0   48G   0% /dev/shm
/dev/sdd1             248M  150M   85M  64% /boot
/dev/mapper/VolGroup00-usrhome
                      3.9G  417M  3.3G  12% /home
/dev/mapper/VolGroup02-tmp
                       20G  171M   19G   1% /tmp
/dev/mapper/VolGroup02-varlog
                      9.8G  3.6G  5.8G  39% /var/log
/dev/mapper/VolGroup01-nwhome
                       10G  935M  9.1G  10% /var/netwitness
/dev/mapper/VolGroup02-concroot
                       30G  940M   30G   4% /var/netwitness/concentrator
/dev/mapper/VolGroup03-concinde
                      300G   38G  263G  13% /var/netwitness/concentrator/index
/dev/mapper/VolGroup02-concmeta
                      2.4T  2.3T  130G  95% /var/netwitness/concentrator/metadb
/dev/mapper/VolGroup02-concsess
                      300G  285G   16G  95% /var/netwitness/concentrator/sessiondb
/dev/mapper/VolGroup01-lcol
                      489G  362G  127G  80% /var/netwitness/logcollector
/dev/mapper/VolGroup01-ldecroot
                       30G  923M   30G   4% /var/netwitness/logdecoder
/dev/mapper/VolGroup01-ldecinde
                       10G   37M   10G   1% /var/netwitness/logdecoder/index
/dev/mapper/VolGroup01-ldecmeta
                      300G  284G   17G  95% /var/netwitness/logdecoder/metadb
/dev/mapper/VolGroup01-ldecpack
                      2.8T  2.7T  149G  95% /var/netwitness/logdecoder/packetdb
/dev/mapper/VolGroup01-ldecsess
                       30G   29G  2.0G  94% /var/netwitness/logdecoder/sessiondb
/dev/mapper/VolGroup03-warec
                      400G   35G  366G   9% /var/netwitness/warehouseconnector
/dev/mapper/VolGroup00-vartmp
                      5.8G   12M  5.5G   1% /var/tmp
[root@hydsiemhyb01 ~]#

Resolution

Please use the below steps to identify the cause for high consumption in /var/netwitness/logcollector.

1. Run du -xh /var/netwitness/logcollector --max-depth=2|sort -h>collectingspace.txt command for consumption details.
2. tail collectingspace.txt command shows as below.

   20M     /var/netwitness/logcollector/rabbitmq/log
   27M     /var/netwitness/logcollector/rabbitmq
   791M    /var/netwitness/logcollector/statdb
   3.3G    /var/netwitness/logcollector/metadb
   17G     /var/netwitness/logcollector/upload/microsoft_dhcp_2008
   373G    /var/netwitness/logcollector/upload/iis_tvm
   389G    /var/netwitness/logcollector/upload
   389G    /var/netwitness/logcollector/upload_chroot
   389G    /var/netwitness/logcollector/upload_chroot/home
   782G    /var/netwitness/logcollector/

3. Above output indicates that the space consumption was due to /var/netwitness/logcollector/upload/iis_tvm which has huge files under /var/netwitness/logcollector/upload/iis_tvm/<eventSource>/save directory.
4. These files were saved after a successful process of logs by Log Collector by choosing "Save on Success" as below for File Collection event source.
   Image description
5. Remove the files under /var/netwitness/logcollector/upload/iis_tvm/<eventSource>/save directory to free up space.
6. service rabbitmq-server start command in collector.
7. Uncheck the "Save On Success" option for all file collection event sources in Collector->Config->Event Sources->File/Config page.

More details on Save On Success