The RSA NetWitness Event Source Monitoring (ESM) feature is causing RSA NetWitness Head Unit server performance issues in large NetWitness deployments.
Symptoms on the RSA NetWitness Head Unit server include:
High memory usage (which can result in Health & Wellness alarms of "High Swap Utilization")
RSA NetWitness UI unavailability on port 443 due to the rsa-sms service crashing/becoming unresponsive
NOTE: The rsa-sms service is a prerequisite of the jettysrv service.
High disk utilization on tokumx service volumes (e.g. /var/lib/netwitness/database/tokumx) due to large collection sizes in MongoDB
The following symptoms have also been reported in the RSA NetWitness 11.x user interface: (both may have other causes)
All hosts showing red status on the Admin > Health & Wellness > Monitoring tab (suggesting an issue with the rsa-sms service)
Large number of services showing as Offline in Admin > Services (suggesting that carlos is having trouble monitoring the service status)
This is caused by the ESM Alarms based on calculated baseline being enabled by default. (This feature is known as as ESM Automatic Monitoring.)
On large environments, a large amount of resources are required to create and maintain baseline data for automatic monitoring and notifications. This is a beta feature at present and turning it off will ONLY turn off the advanced baselining and automatic alerting.
Features that are unaffected by disabling this include:
Regular ESM policy-based alerts
Health & Wellness policy-based alerting
To resolve this issue you must disable ESM Automatic Monitoring and remove some of the large ESM collections in the MongoDB by performing the following steps:
Disable the feature in RSA Security Analytics 10.x by navigating to the Administration > Event Sources > Settings tab and deselect all checkboxes as shown in the example below.
For RSA NetWitness 11.x and any 10.x hosts where the node id does not appear in the mco ping output, you may need to have the collectd service restarted manually using one of the commands below.