Event Source Monitoring (ESM) Beta is alerting and can cause server issues with operating environment in large RSA NetWitness deployments

[#1]

Article Number

000001545

Applies To

RSA Product Set: RSA NetWitness Logs & Network, Security Analytics
RSA Product/Service Type: Event Source Monitoring (ESM)
RSA Version/Condition: 10.6.x, 11.0.x, 11.1.x

Issue

The RSA NetWitness Event Source Monitoring (ESM) feature is causing RSA NetWitness Head Unit server performance issues in large NetWitness deployments.

Symptoms on the RSA NetWitness Head Unit server include:

• High memory usage (which can result in Health & Wellness alarms of "High Swap Utilization")
• RSA NetWitness UI unavailability on port 443 due to the rsa-sms service crashing/becoming unresponsive

  NOTE: The rsa-sms service is a prerequisite of the jettysrv service.

• High disk utilization on tokumx service volumes (e.g. /var/lib/netwitness/database/tokumx) due to large collection sizes in MongoDB

The following symptoms have also been reported in the RSA NetWitness 11.x user interface: (both may have other causes)

• All hosts showing red status on the Admin > Health & Wellness > Monitoring tab (suggesting an issue with the rsa-sms service)
• Large number of services showing as Offline in Admin > Services (suggesting that carlos is having trouble monitoring the service status)

Cause

This is caused by the ESM Alarms based on calculated baseline being enabled by default. (This feature is known as as ESM Automatic Monitoring.)
 
On large environments, a large amount of resources are required to create and maintain baseline data for automatic monitoring and notifications.  This is a beta feature at present and turning it off will ONLY turn off the advanced baselining and automatic alerting.
 
Features that are unaffected by disabling this include:

• Regular ESM policy-based alerts
• Health & Wellness policy-based alerting

Resolution

To resolve this issue you must disable ESM Automatic Monitoring and remove some of the large ESM collections in the MongoDB by performing the following steps:

1. Disable the feature in RSA Security Analytics 10.x by navigating to the Administration > Event Sources > Settings tab and deselect all checkboxes as shown in the example below.

Notes

For RSA NetWitness 11.x and any 10.x hosts where the node id does not appear in the mco ping output, you may need to have the collectd service restarted manually using one of the commands below.

service collectd restart              # NW 10.X
systemctl restart collectd.service    # NW 11.X