NetWitness Correlation Service going offline in UI frequently

[#1]

Article Number

000039852

Applies To

RSA Product Set: RSA NetWitness Logs & Network
RSA Product/Service Type: Correlation Server
RSA Version/Condition: 11.5.X
Platform: CentOS
O/S Version: 7
 

Issue

Correlation service in UI->Admin->Services page goes offline frequently with below warnings.

correlation-server.log

2021-08-26 13:12:40,135 [        scheduled-health-check] WARN                Health|HealthStatus(name=rsa.process.jvm.memory-health, status=Fatal, details={Current usage %=98.42031963763793, Warning Threshold %=80, Fatal Threshold %=90})

Restarting correlation service using below command brings service online. But service goes offline after some time.

systemctl restart rsa-nw-correlation-server.service

Cause

This service goes offline due to the current usage memory of correlation-server is above Fatal Threshold. There could be ESA rules which are consuming high memory.

Resolution

Please follow the below steps to check rules with high memory consumption. 

1. Login to UI and Navigate to Configure->ESA Rules->Services.
2. For Each deployment, sort the rules based on ‘Memory Usage’. Check if there are any rules consuming 100 MB or more. Also, look for any rules skipping Memory metrics. Such rules will have a red exclamatory symbol in the ‘Memory Usage’ column.
3. Again sort the rules based on ‘CPU’. Check if there are any rules consuming 40% or more.look for any rules skipping CPU metrics. Such rules will have a red exclamatory symbol in ‘CPU’ column.
4. Please disable such rules in Step 2 and Step 3 and monitor the memory and cpu usage of the correlation server as below. 

Image description

The disabled rules which were consuming high memory and cpu can be re-worked using ESA Rule Writing Best Practices.