How to aggregate ESA events from the current time in the RSA NetWitness Platform (Version 11.3 and above)

[#1]

Article Number

000001042

Applies To

RSA Product Set: NetWitness Platform
RSA Product/Service Type: Event Stream Analysis, Correlation Server
RSA Version/Condition: 11.3 and above
Platform: CentOS
O/S Version: EL7

Issue

Consider the following scenario: A device that is configured to forward events to ESA has been down for a substantial period of time, and millions of events have accumulated. By the time the situation with the device is rectified, an exorbitant number of events may need to be processed. This can lead to a flood of events sent all at once to the ESA server, which may have debilitating effects on performance. Moreover, the historic alerts in situations of this nature typically contain repetitive information that is not needed.

Resolution

In situations such as those described, or when other circumstances dictate limiting the events that are sent by time, follow these steps to aggregate ESA events to the current time:

1. Within the RSA NetWitness UI, go to Administration > Services. From the services list, select ESA Correlation > Actions > View > Explore.
2. Within the Explore menu of the NetWitness UI, go to correlation > stream > metrics. In 11.3, ESA keeps track of source positions in order for it to be able to resume aggregation from the same position after restarting the service. To clear the backlog, the position tracking information will need to be deleted - this information is stored within the positionTrackingEntity collection within the correlation-server mongoDB instance on node zero.
   It shows two values on the second column next to each server instance: [<sessionid>, <UTC time>] 
    
3. On the Admin Server otherwise known as Node Zero, log into mongo with the following command:

   
   mongo admin -u deploy_admin -p <password>

4. Select the correlation-server database for queries.

   
   > use correlation-server
   

5. Display the list of collections and locate the positionTrackingEntity collection. 

   
   > show collections
   > db.positionTrackingEntity.find().toArray()

6. Within the displayed list of the positionTrackingEntity collection, it shows the specific server instances that you see within the Explore view in the NetWitness UI mentioned above. 
7. Narrow down the list of all server instances to the specific server instances that are behind in time. Change the IP address below according to your device.

   
   db.positionTrackingEntity.find({"_id": {$regex: /10.1.1.1/}}).toArray()

8. Stop the correlation-server. Within the NetWitness UI, go to Administration > Services. From the services list, select ESA Correlation > Actions > View > Stop. You can also stop from the ESA SSH session using this command:

   
   systemctl stop rsa-nw-correlation-server

9. Remove all records of the specific server instances found in the list in Step 7. Change the IP address below according to your device.

   
   db.positionTrackingEntity.remove({"_id": {$regex: /10.1.1.1/}})

10. Run Step 7 again to confirm that those specific server instances have been removed.
11. Start the correlation-server. Within the NetWitness UI, go to Administration > Services. From the services list, select ESA Correlation > Actions > View > Start. You can also start from the ESA SSH session using this command:

    
    systemctl start rsa-nw-correlation-server

12. Within the Explore menu of the UI, go to correlation > stream > metrics. Check the UTC time value for the server instance in question to confirm that aggregation has resumed much closer to the current time.

Notes

This article applies only to version 11.3 and above of the RSA NetWitness Platform.  For earlier versions, please refer to the article entitled: How to aggregate ESA events from the current time in the RSA NetWitness Platform (Version 11.2 and below) for instructions.