How-to Bulk Remove "sources" from Event Source Management (ESM) in RSA NetWitness Platform

[#1]

Article Number

000034830

Applies To

RSA Product Set: Security Analytics
RSA Product/Service Type: Event Source Management
RSA Version/Condition: 10.6.x, 11.x
Platform: CentOS
O/S Version: 6, 7
 

Issue

If monitoring a particular event source type is no longer necessary the following procedure can be used to disable and remove the monitoring.

Warning: the "remove" commands presented below will permanently remove data from the ESM DB, specifically event sources that are currently being monitored by the SA server.  Use with great caution.
 

Resolution

The following commands are run in an SSH session on the SA server.

1. Connect to the ESM database 10.x:

   
   mongo esm

   1a. Connect to the ESM database 11.x:

   
   mongo esm -u deploy_admin -p netwitness --authenticationDatabase admin

2. List all records, each record is a single event source and type:
   Note: A single event source may have multiple event source types.

   
   db.eventsources.find()

   Sample output:

   
   { "_id" : "10.25.53.30-snort", "_class" : "EsmEventSource", "attributes" : { "asoc-es-type" : "snort", "asoc-es-ip" : "10.25.53.30", "asoc-es-logDecoderUuid" : "f859c9f4-93b6-47d0-b673-136852ddfc80", "asoc-es-logCollector" : "S5-LDEC-33", "asoc-es-logCollectorUuid" : "f859c9f4-93b6-47d0-b673-136852ddfc80", "asoc-es-logDecoder" : "S5-LDEC-33" }, "searchAttributes" : { "asoc-es-type" : "SNORT", "asoc-es-ip" : NumberLong(169424158), "asoc-es-logDecoderUuid" : "F859C9F4-93B6-47D0-B673-136852DDFC80", "asoc-es-logCollector" : "S5-LDEC-33", "asoc-es-logCollectorUuid" : "F859C9F4-93B6-47D0-B673-136852DDFC80", "asoc-es-logDecoder" : "S5-LDEC-33" } }
   { "_id" : "10.25.53.30-unknown", "_class" : "EsmEventSource", "attributes" : { "asoc-es-type" : "unknown", "asoc-es-ip" : "10.25.53.30", "asoc-es-logDecoderUuid" : "f859c9f4-93b6-47d0-b673-136852ddfc80", "asoc-es-logCollector" : "S5-LDEC-33", "asoc-es-logCollectorUuid" : "f859c9f4-93b6-47d0-b673-136852ddfc80", "asoc-es-logDecoder" : "S5-LDEC-33" }, "searchAttributes" : { "asoc-es-type" : "UNKNOWN", "asoc-es-ip" : NumberLong(169424158), "asoc-es-logDecoderUuid" : "F859C9F4-93B6-47D0-B673-136852DDFC80", "asoc-es-logCollector" : "S5-LDEC-33", "asoc-es-logCollectorUuid" : "F859C9F4-93B6-47D0-B673-136852DDFC80", "asoc-es-logDecoder" : "S5-LDEC-33" } }

3. List all records for a single event source and type:

   
   db.eventsources.find({"_id" : "10.25.53.30-snort"})

   Sample output:

   
   { "_id" : "10.25.53.30-snort", "_class" : "EsmEventSource", "attributes" : { "asoc-es-type" : "snort", "asoc-es-ip" : "10.25.53.30", "asoc-es-logDecoderUuid" : "f859c9f4-93b6-47d0-b673-136852ddfc80", "asoc-es-logCollector" : "S5-LDEC-33", "asoc-es-logCollectorUuid" : "f859c9f4-93b6-47d0-b673-136852ddfc80", "asoc-es-logDecoder" : "S5-LDEC-33" }, "searchAttributes" : { "asoc-es-type" : "SNORT", "asoc-es-ip" : NumberLong(169424158), "asoc-es-logDecoderUuid" : "F859C9F4-93B6-47D0-B673-136852DDFC80", "asoc-es-logCollector" : "S5-LDEC-33", "asoc-es-logCollectorUuid" : "F859C9F4-93B6-47D0-B673-136852DDFC80", "asoc-es-logDecoder" : "S5-LDEC-33" } }

4. List all records for a single event source type, 'asoc-es-type' of 'snort' :

   
   db.eventsources.find({"attributes.asoc-es-type": "snort"})

   Sample output:

   
   { "_id" : "10.25.53.30-snort", "_class" : "EsmEventSource", "attributes" : { "asoc-es-type" : "snort", "asoc-es-ip" : "10.25.53.30", "asoc-es-logDecoderUuid" : "f859c9f4-93b6-47d0-b673-136852ddfc80", "asoc-es-logCollector" : "S5-LDEC-33", "asoc-es-logCollectorUuid" : "f859c9f4-93b6-47d0-b673-136852ddfc80", "asoc-es-logDecoder" : "S5-LDEC-33" }, "searchAttributes" : { "asoc-es-type" : "SNORT", "asoc-es-ip" : NumberLong(169424158), "asoc-es-logDecoderUuid" : "F859C9F4-93B6-47D0-B673-136852DDFC80", "asoc-es-logCollector" : "S5-LDEC-33", "asoc-es-logCollectorUuid" : "F859C9F4-93B6-47D0-B673-136852DDFC80", "asoc-es-logDecoder" : "S5-LDEC-33" } }

5. Count the number of records for a single event source type, 'asoc-es-type' of 'snort' :

   
   db.eventsources.count({"attributes.asoc-es-type": "snort"})

6. Remove records for a single event source type, 'asoc-es-type' of 'snort' :

   
   db.eventsources.remove({"attributes.asoc-es-type": "snort"})

7. Count the number of records for snort event sources to verify the remove command was successful :

   
   db.eventsources.count({"attributes.asoc-es-type": "snort"})

It's also possible to bulk remove event sources based on either the Log Decoder or Log Collector consuming the events using the following commands:

Discover the UUID of the specific LD or LC by running the following command in and SSH on the specific host 10.x:

/etc/puppet/scripts/node_id.py

Sample output 10.x:

96562317-9414-41c5-81bd-b46bfdeb712a

Discover the UUID of the specific LD or LC by running the following command in and SSH on the specific host 11.x:

cat /etc/salt/minion

Sample output 11.x:

master: nw-node-zero
hash_type: sha256
log_level: warning
log_level_logfile: info
id: 857e0870-ff90-4e4c-a27f-3e38b7dce6a5

Use the UUID obtained in the previous command in the following commands:
Remove based on Log Decoder:

db.eventsources.remove({"attributes.asoc-es-logDecoderUuid" : "96562317-9414-41c5-81bd-b46bfdeb712a"})

Remove based on Log Collector:

db.eventsources.remove({"attributes.asoc-es-logCollectorUuid" : "96562317-9414-41c5-81bd-b46bfdeb712a"})

To extract the Range of stats between 1st to 10th DEC, 2020:

echo 'db.eventsources.find({"searchAttributes.asoc-es-lastSeen" : {$gte: NumberLong("1606803124000"), $lte: NumberLong("1607580724000")} }).pretty()' |mongo esm -u deploy_admin -p netwitness --authenticationDatabase admin > /root/esmrange.txt

To remove Range of stats between 1st to 10th DEC, 2020:

echo 'db.eventsources.remove({"searchAttributes.asoc-es-lastSeen" : {$gte: NumberLong("1606803124000"), $lte: NumberLong("1607580724000")} })' |mongo esm -u deploy_admin -p netwitness --authenticationDatabase admin

To remove stats before 10th DEC, 2020:

echo 'db.eventsources.remove({"searchAttributes.asoc-es-lastSeen" : {$lte: NumberLong("1607580724000")} })' |mongo esm -u deploy_admin -p netwitness --authenticationDatabase admin

To show correct stats in Event Sources->Manage page after removal:

echo 'db.eventsources.reIndex()' |mongo esm -u deploy_admin -p netwitness --authenticationDatabase admin

 

Notes

Please use epochconverter website to find Epoch time for specific date.