ESA Rule deployment failing after updating to RSA NetWitness Platform 11.3.x

[#1]

Article Number

000001693

Applies To

RSA Product Set: NetWitness Logs & Network
RSA Product/Service Type: Core Appliance
RSA Version/Condition: 11.3.x
 

Issue

Few of the customers are experiencing issues with respect to ESA rules after updating to 11.3.x. They are receiving similar errors as below in the correlation-server.log when deploying ESA rules with Context-Hub lists;

in /var/log/netwitness/correlation-server/correlation-server.log of the ESA-Correlation Server
 

"Failed to validate subquery" .. (output truncated) .."Subqueries require one or more views to limit the stream, consider declaring a length or time window"

OR
 

com.espertech.esper.client.deploy.DeploymentItemException: Error starting statement: Error attaching view to event stream: Validation exception initializing virtual data window '{ContextHub_List}': Multiple entries with same key: {Key}_Incidents_datasource...
...
... (output truncated)
...
To index multiple values under a key, use Multimaps.index. [create window {ContextHub_List}.vdw:contexthub(LIST)

 

Cause

This could occur due to several reasons 

1. Presence of duplicate Context-Hub lists with the same name or lists without any values in ADMIN > Services > Context-Hub > view > Config > Lists
2. Presence of duplicate values in the ds_meta collection in the contexthub-server Mongo database on the ESA-Correlation Server 

Resolution

1. Presence of duplicate Context-Hub lists with the same name or lists without any values in ADMIN > Services > Context-Hub > view > Config > Lists
   Please navigate to the Netwitness UI > ADMIN > Services > Context-Hub > view > Config > Lists and check if the lists added to your rules have values in them or if there are any duplicate lists with the same name.
2. Presence of duplicate values in the ds_meta collection in the contexthub-server Mongo database on the ESA-Correlation Server 

Please check and identify if there are any duplicate values in the ds_meta collection of the contexthub-server mongo database. 

1. SSH to the Admin Server/Node Zero 
2. Login to the mongo database, deploy_admin password is required. 

   # mongo admin -u deploy_admin -p {deploy_admin_password}
   
   > use contexthub-server

   
   Check all entries with duplicated names
    

   > db.ds_meta.find()

3. Then remove the duplicates using the ID of each duplicate.
   Example:

   > db.ds_meta.remove({"_id" : ObjectId("5b2a608fe84959015206fbee")})

4. Then SSH to the ESA-Correlation Server and restart contexthub, correlation-server services.

   # systemctl restart rsa-nw-contexthub-server
   
   # systemctl restart rsa-nw-correlation-server

5. After that navigate back to the Admin Server/Node Zero.
6. Login to the mongo database, deploy_admin password is required. 

   # mongo admin -u deploy_admin –p {deploy_admin_password}
   
   > use sa

7. Delete the metaType collection 

   > db.metaType.drop()
   > exit

8. Navigate into the Netwitness UI and refresh the schema on ESA.
   Netwitness UI > Configure -> ESA Rules -> Settings (tab) -> Meta Key References and hitting the refresh button to repopulate the collection (‘metaType’ collection in ‘sa’ database)

Notes

1. If you see an error such as below in /var/log/netwitness/correlation-server/correlation-server.log of the ESA-Correlation Server - 

   java.net.URISyntaxException: Illegal character in authority at index 6: nws://admin:{Password}@{IP}:56005?compression=0&compressionLevel=6

   
   Navigate to  - Netwitness UI > Configure -> ESA Rules and select the relevant data source with the {IP} as shown in the error above, then remove the data source and re-add it. 
   
    
2. If you do not know your deploy_admin password you could find that by running the following command on the Admin Server/Node Zero - 

   # security-cli-client --get-config-prop --prop-hierarchy nw.security-client --prop-name platform.deployment.password --quiet