Error message "Unable to Drill Down" is displayed after upgrading to NetWitness 11.2.1.0

[#1]

Article Number

000001527

Applies To

RSA Product Set: NetWitness Logs & Network
RSA Product/Service Type: Admin Server
RSA Version/Condition: 11.2.1.0
Platform: CentOS 7

Issue

When investigating against a device, the "Unable to Drill Down Error" is displayed after upgrading from 11.2.1.0 or when a submitted query is completely altered after hitting Apply in Investigation.

Image description

Cause

This problem appears to affect customers who were previously on version 10.6 at one point and have updated to 11.2.1.0.

A possible cause of this error is due to a couple of index counters in the Mongo Database being reset as a result of the upgrade. The counter normally increments as users create queries in Investigation but after the upgrade the number has reset its self. As we try to create new queries in the system, it attempts to use a index value that already exists, thus throws an error that can be seen in the /var/netwitness/uax/logs/sa.log file on the Admin Server.

2019-02-04 16:03:46,901 [qtp611437735-66577] ERROR com.rsa.smc.sa.investigation.common.predicates.web.controllers.ajax.PredicateController - 
Error while saving Predicate [ device.host = 'dc1.example.com'  ] 
org.springframework.dao.DuplicateKeyException: { "serverUsed" : "192.168.2.100:27017" , "ok" : 1 , "n" : 0 , "err" : "E11000 duplicate key error collection: 
sa.userPredicate index: _id_ dup key: { : 1893 }" , "code" : 11000}; 
nested exception is com.mongodb.MongoException$DuplicateKey: { "serverUsed" : "192.168.2.100:27017" , "ok" : 1 , "n" : 0 , 
"err" : "E11000 duplicate key error collection: sa.userPredicate index: _id_ dup key: { : 1893 }" , "code" : 11000}

Workaround

SCRIPT:
We have provided a script to correct the database fields that may be the cause of the issue. If you take the zip file attached to this case, unzip it, and run the javascript file, it will do the same as the manual steps listed below. You will still need to stop and start the jetty service before and after running the script. Note, you will need the deployment password to do these steps and you will need to temporarily bring down the web server.

The below includes all of the commands you would need to unzip, stop jetty, execute the provided script, and then start the UI back up. 

[root@nwadmin1 ~]# unzip syncCounters.zip
Archive:  syncCounters.zip
  inflating: syncCounters.js
[root@nwadmin1 ~]# systemctl stop jetty
[root@nwadmin1 ~]# mongo admin -u deploy_admin syncCounters.js
MongoDB shell version v3.6.4
Enter password:
connecting to: mongodb://127.0.0.1:27017/admin
MongoDB server version: 3.6.4
Max user predicate id is 41
Updating user predicate counter to 42
Max predicate id is 41
Updating predicate counter to 42
[root@nwadmin1 ~]# systemctl start jetty

MANUAL:

This option is available to customers who cannot use the script or would rather prefer to do it manually. This article will walk you through how to reset the predicate and userPredicate table counters to the appropriate value.
 
First off, you should stop jetty to ensure that no users continue to make queries while we make our changes. Please note that by doing this you are stopping the User Interface.

systemctl stop jetty

Export a copy of the collection we are going to modify just in case a mistake is made so that we can restore. Note, the password for you database is your deployment password. In my case, it is "netwitness".

mongodump -u deploy_admin -p netwitness --authenticationDatabase=admin -d sa -c counter --out=/root/mongobackup

Then we shall login to make our changes.

mongo sa -u deploy_admin -p netwitness --authenticationDatabase=admin

First, we need to identify what is the largest index value that exists in the userPredicate collection.

db.userPredicate.find().sort({_id:-1}).limit(1).pretty()

Running this command will return something like this. Take note of the _id field.

> db.userPredicate.find().sort({_id:-1}).limit(1).pretty()
{
        "_id" : NumberLong(41),
        "_class" : "com.rsa.smc.sa.investigation.predicates.domain.bean.UserPredicate",
        "userId" : "admin",
        "lastAccessDate" : ISODate("2018-12-29T08:17:06.262Z"),
        "createdBy" : "admin",
        "dateCreated" : ISODate("2018-12-29T08:17:06.262Z"),
        "updatedBy" : "admin",
        "dateUpdated" : ISODate("2018-12-29T08:17:06.262Z"),
        "predicate" : DBRef("predicate", NumberLong(41))
}

Now, we shall attempt to update the counter for userPredicate with the ID value plus 1. Because my value was 41, I want to use 42.

db.counter.update({collection:'UserPredicate'},{$set:{seq:NumberLong(42)}})

Next, we shall repeat the same process but instead for predicate:

> db.predicate.find().sort({_id:-1}).limit(1).pretty()
{
        "_id" : NumberLong(56),
        "_class" : "com.rsa.smc.sa.investigation.predicates.domain.bean.Predicate",
        "displayName" : "winevent_nic",
        "query" : "device.type = 'winevent_nic'",
        "userName" : "admin",
        "createdBy" : "admin",
        "dateCreated" : ISODate("2018-12-29T08:17:06.178Z"),
        "updatedBy" : "admin",
        "dateUpdated" : ISODate("2018-12-29T08:17:06.178Z")
}

Thus, using the same logic, we will want to use 57:

db.counter.update({collection:'Predicate'},{$set:{seq:NumberLong(57)}})

Once this is finished, you may start jetty backup and check and see if the problem still persists.

systemctl start jetty

Notes

If this does not solve your issue, please open a case with RSA Technical Support and reference this article so that we may better assist you.