ESA Alert Summary Page displays the error "not authorized for query on im.system.namespaces" in RSA Security Analytics 10.5

[#1]

Article Number

000031301

Applies To

RSA Product Set: Security Analytics
RSA Product/Service Type: Security Analytics Server, Security Analytics UI, Incident Management (IM), Event Stream Analysis (ESA)
RSA Version/Condition: 10.5.0.0
Platform: CentOS
O/S Version: EL6
 

Issue

Customer on SA 10.5.0.0 and above seeing errors on ESA Alerts Summary page on SA UI and tokumx.log file being filed by messages like: 
The tokumx.log file is reporting errors similar to the examples below.

Fri Sep 18 11:06:45.507 [initandlisten] waiting for connections on port 27017 
Fri Sep 18 11:06:45.507 [websvr] admin web console waiting for connections on port 28017 
Fri Sep 18 11:06:45.591 [conn2] assertion 16550 not authorized for query on im.system.namespaces ns:im.system.namespaces query:{} 
Fri Sep 18 11:06:45.591 [conn2] problem detected during query over im.system.namespaces : { $err: "not authorized for query on im.system.namespaces", code: 16550 } 
Fri Sep 18 11:06:47.203 [initandlisten] connection refused because too many open connections: 819

The ESA Alerts Summary page in the Security Analytics UI is also reporting the following error:

not authorized for query on im.system.namespaces

Issuing the command lsof -i:27017 displays a large number of connections as shown below.

mongod 25979 tokumx 829u IPv4 7300117 0t0 TCP RSAAPP2P:27017->puppetmaster.local:36753 (ESTABLISHED) 
mongod 25979 tokumx 830u IPv4 7300118 0t0 TCP RSAAPP2P:27017->puppetmaster.local:36754 (ESTABLISHED) 
mongod 25979 tokumx 831u IPv4 7300119 0t0 TCP RSAAPP2P:27017->puppetmaster.local:36755 (ESTABLISHED) 
mongod 25979 tokumx 832u IPv4 7300120 0t0 TCP RSAAPP2P:27017->puppetmaster.local:36756 (ESTABLISHED) 
mongod 25979 tokumx 833u IPv4 7300121 0t0 TCP RSAAPP2P:27017->puppetmaster.local:36757 (ESTABLISHED) 
mongod 25979 tokumx 834u IPv4 7300122 0t0 TCP RSAAPP2P:27017->puppetmaster.local:36758 (ESTABLISHED) 
mongod 25979 tokumx 835u IPv4 7300123 0t0 TCP RSAAPP2P:27017->puppetmaster.local:36759 (ESTABLISHED) 
mongod 25979 tokumx 836u IPv4 7300124 0t0 TCP RSAAPP2P:27017->puppetmaster.local:36760 (ESTABLISHED) 
mongod 25979 tokumx 837u IPv4 7300125 0t0 TCP RSAAPP2P:27017->puppetmaster.local:36761 (ESTABLISHED) 
mongod 25979 tokumx 838u IPv4 7300126 0t0 TCP RSAAPP2P:27017->puppetmaster.local:36762 (ESTABLISHED) 
mongod 25979 tokumx 839u IPv4 7300127 0t0 TCP RSAAPP2P:27017->puppetmaster.local:36763 (ESTABLISHED)

Cause

This issue can result from one of the following causes:

• The default username and password for Incident Management Mongodb gets lost on the upgrade process from SA 10.4 to SA 10.5 and above
• The IM service gets hung at some point during the upgrade and this prevents the connection between Incident Management and ESA from being able to establish.  Subsequently, there are many new connection attempts between the two appliance and the ESA service goes down as a result.

Workaround

To resolve the issue, log into the Security Analytics UI and reset the username and password for the Incident Management service's MongoDB database following the instructions in the Security Analytics 10.5 User Guide.

​After making the change, the MongoDB and Java will be stable and there will be no more errors in the tokumx.log file or the ESA Alerts Summary Page.