RSA NetWitness Endpoint Server stopped accepting agent data and is in unhealthy state

[#1]

Article Number

000039663

Applies To

RSA Product Set: RSA NetWitness Platform
RSA Product/Service Type: Endpoint Server
RSA Version/Condition: 11.5
Platform: CentOS
O/S Version: 7
 

Issue

Log Decoder service on Endpoint Log Hybrid shows 0 EPS indicating that no data is received from the Endpoint Agents configured for Windows collection.
'AGENT LAST SEEN' stats do not get updated under the Hosts page.
The logs are received and the stats are updated for a very short period after Endpoint Server service is restarted but the issue continues soon after.

endpoint-server.log shows warnings like below.

2021-03-29 19:21:42,265 [               dataprocessor-6] WARN    EndpointManagement|Retry operation failed for 15 attempts, caused by
org.bson.BsonMaximumSizeExceededException: Payload document size is larger than maximum of 16777216.

 

Cause

The issue can occur when large documents containing FILELESS_SCRIPT exceeds the Mongo DB's limitation of 16MB.
The attached test.js script can be used to determine the documents and their collection that exceed 16MB.

1. Download test.js to the Endpoint Server.
2. Modify test.js to replace netwitness with the actual deploy_admin password.
3. Execute the script.
   mongo test.js
4. Confirm documents exceeding 16MB are returned as shown in the example below.

   ...
   Collection filecontexthistory documents: 4797384
   From Collection = filecontexthistory, document Exceeds Size = 605fa9348c4a4924bfe5b4fc
   From Collection = filecontexthistory, document Exceeds Size = 605fb376d638001695cbe1ea
   From Collection = filecontexthistory, document Exceeds Size = 6062424b336def5ba96163f2
   ...

   
    

Resolution

In order to resolve the issue, delete the documents that exceed 16MB.

1. Back up the collection(s).

   mongoexport --ssl --sslAllowInvalidHostnames -u deploy_admin --authenticationDatabase admin --db endpoint-server --collection <collection_name> --out /root/<collection_name>.json

   e.g. mongoexport --ssl --sslAllowInvalidHostnames -u deploy_admin --authenticationDatabase admin --db endpoint-server --collection filecontexthistory --out /root/filecontexthistory.json
    
2. Delete every document noted in step 4 in Cause.

   mongo admin -u deploy_admin
   use endpoint-server 
   db.<collection_name>.remove({"_id" : ObjectId("<The Returned IDs>")})
   

   e.g. db.filecontexthistory.remove({"_id" : ObjectId("605fa9348c4a4924bfe5b4fc")})
    
3. Exist the Mongo console and run 'mongo test.js' to confirm no document is found to be exceeding the size.
    
4. Restart Endpoint Server service.
   systemctl restart rsa-nw-endpoint-server 

The issue is commonly noticed in environments where the Endpoint agents version is on 11.5.1 or older due to the bug which is fixed in 11.5.2 per ASOC-104219/SACE-14786. To avoid encountering the issue again, the agents need to upgrade to 11.5.2 or later.
 

Notes

If the issue persists, collect the following to investigate further.

1. Mongo outputs.

   mongo admin -u deploy_admin
   use endpoint-server
   db.file.count()
   db.file.find({firstFileName : /FILELESS_SCRIPT/}).count()
   db.filecontexthistory.count()
   db.filecontexthistory.find({firstFileName : /FILELESS_SCRIPT/}).count()

2. SOSReport after enabling the debugging for Endpoint Server service.
   From Endpoint Server service's Explore view, select logging, click on the field next to levels, and enter EndpointManagement:DEBUG,AgentManagement:DEBUG
   Collect SOSReport
3. Number of Endpoint agents deployed and its version.