The RSA NetWitness Endpoint UI shows machines online, but all scan data, tracking and other data is all old and seems gated to a specific day.
Notably, scans will fail, agents removed from the database will not return even when the agent is still sending data to the server, and no new data seems to be ingested despite the presence of the QueuedData folder adding and removing scanx files by the CS.
The SQL Server user for the SQL server log on as user and the SQL service agent log an user are not present in the user permissions of the QueuedData folder. This results in a scenario where constant errors are seen in the AgentBatches table which are highlighted below:
Error [usp_StageWithBulkInsert]: Category(2), Cannot bulk load. The file "\\ecatxxxxxxxx\QueuedData\2012-08-04T07x54x53.111111115Z_b12345678 -5189-3b73-80bd-1a1e1d129f78_4120_0002.scan4" does not exist., CSV File \\ecatxxxxxxxx\QueuedData\2012-08-04T07x54x53.1638315Z_b12345678 -5189-3b73-80bd-1a1e1d129f78_4120_0002.scan4
Firstly, this is NOT a kerberos trust delegation issue in this specific example. The SQL Server and SQL Server Agent jobs log on users have been disallowed from accessing the QueuedData folder. Simplest way to confirm this is to logon to the SQL server as the log on as user and then access the QueuedData folder network path. If it fails, then the issue is the SQL server can't access the QueuedData folder.
This scenario typically will not be seen on a standalone machine. It normally only applies to situations where the QueuedData folder is NOT on the same machine as the SQL server.
The typical cause of this issue is changing the log on as user for the SQL Server service of the SQL database machine any time after installation.
Check the path in the consoleserver.exe.config file and make sure it uses the network share path, as shown in the example below.
If the path for the network is correct in the console server service and data is not being ingested, check the permissions of the QueuedData folder. The issue will be due to the SQL Server log on as user not being present in the QueuedData folder path.