Duplicate AgentID is causing connection errors in RSA NetWitness Endpoint

[#1]

Article Number

000001982

Applies To

RSA Product Set: NetWitness Endpoint (ECAT)
RSA Version/Condition: 4.1.x, 4.2.x, 4.3.x, 4.4.x
Platform: Windows

Issue

When searching for an agent, it never appears in the Machine's List of the UI, or else it disappears and when searching by agentID the ID constantly rotates over time. A machine may never appear at all in the GUI when searching for it and may be associated with connection issues, when in fact there are no actual issues connecting to the agent.

Cause

A gold image or VM template was created at a customer site with the ECAT agent pre-installed as part of the deployment. This gets pushed out to X number of machines, which in turn causes many agents who share the same agentID in the database. The scan4 files get merged together, causing incomplete agent data, agents that do not appear in the list of machines when searching, and the agents entries in the database that have this issue become unreliable for investigations since different machines are mixed together with their data.

It is not currently supported to have gold images with the agent pre-installed. This issue is permanently fixed in the new 11.3 code which will automatically detect a different VM and will rotate the agent ID in response.

Resolution

Identifying Duplicate Agents

1. Run the attached SQL script CaptureIPUpdated.sql below, note the last three lines are commented out for removing the trigger and script later. Leave them commented out for now, so the trigger and table get created.
   • A new table called CapturedAgent will be created in the ECAT$Primary with a trigger that will grab hostnames as they are being overwritten.
2. Wait 48 hours at a minimum to gather enough entries to reliably determine that there are many duplicates.
3. Review the contents of the CapturedAgent table by running SELECT * FROM CapturedAgent in SQL Studio. This will provide a list of agents captured for review. Its organization looks like the example below.  (This is a sample that has been cleared)

AgentID                                  MachineName    OldMachineName     ChangedDate
12345678-1234-ABCD-1234-123456ABCDEF     NEWHOSTNAME    OLDHOSTNAME        2017-01-23 18:18:54.9330000

 

NOTE: Be careful interpreting the results; for instance, a single instance of a hostname change could possibly be legitimate if hostnames are being changed in the environment. Additionally, the OldMachineName field is important, because it may contain entries of 'Unknown'. These indicate new machines that have been added and should not be included, because its expected new entries will happen over time.

ADDITIONAL: Look for the same hostname repeating often; these are certain indicators of a duplicate agentID. The ID will be the same, and the hostname will bounce back and forth. It may cycle over several hostnames, but the agentID will always be identical for these hostnames.

How to remove the duplicate AgentIDs

1. Run the script ParseDuplicateAgents.sql to parse through the contents of the CapturedAgent table. This will generate a single list of hostnames that can be used with SCCM for instance to replace the hosts that are receiving duplicates. Copy this list to a text file.
2. Download the AgentID Scrambler file attached to this article, which is a .bat file.
3. Edit the .bat file in Notepad or some other editor and where it says SET _servicename=<insert_name_of_agent_service_here> replace with the name of the service for the ECAT agent.
4. Run the bat file against the machines list generated in Step 1 using SCCM or a similar tool. Note that in order to avoid errors in the script, it must be able to access 64-bit binaries, meaning it should be ran under SCCM with the Sysnative option.  For example:  C:\Windows\Sysnative\cmd.exe /C ecat_Uninstall_agentid_scrambler.bat
5. You should see the output below for each of the endpoints the .bat file is ran against, the first message is a sanity check for the first registry key which should already be removed.

   Starting ECAT_AgentID_Scrambler.bat
    
   Verifying existence of the following key:
   "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\EcatService"
    
   ERROR: The system was unable to find the specified registry key or value.
    
   It appears the value in that key does not exist.
   This is good since the goal of this script is to delete it.
    
    
   Verifying existence of the following key:
   "HKEY_LOCAL_MACHINE\SOFTWARE\ECAT\Temp"
    
    
   HKEY_LOCAL_MACHINE\SOFTWARE\ECAT\Temp
       ServiceUid    REG_BINARY    F862D0EA6227A742B1B19ECE4AE35EBC
    
    
   Trying to erase ServiceUid value in key...
    
   The operation completed successfully.
    
    
   The above key was successfully deleted.
    
    
   Scrambling completed SUCCESSFULLY.

   NOTE: The second registry key in Temp SHOULD have a message confirming its deletion, if not it was not ran correctly or the endpoint never had the agent installed on it.

6. This will also remove the ECAT agent. If using SCCM, it should be possible to perform two actions, the first to run the .bat file against the agent list and the second to install the agent using the agent packager.  Regardless, the agent will need to be reinstalled on each of these endpoints.
7. Confirm in the UI that each endpoint is showing up within an hour or so. How long it takes to merge depends on the speed of the database, number of agents being replaced and network connectivity, so it may show within minutes in the UI or take longer.
8. Once all duplicates have been removed from the environment, the last 3 commands (they will be commented out in the script) from the CaptureIPUpdated.sql script should be executed to clear the CapturedAgent table, remove the trigger, and delete the table as part of cleanup. If more duplicates are suspected to still exist, run the first command to delete the contents of the table to begin the process of searching out any remaining agents over again from Step 1.

If you are unsure of any of the steps above or experience any issues, contact RSA Customer Support and reference this article number for further assistance.

Notes

Be careful when interpreting the results of the script. When comparing the agentIDs, single hostname changes that don't repeat over multiple days are probably not worthy of being considered. The important results are many agents with the same ID(a dead giveaway) or repeated changes to the hostname.

If the hostname only changes once for an agentID and not again then its likely it was a legitimate action by a system administrator to modify the hostname.