Agents missing from ECAT UI

[CraigCameronWei]

I am doing a small proof of concept deployment of ECAT. During some tests, I reinstalled the product on the workstation acting as the ECAT server. After reinstalling the ECAT console server and ECAT UI, I created a new agent package and confirmed that all tests in the packager pass. Then, I installed the new agent on two machines that had previously had agents deployed. Both of these systems are able to communicate with the ECAT server with the new agent, just as they had with the old.

Unfortunately, every other system I have deployed the new agent to is *not* showing up in ECAT UI. These other agents are communicating with the console server, according to the output from the console server, for example:

18 10:21:28:8614 Query 11 10.10.10.220 -> /getcommand

18 10:21:28:9092 Query 11 completed successfully

In the ECAT UI > Events tab, I see this:

[DavidWaugh1]

Hi

Things I would check are:

• Are all the SQL Agent jobs running successfully
• Can you see any files building up in the c:\ECAT\Server\Queued Data folder?
• In Wirehark, do you see connections on both UDP 444 and TCP 443 (the default ports) from the agents coming in?

[DavidWaugh1]

Can you also run the following query against the ECAT$PRIMARY database using SQL Server Management Studio Express

If you paste the output you will see something similar to the following:

elapsed_time	the_time	youngest_batch	oldest_batch	batches_processed	errors	max_queue_wait_hours	max_queue_wait_minutes
T+0 x60 min	2016-07-18 17:48:12.643	2016-07-18 17:48:02.781	2016-07-18 16:49:57.615	65	0	0.025000	1.500000
T+1 x60 min	2016-07-18 16:47:21.530	2016-07-18 16:47:17.011	2016-07-18 15:50:35.362	67	0	0.003055	0.183333
T+2 x60 min	2016-07-18 15:47:51.203	2016-07-18 15:47:44.901	2016-07-18 14:48:24.494	84	0	0.005277	0.316666
T+3 x60 min	2016-07-18 14:46:20.747	2016-07-18 14:46:14.151	2016-07-18 13:48:45.671	80	0	0.007500	0.450000
T+4 x60 min	2016-07-18 13:48:03.537	2016-07-18 13:47:55.156	2016-07-18 12:49:05.370	61	0	0.005833	0.350000
T+5 x60 min	2016-07-18 12:47:50.840	2016-07-18 12:47:44.812	2016-07-18 11:48:23.248	71	0	0.003611	0.216666
T+6 x60 min	2016-07-18 11:48:11.530	2016-07-18 11:48:04.715	2016-07-18 10:51:14.622	63	0	0.015555	0.933333
T+7 x60 min	2016-07-18 10:41:20.807	2016-07-18 10:41:13.346	2016-07-18 10:11:12.088	10	1	0.009444	0.566666
T+8 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+9 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+10 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+11 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+12 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+13 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+14 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+15 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+16 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+17 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+18 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+19 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+20 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+21 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+22 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+23 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+24 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+25 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+26 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+27 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+28 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+29 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+30 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+31 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+32 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+33 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+34 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+35 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+36 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+37 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+38 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+39 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+40 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+41 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+42 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+43 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+44 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+45 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+46 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+47 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+48 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+49 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+50 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+51 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+52 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+53 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+54 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+55 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+56 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+57 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+58 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+59 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+60 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+61 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+62 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+63 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+64 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+65 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+66 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+67 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+68 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+69 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+70 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+71 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+72 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+73 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+74 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+75 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+76 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+77 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+78 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+79 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+80 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+81 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+82 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+83 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+84 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+85 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+86 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+87 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+88 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+89 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+90 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+91 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+92 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+93 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+94 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+95 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+96 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+97 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+98 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+99 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL

This is a query we use to check system health.

[CraigCameronWei]

• All the agent jobs that have a last run date have completed successfully, except for syspolicy_purge_history. There are 4 ECAT agent jobs that have never run (ECAT_DatabaseCleanup, ECAT_DatabaseMaintenance, ECAT_ProcessAgentMigration, ECAT_ProcessMergeScanBatches).
• Yes, there are files building up in C:\ECAT\Server\QueuedData\.
• Wireshark capture shows UDP port 444 and TCP port 443 connections from agents to the console server

Here's the output of the SQL query:

T+0 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+1 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+2 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+3 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+4 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+5 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+6 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+7 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+8 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+9 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+10 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+11 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+12 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+13 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+14 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+15 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+16 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+17 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+18 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+19 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+20 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+21 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+22 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+23 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+24 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+25 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+26 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+27 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+28 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+29 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+30 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+31 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+32 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+33 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+34 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+35 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+36 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+37 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+38 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+39 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+40 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+41 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+42 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+43 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+44 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+45 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+46 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+47 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+48 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+49 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+50 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+51 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+52 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+53 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+54 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+55 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+56 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+57 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+58 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+59 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+60 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+61 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+62 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+63 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+64 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+65 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+66 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+67 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+68 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+69 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+70 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+71 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+72 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+73 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+74 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+75 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+76 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+77 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+78 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+79 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+80 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+81 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+82 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+83 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+84 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+85 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+86 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+87 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+88 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+89 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+90 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+91 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+92 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+93 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+94 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+95 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+96 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+97 x60 min	2016-07-14 17:20:50.897	2016-07-14 17:20:50.788	2016-07-14 16:34:19.954	51	0	0.003055	0.183333
T+98 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL
T+99 x60 min	NULL	NULL	NULL	NULL	NULL	NULL	NULL

[DavidWaugh1]

Hi

This doesn't look like you are processing anything. Make sure that the user that you are running the SQL Agent service as has full access to the queued data folder.

Sent from my iPhone

[MohamedEssam]

Hi;

Indeed the output of this query mean you are not processing data at all and ecat agent data flow is not working

Please check the attached guide for troubleshooting agent server communication flow

pay attention to point 6 if your SQL database is on remote server

Best Regards

Mohamed Essam

[CraigCameronWei]

SQL Agent Service is running as local system. Looks like SYSTEM has full control on this folder, inherited from C:\

[CraigCameronWei]

SQL database is not on a remote server. The account the services run as has full control on the QueuedData folder.

Any other suggestions?

[MohamedEssam]

Hi Craig;

Well check the attached txt file and run both queries and see if you can notice any error

Also i would suggest to run another query against dbo.machines table there is a column called localIP filter by ip address of this machine

and double check mark as deleted column =1 or 0 (1 mean it will show in ecat ui , 0 mean it won't show)

from your description you mention ECAT SQL jobs merge scan is not running this job is essential to complete scan process try to run it manually

and see if it run successfully

Best Regards

Mohamed Essam

[CraigCameronWei]

I ran that job manually, and now I’m seeing agents populate in the UI. How often should that job be scheduled? Is that configured in ECAT, or do I need to do it in SQL server?

Thanks!