SA Time slowly deviates from NTP

[B_Hill]

Here is the current situation. I admin Netwitness on 4 different networks and all are configured to use NTP. 3 of the networks are fine and keep time between the SA Head and other appliances w/no issue. However, on one network, all 8 NW devices have time that slowly deviates from NTP despite NTP being configured. Based off my rough calculations, time deviates from the NTP server time by about 10 seconds per day. I have to go into each of the 8 NW devices on this network and run the command to manually update time from the NTP server every few days, which is less than ideal and could cause issues if investigating any type of potential network event and using timestamps to determine cause. Any ideas on what could be causing this, or better yet, any corrective action?

[JohnKisner]

Hello B_Hill,

When you run the command manually are you retrieving the time from the admin server or the NTP source that is configured on the admin server? I'm curious if there is a block somewhere. Is even the admin server time deviating?

[B_Hill]

Using the manual command to retrieve time from the NTP server. Yes, the admin server time is deviating and I believe that all other appliances are keeping time with the admin server due to the fact that they all deviate at the same rate. Suppose I could test that theory by just syncing the admin server and see if the others follow suit. 

[B_Hill]

So, the admin server is the one with the issue. Once I manually sync the admin server to the NTP server, the others follow suit after a brief time period (I believe about 10 minutes or so). 

[JohnKisner]

Can you confirm that the NTPd service is running and pointing to an active NTP source? If it is pointing correctly, do you see any errors in /var/log/messages about communication issues?

[B_Hill]

Yes, I can confirm that NTPd service is active (running). It is also pointed correctly and not noticing anything about comm issues.

[JohnKisner]

@B_Hill can you confirm that all the devices are pointing to the same NTP source? I've seen customers have a couple of different NTP sources and they not be in sync like you would think they should be. If they are all pointing at the same source and yet are still deviating, can you confirm if your NetWitness servers are on CentOS 7 (12.3.x and below) or the newer Alma Linux (12.4.x and above)?

[B_Hill]

All devices are not pointing to the NTP source. Only the Node 0 points to the NTP source and the others are getting time from the Node 0, which they are doing correctly. So, just the Node 0 is the one with time accuracy issues and the others just follow whatever his time is, so are also inaccurate. Currently still on CentOS 7, although soon to be on Alma. Hoping within the next 2 months to make the switch to Alma. Also hoping that the switch will address this timing issue as I have not been able to find a fix. Ticket opened with Netwitness.

[JohnKisner]

@B_Hill 

The NTP source that the Node0 is pointing to, is it internal or external source? Can you change the NTP to another source and see if that changes anything? So all the NetWitness servers are staying in sync with each other just that all of them are off by the same amount due to the Node0 being off. You mentioned about needing to go into all 8 devices to update the NTP so it sounded like they were using a different source than Node0.

So if you only update Node0 do the other devices update correctly automatically if left to poll Node0? Can you tell from the logs how often that NTP is polling the time source? The default should be every 60 seconds or so.

[B_Hill]

NTP source is internal. The other 3 networks that I manage have a secondary NTP source, but this one only has one. Changing it wouldn't do any good because when I force the Node 0 to sync, it syncs to the correct time.

At one point, I did go in and manually sync all 8 servers w/NTP source. After that, I figured out the 7 servers all get their time from Node 0 and that Node 0 is the only one to get time from the NTP source. Yes, the other devices all update appropriately off of Node 0. Once I manually force Node 0 to sync to NTP source, all other devices fall in line. Then, time slowly starts slipping again at the rate of about 7-8 seconds per day. 

According to ntpstat, polling the NTP server occurs every 64 seconds.