Hybrid - Aggregation Behind Consistently

[EvanRamos1]

Howdy,

SA/NW Version: 10.6.1.0

Hybrid Con/Dec/Collector.

We are having an odd issue that just started happening last week (after we updated some parsers from Live). During the day(s) our aggregation from our log decoder on our concentrator gets ridiculously behind. Never catches up. We've observed this is only happening during the day. Around 6PM , the problem goes away. Never reaching >1M sessions behind. This is a new issue, nothing has changed from what I can tell in our collections.

The health and wellness alerts say "check for noisy parser" but searching through /var/log/messages i see no errors. Somewhere else to check? Has anyone ran into an issue like this? 

To rule out data -

- Turned off syslog listener (514udp/tcp) on log collector . Left it off for 5 mins - no affect, behind still grows.

- Turned off Windows Collections. Left off for 5 mins - no affect, behind still grows.

- Turned off VMware Collections. Left off for 5 mins - no affect, behind still grows.

- Restarted nwlogdecoder service

We tried tuning sdk/config settings per: 000034117 - Performance of Concentrator Service on Hybrid Appliance is slow in RSA Security Analytics ; hasn't helped. 

We have a support case open with RSA and they have been helpful. However I thought to post the issue here in case someone has ran into this before. Or has any ideas. 

[ArthurCostigan1]

Hi Evan.   Can you tell us what version of Netwitness Logs you are running.  Are you using any custom Meta Keys? 

[EvanRamos1]

Howdy, using 10.6.1.0 for Logs, no custom meta only defaults. 

[DavidWaugh1]

What is the EPS through the Log Decoder? (Logdecoder -> Stats)

The concentrator aggregation rate seems low. If you restart the concentrator service (stop nwconcentrator then start nwconcentrator) does the throughput increase.

If you run 

iostat -m -N -x 5 are you seeing high disk utilisation on any partition (especially the concentrator index partition)

When did the issue first start. eg has it always been slow or has something changed?

[EvanRamos1]

I tried restarting the nwconcentrator service and uhhh it got really mad about that. Throwing lock errors all over the place. So i ended up rebooting the Hybrid box altogether and the problem magically went away. Crappy solution but i do have a case open with a tech dump taken so hopefully root cause can be found. I'm concerned the problem might surface again but for now i'm just monitoring it. 

Thanks for the tips/help!