Problems with Windows event collection from Aggregators

[ShishirKumar1]

Hi All,

I am currently trying to integrate windows aggregators in our environment. The problem that I am facing is related to the rolling of a channel for the windows logs. I have the following error in the logs:

Log for channel Security may have rolled over. Previous/Current record number: xxxx/xxxx. 

As per the RSA link, I have increased the maximum log storage size to 2 GB from 20 MB on the windows aggregator and also tried to change the polling duration.  However, this is still not fixing the issue.

I have been trying with different Poll interval / Poll duration and maximum events. Still, I keep getting the same rollover error. Is there a way to derive an optimum setting for polling interval/ poll duration and maximum events ? Thanks for the assistance in advance.

Current settings:

Poll duration: 50 seconds
Poll interval: 60 seconds
Maximum events: 200000 

[SravanKoneti1]

Hi Shishir,

option1:

In windows side, You can try increasing max log size before overriding old events

option2:

In Netwitness side, Set Max collcetion on collector side and disable debug.

both option details are available in 000029686 - Windows legacy log collection warni... | RSA Link 

[ShishirKumar1]

Hi Sravan,

Thanks for the response. I have used the same link you suggested and increased the maximum log size as well as tried to even set poll interval as -1 and maximum number of logs as 0 (which means unlimited) however it still comes back with this error.

I am not sure what else needs to be done for this error to not appear. Any suggestions ?

[SravanKoneti1]

Hi Shishir,

Please run below command in Log Collector to see if any pending messages for collection.

rabbitmqctl list_queues -p logcollection consumers name messages

Looking at error, Security channel logs are rolling over. Do you think any busy security event id can be excluded in Windows side?

[EricPartington]

not to avoid the problem but could using the Endpoint windows agent help solve your logging problems and get logs out of the system without worrying about polling intervals and batches?

depending on your version of NW that might be a quicker solution

[ShishirKumar1]

Hi Eric,

We are on 10.6.5.1 and so the Endpoint windows agent cannot be use I suppose.

Sravan Koneti‌ - The queue shows zero 

1 rabbitmq.log 0
1 shovel.checkpoint 0
1 shovel.cmdscript 0
1 shovel.file 0
1 shovel.odbc 0
1 shovel.syslog 0
1 shovel.windows 0

But I am still getting the following errors:

[WindowsCollection] [warning]  [processing] [WorkUnit] [processing] Log for channel Security may have rolled over. Previous/Current record number: 34190835/125837258.

I am only collecting forwarded events and the current configuration is:

Polling Interval : 180 secs
Polling Duration: 120 seconds
Maximum event per poll: 200000

Any further suggestions to resolve this please?

[david_waugh]

Hi I would try lowering your polling interval even more, to say even 10 seconds or lower.

You want to get as many events as you can, so polling more frequently should help.

[ShishirKumar1]

Hi David,

Okay I will try that.

But I had a question about the polling duration and polling interval, should the polling duration always be lesser than polling interval.

For example, if I try polling interval of 10 s, what would you suggest the polling duration to be ? 

[david_waugh]

Hi my understanding is:

Polling Interval - How often it will attempt to collect events

Polling Duration: - The maximum it can take to collect the events.

I think I am correct that if a polling interval hasnt finished then it wont start a new one.

eg

If you poll every 10 seconds and have a duration of 60 seconds, then the next poll will be at the minimum of (polling interval, polling duration)