ESA Lag - Sessions Behind constantly

[ShishirKumar1]

Hi All,

We are continuously facing issues related to ESA lag. THE ESA falls behind on specific concentrators and there is delay in alerting. We have a RSA case open for this as well and we are being helped there but also thought of asking this in the forum for any further ideas.

We have reviewed ESA rules and disabled the rules taking maximum memory (which was around 60 MB). A service restart of the ESA works as a quickfix but the lag builds up again on these specific concentrators. There is no lag though between the decoder and the concetrator at any point of time.

Any pointers on what would be causing this? Thanks in advance

[EricPartington]

is your ESA still on SD cards?  depending on when it was installed and if it has been reimaged/SD cards disabled you might want to check.

I thought H&W had that stat somewhere but i cant find it at the moment, try this from the CLI

lsblk --ascii --output NAME,KNAME,MODEL,RM,RO,SCHED,TYPE,SIZE,STATE,MOUNTPOINT

look for Internal Dual SD and the partitions that are running from it.

If the card is there and running partitions consider reimaging the appliance to get off them (dual sd cards) to improve performance.

[JohnSnider]

By default ESA service aggregates 10,000 sessions per aggregation cycle, I found at one customer this was not enough to keep up with the session rate of some concentrators.

In Explore View of the ESA service:

Workflow -> Source -> nextgenAggregationSource

increase the value of "MaxSessions" to a point where the aggregation is maintained at near realtime.  in the case of my customer, I had to increase this to 50000 and also increase the "MaxEpsExpectedPerSource" to 25000 to get it to keep up.  Haven't seen any ill effects on the operation of the ESA service after the change, and alerts are not falling behind.

[ShishirKumar1]

Hi Eric,

Please find the output below: 

NAME KNAME MODEL RM RO SCHED TYPE SIZE STATE MOUNTPOINT
sdb sdb PERC H710P 0 0 cfq disk 7.3T running
`-sdb1 sdb1 0 0 cfq part 6.6T
|-VolGroup00-root (dm-0) dm-0 0 0 lvm 8G running /
|-VolGroup00-swap00 (dm-1) dm-1 0 0 lvm 8G running [SWAP]
|-VolGroup00-usrhome (dm-2) dm-2 0 0 lvm 4G running /home
|-VolGroup00-var (dm-3) dm-3 0 0 lvm 8G running /var
|-VolGroup00-rabmq (dm-4) dm-4 0 0 lvm 20G running /var/lib/rabbitmq
|-VolGroup00-vartmp (dm-5) dm-5 0 0 lvm 4G running /var/tmp
|-VolGroup00-varlog (dm-6) dm-6 0 0 lvm 10G running /var/log
|-VolGroup00-tmp (dm-7) dm-7 0 0 lvm 100G running /tmp
|-VolGroup00-rsaroot (dm-8) dm-8 0 0 lvm 10G running /opt/rsa
`-VolGroup00-apps (dm-9) dm-9 0 0 lvm 6.4T running /opt/rsa/database
sda sda PERC H710P 0 0 cfq disk 256M running
`-sda1 sda1 0 0 cfq part 255M /boot

Could you please check once as I could not find internal sd here.

[ShishirKumar1]

Hi John,

I have made the changes. Awaiting results. For now, there is no immediate impact as we are still facing lag.

[ShishirKumar1]

Hi All,

There is no significant change in the ESA lag yet. I modified the parameters as suggested.

[EricPartington]

provide more information about your architecture as there are many factors that can impact aggregation outside NW that have not been mentioned above.

• is aggregation behind on all concentrators or some?
• what is different about the concentrators that are behind?
• are the sessions always behind or does the system catch up in the off-hours of the network over the weekend?
• what is the linerate of the devices that stay caught up and those that are behind
• is aggregation using compression or encryption between ESA and concentrators?
• how many concentrators are aggregating to ESA (is there only one ESA or multiple)?
• how many ESA rules are deployed?
• what version of NW is this?

[ShishirKumar1]

• is aggregation behind on all concentrators or some? - On 2 of the concentrators continuously .
• what is different about the concentrators that are behind? - They have an EPS of around 5-6K . These are hybrids.
• are the sessions always behind or does the system catch up in the off-hours of the network over the weekend? - They do not catchup. We have to manually restart aggregation once a while
• what is the linerate of the devices that stay caught up and those that are behind - Almost same EPS on the ones that are caught up and the ones that are behind
• is aggregation using compression or encryption between ESA and concentrators? No
• how many concentrators are aggregating to ESA (is there only one ESA or multiple)? One ESA - 8 concentrators
• how many ESA rules are deployed? 200 +
• what version of NW is this? 10.6.5.1

[YogeshMavani]

Hi , Is you issue got resolved , we are also facing same issue. And engineer told us to increase resources on Decoder and concentration. 

[YogeshMavani]

Hi Eric,

My ESA output ,  How to check SD  in it 

NAME KNAME MODEL RM RO SCHED TYPE SIZE STATE MOUNTPOINT
sda sda PERC H730P Mini 0 0 deadline disk 931G running
|-sda1 sda1 0 0 deadline part 1M
|-sda2 sda2 0 0 deadline part 256G
| |-netwitness_vg00-root dm-0 0 0 lvm 29.3G running /
| |-netwitness_vg00-swap dm-1 0 0 lvm 4G running [SWAP]
| |-netwitness_vg00-varlog dm-3 0 0 lvm 10G running /var/log
| `-netwitness_vg00-usrhome dm-4 0 0 lvm 10G running /home
`-sda3 sda3 0 0 deadline part 519M /boot
sdb sdb PERC H730P Mini 0 0 deadline disk 3.7T running
`-sdb1 sdb1 0 0 deadline part 3.3T
`-netwitness_vg00-rsaapps dm-2 0 0 lvm 3.3T running /var/netwitness