RSA NetWitness Concentrator's Session Behind is increasing with low session rate without error message

[#1]

Article Number

000039770

Applies To

RSA Product Set: RSA NetWitness Logs & Network
RSA Product/Service Type: Security Analytics Server
RSA Version/Condition: 11.5.x, 11.6.x
Platform: Centos7
O/S Version: 7
 

Issue

Concentrator's session rate is very low and session behind is increasing. But there is not any error message in /var/log/messages
The rate and behind number are back to normal right after restarting concentrator service, but same problem persists after few minutes.

Image description

Cause

In /var/log/messages, many queries from an external IP address(non-NetWitness).

Jun 21 06:46:38 nw-loghybrid NwConcentrator[13681]: [Engine] [audit] User admin (session 7349590, <external ip>:34942) has logged in
Jun 21 06:46:38 nw-loghybrid NwConcentrator[13681]: [SDK-Query] [audit] User admin (session 7349590, <external ip>:34942) has issued query (channel 7349602) (thread 15902) (priority: 20): flags=1 query="select device.ip,device.type,alias.host,ip.src,city.src,country.src,latdec.src,longdec.src,alert,category,client,ip.dst,city.dst,country.dst,latdec.dst,longdec.dst,ip.dstport,time,policy.name,severity,event.desc,event.type where ip.src exists && time=\"2021-Jun-21 04:13:01\"-\"2021-Jun-21 04:14:01\"" size=1500 queryPriority=20
Jun 21 06:46:38 nw-loghybrid NwConcentrator[13681]: [Engine] [audit] User admin (session 7349628, <external ip>:34946) has logged in
Jun 21 06:46:38 nw-loghybrid NwConcentrator[13681]: [SDK-Query] [audit] User admin (session 7349628, <external ip>:34946) has issued query (channel 7349637) (thread 35224) (priority: 20): flags=1 query="select device.ip,device.type,alias.host,ip.src,city.src,country.src,latdec.src,longdec.src,alert,category,client,ip.dst,city.dst,country.dst,latdec.dst,longdec.dst,ip.dstport,time,policy.name,severity,event.desc,event.type where ip.src exists && time=\"2021-Jun-21 03:59:01\"-\"2021-Jun-21 04:00:01\"" size=1500 queryPriority=20
Jun 21 06:46:38 nw-loghybrid NwConcentrator[13681]: [SDK-Query] [audit] User admin (session 7349058, <external ip>:34886) has finished query (channel 7349067, queued 00:00:00, execute 00:00:00): flags=1 query="select device.ip,device.type,alias.host,ip.src,city.src,country.src,latdec.src,longdec.src,alert,category,client,ip.dst,city.dst,country.dst,latdec.dst,longdec.dst,ip.dstport,time,policy.name,severity,event.desc,event.type where ip.src exists && time=\"2021-Jun-21 04:04:01\"-\"2021-Jun-21 04:05:01\"" size=1500 queryPriority=20
Jun 21 06:46:38 nw-loghybrid NwConcentrator[13681]: [SDK-Query] [info] channel 7349067 memory stats: 0 B total 58.915764 MB max 0 allocs 13 max allocs
Jun 21 06:46:38 nw-loghybrid NwConcentrator[13681]: [Engine] [audit] User admin (session 7349058, <external ip>:34886) has logged out

Too many established connections in concentrator's explorer page : /connections

Image description

Resolution

Administrator needs to confirm on the external IP and ensure that the connections are auto closed once the queries are completed and there are no stale/idle connections with time_wait state.

Closing all connections could resolve this problem temporarily.
To close all connections, go to concentrator's explorer page, right-click 'connections' then click 'Properties'. Then choose 'closeAll' from dropdown, then click 'Send' button.
Image description