Concentrator service frequently has failed with core dump files

[#1]

Article Number

000002096

Applies To

RSA Product Set: NetWitness Logs & Network
RSA Product/Service Type: Core Appliance
RSA Version/Condition: 11.X
Platform: CentOS
O/S Version: 7

Issue

Concentrator service has failed with core dump files almost every day.

Following debug level log occurred during the crash.

Jun 16 00:19:02 concentrator collectd[1716]: restreader.py: Unable to Connect to Endpoint.  Endpoint config: {'username': 'guest', 'password': '********', 'path': 'api/overview', 'verify': False, 'scheme': 'https', 'port': 15671}; error: ('Connection aborted.', BadStatusLine("''",))
Jun 16 00:19:02 concentrator NwConcentrator[170206]: [Rest] [debug] New connection (10.xx.xxx.xxx:39586) on port 50105
Jun 16 00:19:02 concentrator NwConcentrator[170206]: [Rest] [debug] Received a valid HTTP header - GET /sdk
Jun 16 00:19:02 concentrator NwConcentrator[170206]: [Engine] [audit] User admin (session 1051285, 10.xx.xxx.xxx:39586) has logged in
Jun 16 00:19:02 concentrator NwConcentrator[170206]: [Rest] [debug] Received a valid HTTP request
Jun 16 00:19:02 concentrator NwConcentrator[170206]: [SDK-Query] [audit] User admin (session 1051285, 10.xx.xxx.xxx:39586) has issued query (channel 1051294) (thread 173750) (priority: 20): size=3000 query="select * where sessionid=254368466068"
Jun 16 00:19:02 concentrator NwConcentrator[170206]: [Rest] [debug] Found request handler for HTTP resource: /sdk
Jun 16 00:19:02 concentrator NwConcentrator[170206]: [SDK-Msearch] [debug] Search  expanded to sessionid=254368466068
Jun 16 00:19:02 concentrator NwConcentrator[170206]: [Rest] [debug] Sent HTTP response of 52951 bytes (keeping alive)
Jun 16 00:19:02 concentrator NwConcentrator[170206]: [SDK-Query] [audit] User admin (session 1051285, 10.xx.xxx.xxx:39586) has finished query (channel 1051294, queued 00:00:00, execute 00:00:00): size=3000 query="select * where sessionid=254368466068"
Jun 16 00:19:02 concentrator NwConcentrator[170206]: [SDK-Query] [info] channel 1051294 memory stats: 0 B total 3.098221 MB max 0 allocs 31 max allocs
Jun 16 00:19:02 concentrator NwConcentrator[170206]: [Rest] [debug] New connection (10.xx.xxx.xxx:39590) on port 50105
Jun 16 00:19:02 concentrator kernel: Request Handler[178951]: segfault at 70 ip 000055c212b3ddc9 sp 00007f92bfff60b0 error 4 in NwConcentrator[55c211ea1000+18d8000]

Jun 16 00:19:14 concentrator systemd: nwconcentrator.service: main process exited, code=dumped, status=11/SEGV
Jun 16 00:19:14 concentrator systemd: Unit nwconcentrator.service entered failed state.
Jun 16 00:19:14 concentrator systemd: nwconcentrator.service failed.

 

Cause

Concentrator service crash has been occurring due to high number(about 500) of TCP connections with customer's 3rd party device, when the connections are getting released few of the acknowledgement are failing resulting in the crash.

Workaround

1. Connect to the concentrator via SSH. Find how many connections and which device make a high number of connections to the concentrator.

# NwConsole -c login localhost:50005 admin <password of admin> -c connections count
# NwConsole -c login localhost:50005 admin <password of admin> -c connections ls

2. Try to disconnect all connections.

# NwConsole -c login localhost:50005 admin <password of admin> -c connections closeAll

If the 3rd party device make many connections within short duration, disable the connections from 3rd party device.

3. Customer should check and prevent the 3rd party device from making too many connections.