RSA Security Analytics concentrator service is going offline with the following error: Session has exceeded the maximum allowed channels (2000)" in /var/log/messages

[#1]

Article Number

000032178

Applies To

RSA Product Set: Security Analytics
RSA Product/Service Type: SA Security Analytics Concentrator
RSA Version/Condition: 10.3.X, 10.4.X, 10.5.X
Platform: CentOS
O/S Version: 6
 

Issue

The Security Analytics concentrator service demonstrates instability, appearing to going offline then back on in the UI.  When reviewing the concentrator's /var/log/messages file, it is observed that there are thousands of messages such as:

NwCreateChannelByPath returned 0; code 128; error: Session 1585 has exceeded the maximum allowed channels (2000)

These messages reconcile to the same date and time of the first observed instance of the instability.  

Cause

In this instance, the number of threads allocated in the concentrator configuration was set to 15; below the default value of 20, as seen in the screenshot here:

Image description 

Under load, this may cause the concentrator process to become thread bound, as the concentrator process will be forced to wait for a thread to become available before a pending request can be processed. 

To that note, in some instances on busy systems, it may be necessary to increase this value if it is currently set at the default value of 20.
 

Resolution

1. Check the number of allocated threads in the configuration screen for the concentrator.  

• If it is lower than 20, increase it to 20, then restart the nwconcentrator process.  
• If already at 20, increase it in reasonable increments, starting at 25. 

2. To adjust the thread value, evoke the Security Analytics UI,  and click on Administration > Services.
3. Select the concentrator service, then View > Config.
4. Under the System Configuration section, double click on the number value of Threads under the Config Value column.
5. Enter an appropriate value (if not set to 20, enter 20, if already at 20, enter 25).
6. A service restart of nwconcentrator is then required.  From the command line as root, issue the following command:

restart nwconcentrator

Be sure to monitor /var/log/messages closely when the system is under load to ensure the thread value is high enough to support thread requests.