Rate limiting errors in /var/log/messages on an RSA Security Analytics VLC due to collectd errors

[#1]

Article Number

000032294

Applies To

RSA Product Set: Security Analytics
RSA Product/Service Type: Security Analytics Server, Virtual Log Collector
RSA Version/Condition: 10.4.x,10.5.x
Platform: CentOS
O/S Version: EL6

Issue

Rate-limiting messages (as shown below) appear whenever there is a large number of log messages being recorded in the /var/log/messages file.

These errors can be stopped by setting appropriate parameters for $SystemLogRateLimitInterval and  $SystemLogRateLimitBurst in the /etc/rsyslog.conf file, as explained in the article entitled RSA Security Analytics Log Decoder is dropping system messages due to rate-limiting. However, adjusting these parameters may not help if a large number of log messages are logging.

Dec 17 05:05:10 DELGG7SIEMVLC7 rsyslogd-2177: imuxsock begins to drop messages from pid 2059 due to rate-limiting 
Dec 17 05:05:13 DELGG7SIEMVLC7 rsyslogd-2177: imuxsock lost 34281 messages from pid 2059 due to rate-limiting

Cause

One reason that rate-limiting messages may appear is due to a large number of collectd errors being logged, as shown in the example below.

Dec 24 11:42:53 DELGG7SIEMVLC7 collectd[2059]: NgNativeReader_NwLogCollector-FastUpdate: nwsdk failure: NwResponseData returned 0 and NwLastError failed 
Dec 24 11:42:53 DELGG7SIEMVLC7 collectd[2059]: NgNativeReader_NwLogCollector-FastUpdate: nwsdk failure: NwResponseData returned 0 and NwLastError failed 
Dec 24 11:42:53 DELGG7SIEMVLC7 collectd[2059]: NgNativeReader_NwLogCollector-FastUpdate: nwsdk failure: NwResponseData returned 0 and NwLastError failed

Resolution

Follow below steps to stop the collectd errors from logging on the appliance.

1. Connect to the appliance via SSH as the root user.
2. Issue the command below and confirm that it executes successfully.

   puppet agent -t

3. Stop and restart the collectd service.

   service collectd stop
   service collectd start

   Sample output of these commands is shown below.
   Image description
4. Verify the latest /var/log/messages do not have the collectd errors.

If you are unsure of any of the steps above or experience any issues, contact RSA Support and quote this article number for further assistance.