LogCollector service unavailalable and device not showing up to date on RSA Security Analytics 10.4.1.0 User Interface

[#1]

Article Number

000031318

Applies To

RSA Product Set: Security Analytics
RSA Product/Service Type: SA Security Analytics Server
RSA Version/Condition: 10.4.1.0
Platform: CentOS
O/S Version: 6
 

Issue

This issue describes when customers try to  perform an SA upgrade from 10.4.1 to 10.5.0 and  they run into issues with rabbitmq after Log Hybrid  is upgraded. LogCollector device doesn't show up to date on SA UI and also appears unavailable. 
 

[root@igteblrsiemhyb1 tmp]# service rabbitmq-server status 
Status of node sa@localhost ... 
Error: unable to connect to node sa@localhost: nodedown

 

DIAGNOSTICS 
=========== 
attempted to contact: [sa@localhost] 

sa@localhost: 
* connected to epmd (port 4369) on localhost 
* epmd reports: node 'sa' not running at all 
other nodes on localhost: ['rabbitmqctl-35484'] 
* suggestion: start the node 

current node details: 
- node name: 'rabbitmqctl-35484@igteblrsiemhyb1' 
- home dir: /var/lib/rabbitmq 
- cookie hash: K2qzPBHLJ1HEpkGE+faD2g==

Also /var/log/rabbitmq/startup_log file shows following error: 
 

[{app_utils,load_applications,2,[]}, 
{app_utils,load_applications,1,[]}, 
{rabbit,start_apps,1,[]}, 
{rabbit,broker_start,0,[]}, 
{rabbit,start_it,1,[]}, 
{init,start_it,1,[]}, 
{init,start_em,1,[]}] 

{"init terminating in do_boot",{rabbit,failure_during_boot,{error,{"no such file or directory","nw_admin.app"}}}} 
[FAILED]

 

Cause

We've seen a few issues like this that stem from the nw_admin plugin.This is sometimes fixed by editing the /etc/rabbitmq/rsa_enabled_plugins file, removing nw_admin from it, then running puppet agent -t. 

Workaround

Steps for the workaround in order to fix this issue are listed below:

1- Check RabbitMQ port is opened: 
 

"netstat -ntpl |grep 4369" 
tcp 0 0 0.0.0.0:4369 0.0.0.0:* LISTEN 3533/epmd

2 - Take a backup of the /etc/rabbitmq/rsa_enabled_plugins file

3- Edit the /etc/rabbitmq/rsa_enabled_plugins file

4- Remove nw_admin  from it

5- Then run puppet agent -t

6 - Check RabbitMQ service:
 

# service rabbitmq-server status

7- If RabbitMQ is up, restart jettysrv on the SA server:
 

# stop jettysrv

# start jettysrv

8 - Restart the nwlogcollector service from the Log Hybrid device:
 

# stop nwlogcollector 

# start nwlogcollector

9- If this doesn't work, try reinstalling the rsa-puppet-modules rpm:
 

# yum reinstall rsa-puppet-modules

10- If there is no relief from either of these, run the command manually and capture the output to attach to a Jira case for further troubleshooting: 
 

# rabbitmq-plugins enable rabbitmq_federation