This issue describes when customers try to perform an SA upgrade from 10.4.1 to 10.5.0 and they run into issues with rabbitmq after Log Hybrid is upgraded. LogCollector device doesn't show up to date on SA UI and also appears unavailable.
[root@igteblrsiemhyb1 tmp]# service rabbitmq-server status
Status of node sa@localhost ...
Error: unable to connect to node sa@localhost: nodedown
DIAGNOSTICS
===========
attempted to contact: [sa@localhost]
sa@localhost:
* connected to epmd (port 4369) on localhost
* epmd reports: node 'sa' not running at all
other nodes on localhost: ['rabbitmqctl-35484']
* suggestion: start the node
current node details:
- node name: 'rabbitmqctl-35484@igteblrsiemhyb1'
- home dir: /var/lib/rabbitmq
- cookie hash: K2qzPBHLJ1HEpkGE+faD2g==
Also /var/log/rabbitmq/startup_log file shows following error:
[{app_utils,load_applications,2,[]},
{app_utils,load_applications,1,[]},
{rabbit,start_apps,1,[]},
{rabbit,broker_start,0,[]},
{rabbit,start_it,1,[]},
{init,start_it,1,[]},
{init,start_em,1,[]}]
{"init terminating in do_boot",{rabbit,failure_during_boot,{error,{"no such file or directory","nw_admin.app"}}}}
[FAILED]
Steps for the workaround in order to fix this issue are listed below:
1- Check RabbitMQ port is opened:
"netstat -ntpl |grep 4369"
tcp 0 0 0.0.0.0:4369 0.0.0.0:* LISTEN 3533/epmd
2 - Take a backup of the /etc/rabbitmq/rsa_enabled_plugins file
3- Edit the /etc/rabbitmq/rsa_enabled_plugins file
4- Remove nw_admin from it
5- Then run puppet agent -t
6 - Check RabbitMQ service:
# service rabbitmq-server status
7- If RabbitMQ is up, restart jettysrv on the SA server:
# stop jettysrv
# start jettysrv
8 - Restart the nwlogcollector service from the Log Hybrid device:
# stop nwlogcollector
# start nwlogcollector
9- If this doesn't work, try reinstalling the rsa-puppet-modules rpm:
# yum reinstall rsa-puppet-modules
10- If there is no relief from either of these, run the command manually and capture the output to attach to a Jira case for further troubleshooting:
# rabbitmq-plugins enable rabbitmq_federation