Health & Wellness uses an old IP for connecting to a device - How to Resolve

[XavierTrepanier]

Health and Wellness leverages RabbitMQ to be able to collect the actual status of any components of the RSA Netwitness platform. After changing an IP on a component the Health and Wellness keep communicating with the previous IP. To be able to resolve this issue you need to do the following:

Open your browser and log in to the RabbiMQ Management interface: https://IP_of_your_head_unit:15671

Log in using the deploy_admin account 

When logged in, go to the Admin Tab

And in the Admin Tab, Select the Federation Upstreams on the right 

Identify the wrong upstream and take note of the virtual host, URI, Expires and the Name of this upstream

Create a new upstream and enter the right information for the URI, with the new IP, the Name, the Virtual Host and the Expires:

When adding this new upstream, it will match the upstream name and automatically replace the one with the wrong information.

And now the device is in a ready state and the health status changed from RED to GREEN

[RichardB]

Thanks a lot for this guide. There is a CLI procedure for this that might be easier to use. The commands need to be run on the NetWitness server:

1. List all upstream parameters:

# rabbitmqctl list_parameters -p /rsa/system
Listing runtime parameters
federation-upstream     carlos-upstream-e583a899-8640-4568-88c1-09655784a9fd    {"uri":"amqps://10.1.2.3:5671?auth_mechanism=external","expires":3600000}
federation-upstream     carlos-upstream-d8a76073-88c1-4831-8640-1401b5a38b3b    {"uri":"amqps://10.1.2.4:5671?auth_mechanism=external","expires":3600000}
federation-upstream     carlos-upstream-132a7cbb-3adb-16cf-b969-00eb3353a067    {"uri":"amqps://10.1.2.5:5671?auth_mechanism=external","expires":3600000}

2. Update one of the hosts to a new IP address:

# rabbitmqctl set_parameter -p /rsa/system federation-upstream carlos-upstream-132a7cbb-3adb-16cf-b969-00eb3353a067 '{"uri":"amqps://10.9.8.7:5671?auth_mechanism=external","expires":3600000}'
Setting runtime parameter "carlos-upstream-132a7cbb-3adb-16cf-b969-00eb3353a067" for component "federation-upstream" to "{\"uri\":\"amqps://10.9.8.7:5671?auth_mechanism=external\",\"expires\":3600000}"

[RichardB]

If you have any NetWitness appliances behind NAT the RabbitMQ config needs to be fixed every time the node-zero is upgraded. With lots of appliances this gets old pretty fast. One of my collegues wrote a script to automate the process:

#!/usr/bin/env bash

# Script to set the correct IP addresses in Health&Wellness
# To be executed as 'root' user on the nw-node-zero.

# ---------------------------------------------------------------------------

declare -r IP_LIST="/tmp/ip_list.txt"
declare -r RM_LIST="/tmp/rm_list.txt"

declare LINE UUID ITEM

declare -A nodes names

# ---------------------------------------------------------------------------

echo "Querying nodes and IP addresses, please wait...."

upgrade-cli-client --list > ${IP_LIST}

while read LINE
do
        UUID=$(echo $LINE | awk '{print substr($2,4,length($2)-4)}')
        nodes[${UUID}]="$(echo $LINE | awk '{print substr($3,6,length($3)-6)}')"
        names[${UUID}]="$(echo $LINE | awk '{print substr($4,6,length($4)-6)}')"
done < ${IP_LIST}

# ---------------------------------------------------------------------------

echo "Obtaining current rabbitMQ information...."

rabbitmqctl list_parameters -p /rsa/system > ${RM_LIST}

while read -u3 LINE
do
        if [[ ${LINE} != +(*upstream*) ]]
        then
                continue
        fi

        for ITEM in ${!nodes[@]}
        do
                if [[ ${LINE} = +(*${ITEM}*) ]]
                then
                        if [[ ${LINE} != +(*${nodes[$ITEM]}*) ]]
                        then
                                read -p "Setting system '${names[$ITEM]}' to IP address ${nodes[$ITEM]}. Is this correct (y/n)? " ANSWER
                                if [[ ${ANSWER} = "y" ]]
                                then
                                        rabbitmqctl set_parameter -p /rsa/system federation-upstream carlos-upstream-${ITEM} '{"uri":"amqps://'${nodes[${ITEM}]}':5671?auth_mechanism=external","expires":3600000}'
                                fi
                        fi
                fi
        done
done 3< ${RM_LIST}

# ---------------------------------------------------------------------------

echo "Cleaning up!"

rm -f ${IP_LIST}
rm -f ${RM_LIST}

exit

# ---------------------------------------------------------------------------

[RichardB]

Since NetWitness 11.5 RabbitMQ is using the salt id of the nodes so the above manual fix is no longer needed. Additionally, NAT IP addresses are now officially supported. See System Maintenance: Manage Custom Host Entries