Upgrade of Node X failed with an update error message in the NetWitness Platform UI for 11.5.x

[#1]

Article Number

000039749

Applies To

RSA Product Set: NetWitness Platform
RSA Product/Service Type:  RSA NetWitness Platform
RSA Version/Condition: 11.5.x
Platform: CentOS
O/S Version: 7

Issue

When upgrading Node X to NetWitness Platform version 11.5.x, the NetWitness Platform UI displays a general update error.

The upgrade to NetWitness Platform 11.5.x completes successfully in the backend for Node X but the following items are seen:

• No error appears in the chef-solo.log and the original NetWitness version is still displayed in the Admin Server UI with an update error message being displayed.
• The following error message is in the node-infra-server.log file, found at /var/log/netwitness/node-infra-server/

ERROR CONFIGURATION|call-home failed org.springframework.amqp.AmqpIOException: javax.net.ssl.SSLException: Fatal Alert received: Bad Certificate
2021-04-29 07:13:08,130 [permission-synchronizer] WARN Security|Certificate for CN=e63040ff-ece4-4ae4-95f5-77031e870fae,OU=NetWitness Platform,O=RSA,L=Reston,ST=VA,C=US issued by CN=Puppet CA: e63040ff-ece4-4ae4-95f5-77031e870fae is not trusted

• The following error message is in the orchestration-server.log on the Admin server, found at /var/log/netwitness/orchestration-server/

com.rsa.asoc.launch.api.transport.client.RequestTimeoutException: Request to node-infra-server.<uuid-for-node-infra-server-in-vlc>./rsa/platform/node/run timed out.

 

Cause

The issue is caused by a lack of communication between the RabbitMQ service and the Node-Infra service due to a broken trust store.

Workaround

Important: Do not restart the Jetty service or reboot the NetWitness server until all the steps below are completed.

1. SSH into the Node X that is having the issue.
2. Stop the node-infra service using the following command:

# systemctl stop rsa-nw-node-infra-server

3. Take a backup of /etc/netwitness/node-infra-server using the following command:

# cp -r /etc/netwitness/node-infra-server /root/

4. Remove the keystore and lockbox using the following command:

# cd /etc/netwitness/node-infra-server/
# rm -rf lockbox.ss* key*

5. Create a backup and then remove the rsa-nw-node-infra-server-opts-managed.conf file.

# cp /etc/systemd/system/rsa-nw-node-infra-server.service.d/rsa-nw-node-infra-server-opts-managed.conf /root/
# rm /etc/systemd/system/rsa-nw-node-infra-server.service.d/rsa-nw-node-infra-server-opts-managed.conf

6. Remove the node-infra-server.completed file.

# rm /etc/pki/service/bootstrap/node-infra-server.completed

7. Check the node.json file to confirm the descriptor block contains the correct upgrade version. If upgrading to 11.5.1.0, the descriptor block should show 11.5.1.0. If the version returned by the below command does not match the upgrade version, contact Customer Support before proceeding further and provide this article number for reference.

# jq '.global.descriptor.version' /etc/netwitness/config-management/node.json
"11.5.1.0"

8. SSH into the Admin server (Node 0).
9. Run the following command to refresh the Node X information. Once the refresh is completed successfully, the upgrade should show complete.

# nw-manage --refresh-host --host-key <IP address of Node X>