Upgrading RSA NetWitness hosts to 11.5.x is stuck at "In Queue for Update" status

[#1]

Article Number

000001019

Applies To

RSA Product Set: RSA NetWitness Platform
RSA Product/Service Type: Core Appliance
RSA Version/Condition: 11.5.x
Platform: CentOS
O/S Version: 7

Issue

When attempting to upgrade NetWitness hosts, some get stuck/not progressing in the NetWitness UI with "In Queue for Update".


As the Update button is greyed out for the affected hosts, no further attempts can be made to upgrade them through the UI.

The following error is found in /var/log/netwitness/config-management/chef-solo.log.

    ================================================================================
    Error executing action `run` on resource 'execute[generate-dhparams]'
    ================================================================================

    TypeError
    ---------
    no implicit conversion of nil into String

    Cookbook Trace:
    ---------------
    /var/lib/netwitness/config-management/cache/cookbooks/nw-mongo/recipes/config.rb:28:in `exist?'
    /var/lib/netwitness/config-management/cache/cookbooks/nw-mongo/recipes/config.rb:28:in `block (2 levels) in from_file'

    Resource Declaration:
    ---------------------
    # In /var/lib/netwitness/config-management/cache/cookbooks/nw-mongo/recipes/config.rb

     24: execute 'generate-dhparams' do
     25:   command 'openssl genpkey -genparam -algorithm DH ' \
     26:           "-out #{node['nw-mongo']['ssl_dhparams']['path']} " \
     27:           "-pkeyopt #{node['nw-mongo']['ssl_dhparams']['pkeyopt']}"
     28:   not_if { ::File.exist?(node['nw-mongo']['ssl_dhparams']['path']) }
     29: end
     30:

    Compiled Resource:
    ------------------
    # Declared in /var/lib/netwitness/config-management/cache/cookbooks/nw-mongo/recipes/config.rb:24:in `from_file'

    execute("generate-dhparams") do
      action [:run]
      default_guard_interpreter :execute
      command "openssl genpkey -genparam -algorithm DH -out  -pkeyopt "
      backup 5
      declared_type :execute
      cookbook_name "nw-mongo"
      recipe_name "config"
      domain nil
      user nil
      not_if { #code block }
    end

Cause

This failure is due to /etc/netwitness/config-management/node.json not being updated with the following lines which are parameters that are required to generated /etc/pki/nw/mongo/dhparams-rfc5114-3.pem which is new to NetWitness 11.5 and above.

"ssl_dhparams" :
{ "pkeyopt" : "dh_rfc5114:3", "path" : "/etc/pki/nw/mongo/dhparams-rfc5114-3.pem" }

 

Resolution

Run the upgrade from the command line

First try to resolve the issue, by performing the upgrade from the command line by running the command below on the NW Admin Server.

upgrade-cli-client --upgrade --host-key <ID, IP address, hostname or display name of affected host> --version 11.5.0.0 --verbose

Substitute
<ID, IP address, hostname or display name of affected host> with the correct detail of the server to upgrade.
11.5.0.0 with the correct version want to upgrade to.

See Upgrade 11.5: Appendix A. Offline Upgrade Using CLI for more information.
 

Or clear the "In Queue for Update" status

If the above doesn't resolve, then follow the steps below to clear the current status and start the upgrade again from the NetWitness UI.

1. SSH to the affected host.
2. (Conditional) Modify /etc/yum.repos.d/nw-rsa-base.repo to set the baseurl as ‘baseurl=https://nw-node-zero/nwrpmrepo/11.5.0.0/RSA’ only if the URL is NOT pointing to the version wanted.
3. (Conditional) Manually update to rsa-nw-component-descriptor-11.5.0.0 only if the package is not updated yet.

   rpm -qa |grep rsa-nw-component-descriptor
   yum update rsa-nw-component-descriptor

4. SSH to the Admin server.
5. Back up the appliance_update collection.

   mongoexport --db sa --collection appliance_update --out /root/appliance_update.json -u deploy_admin -p <PASSWORD> --authenticationDatabase admin

   Note: Replace <PASSWORD> with the actual password for the deploy_admin account.
6. Modify the Mongo document to change the "IN_QUEUE" state to “REBOOTED”.

   mongo admin -u deploy_admin -p
   use sa
   db.appliance_update.find() – confirm the UUID of the affected host and its "IN_QUEUE" status.
   db.appliance_update.update({ "_id" : "<UUID>" }, { $set: { status : "REBOOTED", acceptWarning : false}})
   db.appliance_update.update({ "_id" : "<UUID>" }, { $set: {"version" :{ "major" : 11, "minor" : 3, "servicePack" : 1, "patch" : 0 }}}) - To revert back to the current version(11.3.1.0 in this example) if the version was also changed to 11.5.0.0. Please modify the command to match the current version.
   exit
   

    
7. Refresh the UI and start the upgrade again.