NetWitness Appliance after Successful Bootstrap (nwsetup-tui), Unable to Install Service on Newly Orchestrated Appliance (with 12.x OVA)

[#1]

Article Number

000003237

Applies To

NetWitness Product Set: NetWitness Platform
NetWitness Product/Service Type: NetWitness Platform
NetWitness Version/Condition: 12.x or later
Platform: CentOS / Alma

Issue

After successful Bootstrap (nwsetup-tui), you are unable to install a service on a newly orchestrated appliance (with 12.x OVA). This occurs when the NwAdmin server has been upgraded in the past by skipping minor versions (for example, jumping from 12.1 to 12.3 and skipping 12.2), resulting in missing records of the intermediate upgrade in the reference database.

When installing the service (category) from UI, the installation fails with the following error in /var/log/netwitness/config-management/chef-solo.log on the NodeX:

FATAL: Mixlib::ShellOut::ShellCommandFailed: nw_pki_certificate_wrapper[nw-pki] (nw-pki::certificates line 7) had an error: Mixlib::ShellOut::ShellCommandFailed: nw_pki_certificate[/etc/pki/nw/node/node-cert.pem]
(/var/lib/netwitness/config-management/cache/cookbooks/nw-pki/resources/certificate_wrapper.rb line 120) had an error: Mixlib::ShellOut::ShellCommandFailed: ruby_block[sign-csr-/etc/pki/nw/node/node-key.csr]
(/var/lib/netwitness/config-management/cache/cookbooks/nw-pki/resources/certificate.rb line 163) had an error: Mixlib::ShellOut::ShellCommandFailed: execute[issue[ss]:/etc/pki/nw/node/node-cert.pem]
(/var/lib/netwitness/config-management/cache/cookbooks/nw-pki/resources/certificate.rb line 266) had an error: Mixlib::ShellOut::ShellCommandFailed: Command execution failed.
STDOUT/STDERR suppressed for sensitive resource.

Cause

This issue arises in 12.x versions as these versions are not delta versions and do not pull in the previous version repositories and documents. This will be addressed in future releases.

Resolution

Before proceeding, ensure a backup of the MongoDB data is performed using mongoexport and mongodump. This is crucial to avoid data loss in case of any issues during the update process.

Add the missing version documents in MongoDB - database = "configuration", collection = "collection" and documents with _id: "service" and _id: "userpref". For example, if you are trying to install a 12.1 OVA/ISO to 12.3 NwAdmin and the customer has not been on 12.1. The customer has 11.3, 11.4, 11.5, 11.6, 11.7, 12.2, and 12.3, so you need to add 12.1 using the following commands in the MongoDB database:

Follow these steps to add the missing version documents:

1. Create a backup using mongoexport and mongodump: 

   mongoexport --ssl --sslAllowInvalidHostnames -u deploy_admin --authenticationDatabase admin --db configuration --collection collection --out /root/collect.json

   Output: 

   2024-08-09T19:55:45.808+0000 WARNING: --sslAllowInvalidCertificates and --sslAllowInvalidHostnames are deprecated, please use --tlsInsecure instead
   Enter password:
   
   2024-08-09T19:55:47.257+0000 connected to: mongodb://localhost/
   2024-08-09T19:55:47.269+0000 exported 2 records

   mongodump --ssl --sslAllowInvalidHostnames -u deploy_admin --authenticationDatabase admin --db configuration --collection collection --out /root/mongodump_collection.out

   Output: 

   2024-08-09T19:57:43.081+0000 WARNING: --sslAllowInvalidCertificates and --sslAllowInvalidHostnames are deprecated, please use --tlsInsecure instead
   Enter password:
   
   2024-08-09T19:57:44.513+0000 writing configuration.collection to /root/mongodump_collection.out/configuration/collection.bson
   2024-08-09T19:57:44.516+0000 done dumping configuration.collection (2 documents)

2. Connect to MongoDB and switch to the configuration database (only on NwAdmin/Node-zero): 

   mongo admin -u deploy_admin
   > use configuration

3. Add version details for 12.1 using the update command: 

   db.collection.update({"_id":"service"}, { $set: { "versions" : [{ "major" : 11, "minor" : 3}, { "major" : 11, "minor" : 4}, { "major" : 11, "minor" : 5}, { "major" : 11, "minor" : 6}, { "major" : 11, "minor" : 7},{ "major" : 12, "minor" : 1}, { "major" : 12, "minor" : 2}, { "major" : 12, "minor" : 3} ]}})
   WriteResult({ "nMatched" : 1, "nUpserted" : 0, "nModified" : 1 })
   
   db.collection.update({"_id":"userpref"}, { $set: { "versions" : [{ "major" : 11, "minor" : 3}, { "major" : 11, "minor" : 4}, { "major" : 11, "minor" : 5}, { "major" : 11, "minor" : 6}, { "major" : 11, "minor" : 7},{ "major" : 12, "minor" : 1}, { "major" : 12, "minor" : 2}, { "major" : 12, "minor" : 3} ]}})
   WriteResult({ "nMatched" : 1, "nUpserted" : 0, "nModified" : 1 })

4. Verify that the documents look like: 

   db.collection.find()
   { "_id" : "service", "versions" : [ { "major" : 11, "minor" : 3 }, { "major" : 11, "minor" : 4 }, { "major" : 11, "minor" : 5 }, { "major" : 11, "minor" : 6 }, { "major" : 11, "minor" : 7 }, { "major" : 12, "minor" : 1 }, { "major" : 12, "minor" : 2 }, { "major" : 12, "minor" : 3 } ], "_class" : "com.rsa.asoc.config.server.storage.repo.CollectionEntity" }
   
   { "_id" : "userpref", "versions" : [ { "major" : 11, "minor" : 3 }, { "major" : 11, "minor" : 4 }, { "major" : 11, "minor" : 5 }, { "major" : 11, "minor" : 6 }, { "major" : 11, "minor" : 7 }, { "major" : 12, "minor" : 1 }, { "major" : 12, "minor" : 2 }, { "major" : 12, "minor" : 3 } ], "_class" : "com.rsa.asoc.config.server.storage.repo.CollectionEntity" }

5. Retrigger the installation from the UI.

Notes

• NwAdmin: AdminServer
• NodeX: All NetWitness Components except Admin Server
• MongoDB Commands: Run only on NwAdmin/Node-zero