Pre Upgrade Checks for 12.4.1.0

[#1]

Run Pre-Upgrade Checks

NetWitness strongly recommends that you run the pre-upgrade checks before you upgrade to NetWitness Platform 12.4.1.0 to identify any issues that may result in upgrade failure.

Note:
- Ensure that the deploy_admin password is valid and up to date. Do not change it from the UI. Use https://community.netwitness.com/t5/netwitness-platform-online/manage-the-deploy-admin-account/ta-p/669802 to change the password.
- If any probe fails when you run the pre-upgrade checks, refer https://community.netwitness.com/t5/netwitness-knowledge-base/netwitness-pre-upgrade-check-known-issues-master-list/ta-p/708695 to mitigate the issue associated with the failed probe.

Before you begin

You must first download the Standalone RPM using https://community.netwitness.com/t5/netwitness-platform-downloads/netwitness-platform-standalone-precheck-tool/ta-p/709096 and refer to the read me file for instructions on how to install the Standalone RPM and then run the pre-check.

To run the pre-upgrade checks:

1. SSH to Admin Server.

2. Using the Upgrade Precheck tool, run the following commands in sequence:

   1. nw-precheck-tool-standalone os-migration-checklist: This command allows the Upgrade Precheck tool to perform sanity checks for the list of probes in the OS Migration Checklist.

   2. nw-precheck-tool-standalone upgrade-checklist: This command allows the Upgrade Precheck tool to perform sanity checks for the list of probes in the Upgrade Checklist.

   3. nw-precheck-tool-standalone network-checklist: This command allows the Upgrade Precheck tool to perform sanity checks for the list of probes in the Network Checklist.

   4. nw-precheck-tool-standalone cert-checklist: This command allows the Upgrade Precheck tool to perform sanity checks for the list of probes in the Certificate Checklist.

   OS Migration Checklist

   The Upgrade Precheck tool performs the sanity checks for the following list of probes in the OS Migration checklist:

   • Version Check Probe: Checks whether the NetWitness version of the system is the later version of 12.2.0.0 or not.

   • AVX / VMX Probe: Checks if the AVX / VMX flags are enabled or not for the nodes that require them.

   • NFS Mount Probe: Checks if NFS type mount point is active on any of the nodes.

   • Multiple kernel-devel Package Probe: Checks if Decoder and PacketHybrid have multiple versions of kernel-devel package or not.

   • PF Ring Capture Device Probe: Checks for PF_ring capture device on decoders and raises a warning to change PF_ring capture device to DPDK capture device.

   • BTRFS mount Probe: Check if BTRFS partition is mounted.

     Note: LEAPP and Alma OS doesn't support BTRFS partition.

   • Disk space check: Checks to ensure that enough disk is free in the / partition on each node.

   • Fips Mode Check: Checks to ensure that the Fips mode is disabled (set to false) on all nodes.

   • Mountcheck probe: Checks if all the partitions or file directories are mounted properly.

   • Salt Master Communication Check: Verifies the salt communication from Node 0 to all the Nodes.

   • CIFS Check Probe: Identifies any CIFS file systems on the hosts which are unsupported for in place upgrade.

   • XFS Ftype Check Probe: Checks if there are any XFS file systems with ftype set to 0 on the host.

   • CCM Custom Content Check Probe: Checks if .envision files are being extracted in their proper locations.

   • Optical Drive Check Probe: Checks if any optical device is mounted on any hardware device. If any optical device is detected, the probe flags the user to unmount it.

   Upgrade Checklist

   The Upgrade Precheck tool performs the sanity checks for the following list of probes in the upgrade checklist:

• Security Client File Check: Ensures security-client-amqp.yml file is not present.

• Node-0 NW Service-id Status Check: Ensures all the service-id is intact with all the different services in Node 0.

• Broker Service Trustpeer Symlink File Check: Ensures Broker Service Trustpeer Symlink file (/etc/netwitness/ng/broker/trustpeers/) is not broken.

• Node-0 NW Services Status Check: Checks the status of all the services in Node 0.

• Yum External Repo Check: Ensures external repos are not available and not enabled.

• Node-0 RPM DB Index Check: Checks if the RPM DB is corrupted or not.

• Salt Master Communication Check: Verifies the salt communication from Node 0 to all the Nodes.

• Node-0 Certificates Check: Checks if any certificates are missing, expired, or invalid issuer type.

• Mongo Authentication: Validates the deploy_admin credentials fetched from security-cli-client using Mongo client.

• Rabbitmq Authentication: Validates the deploy_admin credentials fetched from security-cli-client using RabbitMQ.

• (Component Hosts) Node X NW Service Status Check: Verifies the status of services (Active or Inactive) on all the Node X.

• (Component Hosts) Node X Certificates Check: Checks the certificate expiry, missing, corrupted, and issuer mismatch in all the categories of Node X.

• Provide Nodes CPU-Memory Info: Provides CPU and Memory details of all the nodes along with the real-time available memory.

• (Admin Server) Node 0 File System Utilization Check: Verifies the disk partition utilization of /var/netwitness/mongo, /var/netwitness, and root on Node 0.

• (Component Hosts) Node X File System Utilization Check: Verifies the disk partition utilization of /var/netwitness/mongo, /var/netwitness, and root for ESA Primary and Endpoint Log Hybrid services on Node X.

• Mongo File (ESAPrimary) Permission Mode Check: Checks the ESA Primary node in the system or stack and verifies the permission mode of Mongo file.

• Orchestration Server Normal Mode Check: Checks if the orchestration service is running in normal or safe mode.

• (Admin Server) Node 0 Init status Check: Checks if there are any issues that might fail init process.

• Fips Mode Check: Checks to ensure that the Fips mode is disabled (set to false) before and after upgrade.

• Node-X RPM DB Index Check: Checks for the status of RPM DB on Node-X to make sure it is not corrupted.

• Node-Z Yum Proxy Check: Checks for the existence of yum.conf file and availability of proxy within the file on Node -Z.

• Node-X Yum Proxy Check: Checks for the existence of yum.conf file and availability of proxy within the file on Node -X.

• Host Info Check Probe: Checks if the required fields of information of all the hosts in the system (Host IP, Hostname, Installed Services, and Raw Version) are available.

• Node-Z Cipher Check Probe: Checks if the required ciphers are available in the location /etc/rabbitmq/rabbitmq.config on Node-0.

• Node-X Cipher Check Probe: Checks if the required ciphers are available in the location /etc/rabbitmq/rabbitmq.config on all Node-X.

• Node-X Hardware Version Check Probe: Checks for the hardware version of all reachable Node-X.

• Node-Z Hardware Version Check Probe: Checks for the hardware version of the Admin server.

• PuppetCA Certificates Check Probe: Checks if the stale puppet CA certificates are present in the location /etc/pki/nw/trust/truststore.pem.

• AdminCertCheck Probe: Verifies if the admin-certs across all the nodes are the same as the admin-certs on the Admin Server.

• NTP Probe: Checks all the nodes to ensure they are in sync with the NTP server.

• StaleCerts Check Probe: Checks the mongo and warns if there are any unused stale certificates in it.

• NodeCertIDCheck Probe: Checks the subject field of the node-cert and ensures that it is the same as the node-ID of the host.

• Deploy Admin password expiry check Probe: Verifies if the deploy_admin password is expired on Node-0.

• File / Folder permission check: This probe checks if the files / folders have the appropriate permissions.

Network Checklist

The Upgrade Precheck tool performs the sanity checks for the following list of probes in the network checklist:

• (Admin Server) Node 0 closed ports Check: Checks if the service ports required for NetWitness services are open and listening on Node 0.

• (Component Hosts) Node X closed ports Check: Checks if the service ports required for NetWitness services are open and listening on Node X.

Certificate Checklist

The Upgrade Precheck tool performs the sanity checks for the following list of probes in the Certificate checklist:

• Node 0 Service Certificates Validity Check: Checks the validity of service certificates in the location /etc/pki/nw/service/ on Node-0.

• Node X Service Certificates Validity Check: Checks the validity of service certificates in the location /etc/pki/nw/service/ on Node-X.

• Node Certificates Validity Check on Node-0: Checks the validity of node certificates in the location /etc/pki/nw/service on Node-0.

• Node Certificates Validity Check on Node-X: Checks the validity of node certificates in the location /etc/pki/nw/service on Node-X.

• Root CA Certificates Validity Check: Checks the validity of Root CA certificates in the location /etc/pki/nw/ca.