Upgrade Preparation Tasks for 12.5.1.0

[#1]

Prepare to Upgrade NetWitness Platform

Complete the following tasks to prepare for the upgrade to NetWitness Platform 12.5.1.0

Task 1. (Important) Prepare to Upgrade AlmaLinux OS

Unsupported File System

Unmount and Remove BTRFS

BTRFS is a copy-on-write (CoW) filesystem for Linux that aims to implement advanced filesystem features while focusing on fault tolerance, repair, and easy administration. The BTRFS file system is deprecated from Red Hat Enterprise Linux 8, and AlmaLinux OS does not support the BTRFS file system. NetWitness does not use BTRFS by default, but in some categories like network decoder, network hybrid, etc., BTRFS module exists and is loaded. If BTRFS is mounted as a filesystem, perform the below steps to unmount the BTRFS partition manually (If BTRFS is not mounted, skip the below steps):

1. Relocate the data.

2. Unmount the BTRFS partition using the following command.

   umount <btrfs partition path>. You can get the btrfs partition info from /etc/fstab or df -hT commands.

3. Remove the BTRFS partition from /etc/fstab.

4. Verify if the kernel moduleis still loaded using lsmod | grep btrfs. If the kernel module is still loaded , use modprobe -r btrfs to unload the btrfs kernel module.

5. Trigger/Retrigger the Upgrade.

For more information, refer the KB article How to check Upgrade Fails and Data in Partition is Lost when BTRFS is Mounted or Loaded.

Unmount NFS

NFS type file systems active on the nodes cause the upgrade to fail for the nodes. You should manually unmount these mount points from the CLI of each node where it is found. Perform the below steps to unmount the NFS manually:

1. SSH to the nodes where NFS mount point is detected.

2. In each node, run mount | grep 'type nfs' and get the directory path of the NFS mount point.

   Note: Before unmounting NFS, you must stop the NetWitness services that rely upon NFS. For example: If Archiver and Warehouse Connector service is running on NFS, you must run the following commands to stop the services before unmounting NFS.
   systemctl stop nwarchiver
   systemctl stop nwwarehouseconnector
   

3. Execute umount <dir_path> from the terminal, where <dir_path> is the directory path from Step b.

4. Open /etc/fstab file in an editor of choice and comment out the line(s) that pertain to NFS mount point(s).

5. Run the NetWitness upgrade.

6. After upgrade has completed successfully, un-comment the respective entry from /etc/fstab and execute mount -a from the terminal to add the NFS mount point(s) back.

AVX/VMX CPU Instruction Set Check

AVX/VMX CPU flag needs to be enabled for NetWitness Platform 12.5.1.0. Run the following command:
salt '*' cmd.run "lscpu | grep -E 'avx|vmx'" to check if AVX/VMX CPU instruction set is enabled. Refer the KB Article NetWitness MongoDB 5.0 uses AVX Intstruction set for mongod.mongos and legacy mongo shell for more information.

Note: For NetWitness hardware appliance, AVX/VMX CPU instruction set is enabled by default.

PF_RING to DPDK Migration Support

The decoder capture config will not be valid for customers using PF_RING capture (CentOS) and directly upgrading to 12.5.1.0 (AlmaLinux). First, they must migrate PF_RING devices to DPDK and then upgrade.

Refer to Migrate PF_RING Devices to DPDK for migration instructions.

Task 2. (Recommended but optional). Remove Legacy Package Repositories

You can free up the disk space by removing obsolete repositories from previous releases.

To remove the obsolete repositories

1. Determine the version of the oldest NetWitness Platform host in your environment by using the NetWitness Repo tool. Do the following:

   • SSH to the Admin Server as a root user.

   • Run the following command:

     nw-repo-tool --list-obsolete

     After running this command, you will get a list of all the obsolete repositories.

2. Run the following command to remove all the obsolete repositories.

   nw-repo-tool --purge-obsolete

Task 3. Prepare ESA Deployments for Migration to 12.5.1.0

Before upgrading to 12.5.1.0, NetWitness recommends that all the ESA deployments maintain an error-free state. You must remove any unused ESA deployments, as ESA deployments will be migrated to policies and groups after upgrading to 12.5.1.0. Each deployment will be converted into a policy and group and will be available to manage only after the upgrade of the Correlation servers to the 12.5.1.0 version.

Manage ESA Deployments and Data Sources

You can only manage the ESA deployments and Data Sources through Centralized Content Management.

Go to  (CONFIGURE) > Policies > Content > Event Stream Analysis page to manage the ESA deployments and Data Sources. You can only manage the ESA Rules in the ESA Rules page. Refer the following figures.

Make sure that you plan the upgrade process so that Correlation servers are upgraded immediately after the Admin Server is done. The deployments will not be accessible until the corresponding Correlation servers are upgraded. However, the correlation servers will still continue to process the Alerts and Events. You must upgrade the ESA hosts immediately after upgrading the Admin Server.

For more information on Centralized Content Management and managing the deployments, see Centralized Content Management Guide for NetWitness.

Important: If there is any need to import ESA Rules and enrichments, NetWitness recommends importing those missing rules and enrichments before the upgrade.

The pre-upgrade and post-upgrade states of deployments are represented in the following table.

SlNo	Pre-upgrade Deployment State	Post-upgrade Deployment State
		Creates Policy	Creates Group	The policy will be Published
1	Healthy deployment	Yes	Yes	Yes
2	Deployment with errors	Yes	Yes	Yes
3	Deployment with only rules	Yes	No	No
4	Deployment with no rules	No	No	No

Healthy deployment contains no errors, and the required resources such as ESA Server, Data source, and ESA rules are added.

Note: NetWitness recommends that all the deployments maintain an error-free state. You must remove any unnecessary or unused ESA deployments.

Task 4. Third-Party Package Removal

Any third-party packages installed on hosts out of the NetWitness repository are subjected to removal based on the Upgrade dependencies as part of OS migration.

Task 5. Single Sign-on (SSO): Enable SAML Response Signing in Microsoft Azure ADFS

The following configuration is only applicable to cases where the SAML response from Microsoft Azure ADFS was only encrypted by not signed. If your Microsoft Azure ADFS is already configured to sign and encrypt SAML responses, you can ignore this configuration and proceed with the upgrade process.

If you are not signing the SAML response, NetWitness recommends you to configure Microsoft Azure ADFS to encrypt and sign the SAML responses before upgrading your NetWitness Platform to version 12.5.1.0 for a successful Single Sign-on (SSO) login. To enable response signing in Active Directory Federation Service (AD FS), run the following command in powershell:

Set-AdfsRelyingPartyTrust -TargetName <<relying-party-name>> -SamlResponseSignature MessageAndAssertion

Important: It is mandatory to configure Microsoft Azure ADFS to sign SAML responses before upgrading to version 12.5.1.0 of the NetWitness Platform. Without complying with these requirements, you may not be able to log in using SSO.

Task 6. (Optional). Disable STIG-based FIPS Kernel Controls

If you enabled STIG-based FIPS Kernel controls, you must disable them before initiating the NetWitness Platform upgrade process to avoid boot errors. To disable STIG-based FIPS Kernel controls, run the following commands:

manage-stig-controls --disable-control-groups 3 --host-all

grub2-mkconfig -o /boot/grub2/grub.cfg

After you upgrade NetWitness Platform, ensure that you enable STIG-based FIPS Kernel controls.

Note: STIG-based FIPS Kernel controls which require modifications to kernel boot options are not enabled by NetWitness out-of-the-box.

Task 7. (Optional). Verify Connection for Live Server

Note: This optional task is applicable to you only if you are upgrading NetWitness Platform through live.

Go to admin/system/live services and do a test connection to verify if you are able to connect to the live server as this is essential for the source-server from 12.x and above. This is an optional step and applicable only for customers who have configured live.

Task 8. Synchronize Time on Component Hosts with NW Server Host

Before you upgrade hosts, make sure that the time on each host is synchronized with the time on the NetWitness Server.

The NTP time synchronization method depends on your NetWitness Platform version.

• For NetWitness versions lower than 12.4: Use ntpd for time synchronization.

• For NetWitness versions 12.4 and later: Use chrony for time synchronization.

To synchronize the time, do one of the following:

1. Configure the NTP Server.

   For more information, see Configure NTP Servers topic in the System Configuration Guide.

2. Perform the following steps:

   1. SSH to the Admin Server host.

   2. Run the following commands based on your NetWitness version:

      • If the NetWitness version is lower than 12.4. Follow these steps:

        Important: To ensure correct time synchronization, ntpd must be running on node0. Shutting down ntpd on node0 will cause the ntpdate commands on other nodes to fail.

        • salt \* service.stop ntpd

        • salt \* cmd.run 'ntpdate nw-node-zero'

        • salt \* service.start ntpd

      • If the NetWitness version is 12.4 or later. Follow these steps:

        • salt \* service.stop chronyd

        • salt \* cmd.run "chronyc makestep"

        • salt \* service.start chronyd