Critical Advisory: Preventative Steps for Kernel Update Issue & HotFix

[#1]

Summary

A new issue has been discovered where some devices will experience a kernel panic after upgrading to 12.4.1. This is due to kernel updates failing to generate the required initramfs and vmlinuz binaries which are the equivalent of the newest kernel. A kernel panic can occur if the node is rebooted without these files.

A HotFix is available. We recommend you upgrade to this HotFix considering the relevant scenario. Below are various scenarios to apply, prevent, or recover from the issue.

You can contact NetWitness Customer Support to download and apply this HotFix. You can also download the attached HotFix Release Notes to learn more about this HotFix and how to apply it.

For additional documentation, downloads, and more, visit the NetWitness Platform page.

Affected Products

Upgrades from NetWitness Platform 12.4.0.0 to NetWitness Platform 12.4.1.0

Recommendation

Follow the steps below based on the scenario that applies.

Important Note:

• You can upgrade to this HotFix only using the CLI mode. You cannot use online/offline smc mode for applying this HotFix.
• This HotFix is only valid for upgrading the NetWitness platform from 12.4.0.0 to 12.4.1.0. Users are not required to use it to perform an upgrade from a version prior to 12.4.0.0.

Scenario 1: Upgrade to NetWitness 12.4.1.0 and Apply HotFix

This section applies if you have not done an upgrade to 12.4.1.0 and provides the procedure to upgrade to NetWitness Platform 12.4.1.0 and apply the HotFix. You can upgrade with this HotFix only in CLI mode.
Note: This HotFix is only valid for upgrading the NetWitness Platform from 12.4.0.0 to 12.4.1.0. You cannot use it to upgrade from a version prior to 12.4.0.0.

To install Config Management RPM

1. Run the upgrade initialization using the following command:

        upgrade-cli-client --init --version 12.4.1.0 --stage-dir <path of the staging directory>

2. Once initialization is complete, download or copy the following config-management rpm to Admin Server:

        rsa-nw-config-management-12.4.1.0-2407121516.5.76e0414.el8.noarch.rpm

3. Place the config-management rpm in the following directory:

          cd /var/netwitness/common/repo

4. Using salt, run the following command and download config-management rpm on all connected NW hosts:

         salt "*" cmd.run "wget --no-check-certificate https://nw-node-zero/nwrpmrepo/rsa-nw-config-management-12.4.1.0-2407121516.5.76e0414.el8.noarch.rpm" >> cfg-mgmt_download.out

5. Using salt, run the following command and upgrade the updated config-management rpm on all connected NW hosts:

       salt '*' cmd.run "yum update -y rsa-nw-config-management-12.4.1.0-2407121516.5.76e0414.el8.noarch.rpm" >> cfg-mgmt_install.out

6. Using salt, run the following command and check the installed config-management package version:

           salt '*' cmd.run "rpm -q rsa-nw-config-management" >> cfg-mgmt_version.out

7. Proceed to upgrade the Admin Server and other Node-X NW hosts.

Scenario 2: [Mixed Mode] A Few Node-X Are Upgraded to 12.4.1 and Recovered from the Kernel-Panic Issue. Follow These Steps to Prevent Kernel-Panic for Other Node-X

You can use this procedure to upgrade Node-x without getting the kernel-panic issue.

To upgrade Node-x

1. SSH to the Admin Server.
2. Download or copy the following config-management rpm to Admin Server:

        rsa-nw-config-management-12.4.1.0-2407121516.5.76e0414.el8.noarch.rpm

3. Place the config-management rpm in the following directory:

          cd /var/netwitness/common/repo

4. Using salt, run the following command and download the config-management rpm on all connected NW hosts:

    salt "*" cmd.run "wget --no-check-certificate https://nw-node-zero/nwrpmrepo/rsa-nw-config-management-12.4.1.0-2407121516.5.76e0414.el8.noarch.rpm" >> cfg-mgmt_download.out

5. Using salt, run the following command and upgrade the updated config-management rpm on all connected NW Node-X hosts:

        salt '*' cmd.run "yum update -y rsa-nw-config-management-12.4.1.0-2407121516.5.76e0414.el8.noarch.rpm" >> cfg-mgmt_install.out

6. Using salt, run the following command and check the installed config-management package version:

         salt '*' cmd.run "rpm -q rsa-nw-config-management" >> cfg-mgmt_version.out

7. Proceed to upgrade the other Node-X NW hosts.

Scenario 3: [Mixed Mode] AdminServer and Few Node-X Are Upgraded to 12.4.1 and Initramfs files are Not Generated / NOT Rebooted. Follow These Steps to Regenerate Required Files for Upgraded hosts and Prevent Kernel-Panic for Other Node-X

You can use this procedure to regenerate initramfs files on upgraded systems and to upgrade Node-x without getting the kernel-panic issue.

To upgrade Node-x

1. SSH to the Admin Server.
2. Download or copy the following config-management RPM to Admin Server:

        rsa-nw-config-management-12.4.1.0-2407121516.5.76e0414.el8.noarch.rpm

3. Place the config-management rpm in the following directory:

          cd /var/netwitness/common/repo

4. Using salt, run the following command and download the config-management rpm on all connected NW hosts:

         salt "*" cmd.run "wget --no-check-certificate https://nw-node-zero/nwrpmrepo/rsa-nw-config-management-12.4.1.0-2407121516.5.76e0414.el8.noarch.rpm" >> cfg-mgmt_download.out

5. Using salt, run the following command and upgrade the updated config-management rpm on all connected NW Node-X hosts:

     salt '*' cmd.run "yum update -y rsa-nw-config-management-12.4.1.0-2407121516.5.76e0414.el8.noarch.rpm" >> cfg-mgmt_install.out

6. Using salt, run the following command and check the installed config-management package version:

         salt '*' cmd.run "rpm -q rsa-nw-config-management" >> cfg-mgmt_version.out

7. Proceed to re-upgrade AdminServer and Node-X NW hosts to regenerate required initramfs files.

Disaster Recovery: Recover a Node-0 or Node-X if You Encounter Kernel- Panic State and Fail-to-Boot Error

This section provides the procedure to recover a Node-0 or Node-x if you encounter a Kernel-Panic state and fail-to-boot error.

To recover a Node-0 or Node-x if you enter the kernel-panic state

1. Do one of the following:
   • Contact the NetWitness Support and get the AlmaLinux-8-latest-x86_64-Live-GNOME.iso file.
   • If you are open to downloading directly from the official Alma source, go to the following link and download the ISO file: https://repo.almalinux.org/almalinux/8/live/x86_64/

2. Mount it as a USB boot stick or Virtual CD/DVD/ISO using the iDRAC console's Virtual Media option.
3. Once the media is attached using either iDRAC or USB Boot stick, boot into it using the steps in Support Live Image Version 3.0 User's Guide Dell .
4. Boot into the AlmaLinux Live ISO:

             a. Select Start AlmaLinux Live 8.10 and click Enter.

                  

5. Open the terminal and run the following commands to mount the filesystems from the NW host to the Live system:

   >sudo -i

• Run lsblk to check the disks.
• Run vgscan and lvscan to list volume groups and logical volumes.
• Run vgchange -ay netwitness_vg00 to activate netwitness_vg00 volume group
• Assuming /dev/sda1 is the /boot partition and /dev/sda2 is the netwitness_vg00 volume, run the following commands:
  • In case of hybrids, this might change to /boot in /dev/sda2 and netwitness_vg00 vg in /dev/sda3.
  • mkdir /mnt/sysimage
  • mount /dev/sda2 /mnt/sysimage
  • ls /mnt/sysimage
  • mount /dev/sda1 /mnt/sysimage/boot
  • ls /mnt/sysimage/boot
  • mount --bind /proc /mnt/sysimage/proc
  • mount --bind /dev /mnt/sysimage/dev
  • mount --bind /sys /mnt/sysimage/sys
  • chroot /mnt/sysimage
  • Regenerate dracut files using - dracut -f --regenerate-all -v
  • Ensure that all the required boot files are created in /boot folder - initramfs, vmlinuz*, config* , System.map* and symvers* files.
  • Update the grub conf using - grub2-mkconfig -o /boot/grub2/grub.cfg
  • exit
  • umount /mnt/sysimage/boot
  • umount /mnt/sysimage/proc
  • umount /mnt/sysimgae/dev
  • umount /mnt/sysimage/sys
  • umount /mnt/sysimgae

6. Reboot the box.
7. Boot in with default Hard Disk: C or 'Integrated RAID Controller 1 - PERC 740P Mini' [For a Series6 - PowerEdge R640/740xd appliance] and with 'Integrated RAID Controller 1 - PERC 730P Mini' [For a Series5 - PowerEdge R630/730xd appliance].
8. Host should boot in now without kernel panic as required boot files are re-created.