New HotFix: Addresses Kernel Panic After Upgrading to 12.4.1

[Will_G]

A new issue has been discovered where some devices will experience a kernel panic after upgrading to 12.4.1.  This is due to kernel updates failing to generate the required initramfs and vmlinuz binaries which are the equivalent of the newest kernel.  A kernel panic can occur if the node is rebooted without these files.

A hot fix is available. We recommend you upgrade to this hotfix after your 12.4.1.0 upgrade.

Below are additional steps to prevent the issue.

If you have any questions, please contact NetWitness Technical Support

Prevention Method:

1. Before rebooting the node after a successful upgrade, inspect the boot files
1. [ ls -l /boot/vmlinuz-* /boot/initramfs-* ]
2. If the initramfs file is missing for the newly installed kernel, execute the following command to regenerate them

1. dracut -f --regenerate-all && grub2-mkconfig -o /boot/grub2/grub.cfg

3. Confirm the grub config is updated appropriately using following command

1. cat /boot/grub2/grub.cfg | grep -A1 " BEGIN /etc/grub.d/10_linux"

If node is already experiencing Kernel Panic State, here are the ways to recover it with Zero Data/Config Loss

1. Download AlmaLinux-8-latest-x86_64-Live-GNOME.iso from Index of /almalinux/8/live/x86_64/ or from https://nw-fs.tier2.reston.netwitness.com/tools/Platform/alma8/LiveOS/
2. Mount it as a USB boot stick or Using “Virtual CD/DVD/ISO" | Virtual Media option using the iDRAC console.
3. Once the media is attached using either iDRAC or USB Boot stick, boot into it using steps in  Support Live Image Version 3.0 User's Guide | Dell India .
4. Boot into the AlmaLinux Live ISO
• Select “Start AlmaLinux Live 8.10“ and click Enter

5. Open the terminal and run the following commands to mount the filesystems from NW host to live system
• Run lsblk to check the disks
• Assuming /dev/sda1 is the /boot partition and /dev/sda2 is the netwitness_vg00 volume, run the following commands

In case of hybrids this might change to /boot in /dev/sda2 and netwitness_vg00 vg in /dev/sda3.

• mkdir /mnt/sysimage
• mount /dev/sda2 /mnt/sysimage
• ls /mnt/sysimage

                           Check required output should be similar to this. Else try mounting /dev/sda3 and check the files.

• mount /dev/sda1 /mnt/sysimage/boot
• ls /mnt/sysimage/boot

          The output should be similar to this, else try with mounting /dev/sda2

• mount --bind /proc /mnt/sysimage/proc
• mount --bind /dev /mnt/sysimage/dev
• mount --bind /sys /mnt/sysimage/sys
• chroot /mnt/sysimage
• Regenerate dracut files using - dracut -f --regenerate-all -v
• Ensure that all the required boot files are created in /boot folder - initramfs, vmlinuz*, config* , System.map* and symvers* files.
• Update the grub conf using - grub2-mkconfig -o /boot/grub2/grub.cfg
• exit
• umount /mnt/sysimage/boot
• umount /mnt/sysimage
5. Reboot the box
6. Boot in with default Hard Disk:C or “Integrated RAID Controller 1 - PERC 740P Mini“ [For an Series6 - PowerEdge R640/740xd appliance] and with “Integrated RAID Controller 1 - PERC 730P Mini“ [For an Series5 - PowerEdge R630/730xd appliance]
7. Host should boot in now without kernel panic as required boot files are re-created.