Health & Wellness (rsa-sms) Refuses to Start Post-Upgrade

[#1]

Article Number

000003242

Applies To

NetWitness Product Set: NetWitness Platform
NetWitness Product/Service Type: Node-Zero (Admin Server), Health & Wellness (legacy)
NetWitness Version/Condition: 12.4.x
Platform: AlmaOS

Issue

After an upgrade (and potentially in other situations), the Legacy Health & Wellness service refuses to start. The symptoms can include:

• after running a 'systemctl restart rsa-sms' and then 'systemctl status rsa-sms', you see the service immediately fails

[root@NW-NODE-ZERO home]# systemctl status rsa-sms
● rsa-sms.service - RSA NetWitness SMS :: Server
   Loaded: loaded (/usr/lib/systemd/system/rsa-sms.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Fri 2024-10-11 18:39:36 UTC; 321ms ago
  Process: 2266368 ExecStart=/opt/rsa/sms/bin/sms start (code=exited, status=0/SUCCESS)
 Main PID: 2266422 (code=exited, status=1/FAILURE)

Oct 11 18:39:30 NW-NODE-ZERO systemd[1]: Starting RSA NetWitness SMS :: Server...
Oct 11 18:39:31 NW-NODE-ZERO sms[2266373]: Starting RSA NetWitness SMS :: Server...
Oct 11 18:39:31 NW-NODE-ZERO systemd[1]: Started RSA NetWitness SMS :: Server.
Oct 11 18:39:36 NW-NODE-ZERO systemd[1]: rsa-sms.service: Main process exited, code=exited, status=1/FAILURE
Oct 11 18:39:36 NW-NODE-ZERO systemd[1]: rsa-sms.service: Failed with result 'exit-code'.

• When this condition occurs, the service can potentially crash before it even attempts to write logs to : /opt/rsa/sms/logs/sms.log
• Attempting to run the service from the command line in console mode fails with (see example syntax in BOLD to run rsa-sms in the console mode): /opt/rsa/sms/bin/sms console

[root@NW-NODE-ZERO home]# systemctl stop rsa-sms
[root@NW-NODE-ZERO home]# /opt/rsa/sms/bin/sms console
Running RSA NetWitness SMS :: Server...
wrapper  | --> Wrapper Started as Console
wrapper  | Launching a JVM...
jvm 1    | Wrapper (Version 3.2.3) http://wrapper.tanukisoftware.org
jvm 1    |   Copyright 1999-2006 Tanuki Software, Inc.  All Rights Reserved.
jvm 1    | 
jvm 1    | 2024-10-11 18:39:58,133 [WrapperSimpleAppMain] WARN  com.rsa.netwitness.carlos.i18n.MessageFactory - Can't find bundle for base name com.rsa.netwitness.sms.common.util.SystemUtilsMessages, locale en_US; fabricating bundle dynamically.
jvm 1    | 2024-10-11 18:39:58,212 [WrapperSimpleAppMain] INFO  com.rsa.netwitness.carlos.common.ApplicationSupport - Lockbox home :  /root/rsa/home
jvm 1    | 2024-10-11 18:39:58,238 [WrapperSimpleAppMain] INFO  com.rsa.netwitness.carlos.lockbox.CarlosLockBoxFactory - A BSafe implementation of secure store found at path /root/rsa/home/lockbox.ss, this will be migrated to new secret store
jvm 1    | 2024-10-11 18:39:59,667 [WrapperSimpleAppMain] ERROR com.rsa.netwitness.carlos.common.ApplicationSupport - Failed to initialize Carlos Service Secure Store
jvm 1    | com.rsa.netwitness.carlos.lockbox.SecureStoreException: Ssv values have changed
jvm 1    | 	at com.rsa.netwitness.carlos.lockbox.CarlosLockBoxFactory.defaultLockBox(CarlosLockBoxFactory.java:114)

 

Cause

The lockbox files located in /root/rsa/home/ were either corrupted or not properly updated during the upgrade.

Resolution

The resolution is as follows:

• Stop the rsa-sms service:

  •  systemctl stop rsa-sms

• Backup and remove the files under /root/rsa/home/*

  • mkdir /root/rsa-sms-lockbox-backup/ && mv /root/rsa/home/* /root/rsa-sms-lockbox-backup/

• Start the rsa-sms service: 

  • systemctl start rsa-sms

• Check the status after a few minutes to ensure the service remains Started and Active:

  • systemctl status rsa-sms

Workaround

Alternatively, you can run this single command in an attempt to restore the rsa-sms state:

 systemctl stop rsa-sms && mkdir /root/rsa-sms-lockbox-backup/ && mv /root/rsa/home/* /root/rsa-sms-lockbox-backup/ && systemctl start rsa-sms && systemctl status rsa-sms

Example Output:
 

root@NW-NODE-ZERO home]# systemctl stop rsa-sms && mkdir /root/rsa-sms-lockbox-backup/ && mv /root/rsa/home/* /root/rsa-sms-lockbox-backup/ && systemctl start rsa-sms && systemctl status rsa-sms
● rsa-sms.service - RSA NetWitness SMS :: Server
   Loaded: loaded (/usr/lib/systemd/system/rsa-sms.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2024-10-11 18:42:09 UTC; 36ms ago
  Process: 2267324 ExecStart=/opt/rsa/sms/bin/sms start (code=exited, status=0/SUCCESS)
 Main PID: 2267378 (wrapper-linux-x)
    Tasks: 22 (limit: 263538)
   Memory: 35.5M
   CGroup: /system.slice/rsa-sms.service
           ├─2267378 /opt/rsa/sms/bin/./wrapper-linux-x86-64 /opt/rsa/sms/conf/wrapper.conf wrapper.syslog.ident=sms-driver wrapper.pidfile=/opt/rsa/sms/logs>
           └─2267393 java -Xmx16g -XX:+UseG1GC -Djavax.net.ssl.keyStore=/etc/pki/nw/carlos/rsa-nw-sms-server.p12 -Dclover.initstring=/tmp/clover/clover.db -D>

Oct 11 18:42:09 NW-NODE-ZERO systemd[1]: Starting RSA NetWitness SMS :: Server...
Oct 11 18:42:09 NW-NODE-ZERO sms[2267329]: Starting RSA NetWitness SMS :: Server...
Oct 11 18:42:09 NW-NODE-ZERO systemd[1]: Started RSA NetWitness SMS :: Server.

 

Notes

This has been observed on upgrades from 12.2.x to 12.4.2, but may occur in other scenarios as well. If the messages when running rsa-sms in console mode show something similar to "Ssv values have changed", then this workaround will likely fix it.