Netwitness NodeX host services install fails due to invalid certificates.

[#1]

Article Number

000002107

Applies To

RSA Product Set: NetWitness Platform
RSA Product/Service Type: Core Appliance
RSA Version/Condition: 11.x
Platform: CentOS 7

Issue

The NodeX host services install fails with the below error.

/var/log/netwitness/config-management/chef-solo.log:

[2022-09-02T14:49:42+00:00] ERROR: nw_base_nwconsole[refresh-nw-concentrator-trust-peers] (nw-concentrator::trusts line 31) had an error: Mixlib::ShellOut::ShellCommandFailed: execute[list-peercerts] (/var/lib/netwitness/config-management/cache/cookbooks/nw-base/resources/nwconsole.rb line 134) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of NwConsole -k -c "tlogin server=localhost port=56005 username=admin group=Administrators cert=/etc/pki/nw/node/node-cert.pem key=/etc/pki/nw/node/node-key.pem" -c "send /sys peerCert op=list " -q ----
STDOUT: Could not create trusted session: server could not validate and trust our certificate
STDERR:
---- End output of NwConsole -k -c "tlogin server=localhost port=56005 username=admin group=Administrators cert=/etc/pki/nw/node/node-cert.pem key=/etc/pki/nw/node/node-key.pem" -c "send /sys peerCert op=list " -q ----
Ran NwConsole -k -c "tlogin server=localhost port=56005 username=admin group=Administrators cert=/etc/pki/nw/node/node-cert.pem key=/etc/pki/nw/node/node-key.pem" -c "send /sys peerCert op=list " -q returned 1; ignore_failure is set, continuing
[2022-09-02T14:49:42+00:00] INFO: Processing systemd_service_drop_in[nwconcentrator-opts-managed] action create (nw-concentrator::services line 12)
[2022-09-02T14:49:42+00:00] INFO: Processing service[nwconcentrator] action enable (nw-concentrator::services line 22)
[2022-09-02T14:49:42+00:00] INFO: Processing service[nwconcentrator] action start (nw-concentrator::services line 22)
[2022-09-02T14:49:42+00:00] INFO: Processing log[Unable to find Common Node Certificate] action write (nw-concentrator::collectd line 12)
[2022-09-02T14:50:44+00:00] ERROR: Running exception handlers
[2022-09-02T14:50:44+00:00] ERROR: Exception handlers complete
[2022-09-02T14:50:44+00:00] FATAL: Stacktrace dumped to /var/lib/netwitness/config-management/cache/chef-stacktrace.out
[2022-09-02T14:50:44+00:00] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report
[2022-09-02T14:50:44+00:00] FATAL: NoMethodError: nw_base_nwconsole[set_pin_dir_and_size for nw-concentrator] (nw-concentrator::pin line 😎 had an error: NoMethodError: execute[configure_pin_dir] (/var/lib/netwitness/config-management/cache/cookbooks/nw-base/resources/nwconsole.rb line 214) had an error: NoMethodError: undefined method `split' for nil:NilClass

Cause

This issue is due to invalid certificates in NodeX host.

Note the NodeX UUID from the below command.
cat /etc/salt/minion 

From Admin Server, Please run the below command to test the validity NodeX certificates.

salt '<UUID>' cmd.shell runas=root cmd='hostname; NwConsole -q -k -c "tlogin server=localhost port=56005 username=admin group=Administrators cert=/etc/pki/nw/node/node-cert.pem key=/etc/pki/nw/node/node-key.pem" -c "send /sys peerCert op=list"'

Example:

[root@AdminServer ~]# salt '846c32b7-9dbc-445c-9cc8-4d6d696f2fab' cmd.shell runas=root cmd='hostname; NwConsole -q -k -c "tlogin server=localhost port=56005 username=admin group=Administrators cert=/etc/pki/nw/node/node-cert.pem key=/etc/pki/nw/node/node-key.pem" -c "send /sys peerCert op=list"'
846c32b7-9dbc-445c-9cc8-4d6d696f2fab:
    LogHYBRID
    Could not create trusted session: server could not validate and trust our certificate
    (F) 2022-Sep-02 15:34:14 [ChannelManager::messageHandler]  Socket Error: Operation canceled
ERROR: Minions returned with non-zero exit code

Resolution

Please follow the below steps to copy trusted certificates from AdminServer to NodeX.

• Take backup of existing certificates from NodeX host using the below commands.

mv /etc/pki/nw/node/node-cert.pem /root/
mv /etc/pki/nw/node/node-key.pem /root/

• Please download the below files from AdminServer and upload them to NodeX host in the same directories.

/etc/pki/nw/node/node-cert.pem
/etc/pki/nw/node/node-key.pem

• Change the ownership of the files to netwitness:nwpki using the below commands in NodeX host.

cd /etc/pki/nw/node/
chown netwitness:nwpki node-cert.pem
chown netwitness:nwpki node-key.pem

• Verify the validity of NodeX certificates by running the below command in Admin Server.

[root@AdminServer]# salt '846c32b7-9dbc-445c-9cc8-4d6d696f2fab' cmd.shell runas=root cmd='hostname; NwConsole -q -k -c "tlogin server=localhost port=56005 username=admin group=Administrators cert=/etc/pki/nw/node/node-cert.pem key=/etc/pki/nw/node/node-key.pem" -c "send /sys peerCert op=list"'
846c32b7-9dbc-445c-9cc8-4d6d696f2fab:
    LogHYBRID
    "f24e95c5.0" C = US, ST = VA, L = Reston, O = RSA, OU = NetWitness, CN = 3c95eea0-2561-426e-a9e8-3c2ada58f9ec
     sha-1:CF:6A:A2:B8:D4:17:90:28:66:F4:4E:3C:79:A7:6A:28:1A:A3:CC:CC
    "67342faa.0" C = US, ST = VA, L = Reston, O = RSA, OU = NetWitness, CN = nwappliance
     sha-1:BB:E0:8B:77:87:B0:D8:E5:54:53:38:F0:AD:39:8A:53:8F:56:A1:10
    "eeb9f4d7.0" C = US, ST = VA, L = Reston, O = RSA, OU = NetWitness, CN = 3c95eea0-2561-426e-a9e8-3c2ada58f9ec
     sha-1:CF:6A:A2:B8:D4:17:90:28:66:F4:4E:3C:79:A7:6A:28:1A:A3:CC:CC
    "1386a7d5.0" CN = rsa-nw-respond-server
     sha-1:78:2A:2F:92:20:3F:42:8E:54:B3:D6:98:A6:32:2E:38:BD:37:9A:0C
    "4349e381.0" CN = rsa-nw-metrics-server
     sha-1:49:0A:37:CA:10:6E:37:88:44:7C:77:07:5F:04:93:1F:8B:41:42:05

• Try installing the services on the Hosts page, and it will be successful now.