Certs used for trusted connections in RSA Security Analytics are stale after "Remove and Repurpose"

[#1]

Article Number

000027904

Applies To

RSA Product Set: Security Analytics
RSA Product/Service Type: SA Core Appliance
RSA Version/Condition: 10.4.0.1,10.4.0
Platform: Linux

Issue

Errors similar to this below are observed in /var/log/messages:

Oct 16 18:45:13 xxxxx collectd[22183]: NgNativeReader_NwBroker-SlowUpdate: client not initialized--cannot accept stat /sys/license/stats/license.status
Oct 16 18:45:14 xxxxx collectd[22183]: NgNativeReader_NwBroker-SlowUpdate: nwsdk failure: NwOpen returned 0; code 0; error: Could not find trusted session id in hello response
Oct 16 18:45:14 xxxxx collectd[22183]: NgNativeReader_NwBroker-SlowUpdate: failed to connect to device: failed to connect to nws://admin@localhost:56003/?group=Administrators&cert=%2Fvar%2Flib%2Fpuppet%2Fssl%2Fcerts%2F3bcfd8ce-e7f6-4bbe-be9e-52462992a533.pem&key=%2Fvar%2Flib%2Fpuppet%2Fssl%2Fprivate_keys%2F3bcfd8ce-e7f6-4bbe-be9e-52462992a533.pem
Oct 16 18:45:14 xxxxx collectd[22183]: NgNativeReader_NwBroker-SlowUpdate: client not initialized--cannot accept stat /sys/license/stats/license.version Oct 16 18:45:18 gsodc1loganal01 collectd[22183]: NgNativeReader_NwBroker-FastUpdate: nwsdk failure: NwOpen returned 0; code 0; error: Could not find trusted session id in hello response
Oct 16 18:45:18 xxxxx collectd[22183]: NgNativeReader_NwBroker-FastUpdate: failed to connect to device: failed to connect to nws://admin@localhost:56003/?group=Administrators&cert=%2Fvar%2Flib%2Fpuppet%2Fssl%2Fcerts%2F3bcfd8ce-e7f6-4bbe-be9e-52462992a533.pem&key=%2Fvar%2Flib%2Fpuppet%2Fssl%2Fprivate_keys%2F3bcfd8ce-e7f6-4bbe-be9e-52462992a533.pem
Oct 16 18:45:18 xxxxx collectd[22183]: NgNativeReader_NwIPDBExtractor-FastUpdate: nwsdk failure: NwOpen returned 0; code 0; error: Could not find trusted session id in hello response
Oct 16 18:45:18 xxxxx collectd[22183]: NgNativeReader_NwIPDBExtractor-FastUpdate: failed to connect to device: failed to connect to nws://admin@localhost:56025/?group=Administrators&cert=%2Fvar%2Flib%2Fpuppet%2Fssl%2Fcerts%2F3bcfd8ce-e7f6-4bbe-be9e-52462992a533.pem&key=%2Fvar%2Flib%2Fpuppet%2Fssl%2Fprivate_keys%2F3bcfd8ce-e7f6-4bbe-be9e-52462992a533.pem

Cause

When the appliance is re-added (and re-provisioned) through the Security Analytics UI, the puppet recipes only check whether the key and certificate material used for trusted connections (/etc/netwitness/ng/<svc>/trust(peers|certs) and /opt/rsa/carlos/keystore) exist.  They are not checked for whether they exist and are synchronized with the values in /var/lib/puppet/ssl.

Examples of <svc> include the following:

• appliance
• logdecoder

As a consequence, trusted connections do not work after re-provisioning the appliance.


When this occurs, the only remedy is to manually remove the trusted certs and re-run puppet.

Resolution

The Steps below can be performed to correct this issue.

1. For Security Analytics classic nodes (decoder, logdecoder, concentrator,broker,etc)
   1. Connect to appliance via SSH and login with root user
   2. Create temp folder under /tmp directory for each service to store backup certificate, example, #mkdir /tmp/logdecoder
   3. Navigate to appropriate directory with following command, #cd /etc/netwitness/ng/<svc>/trustpeers
   4. Issue following command to backup and remove certificate for each service, # mv * /tmp/logdecoder/
   5. Re-run puppet in the foreground, #puppet agent -t, or wait up to 30 minutes for puppet to run automatically.
   6. Verify that new trusted peer certificate are created. 
   7. Restart the collectd service with the command service collectd restart
2. For Security Analytics appliance that has Carlos service (ESA, Malware,etc)
   1. Connect to appliance via SSH and login with root user
   2. Create temp folder under /tmp directory for appliance to store backup keystore, example, #mkdir /tmp/esa
   3. Navigate to appropriate directory with following command, #cd /opt/rsa/carlos
   4. Issue following command to backup and remove keystore file, # mv * /tmp/esa/
   5. Re-run puppet in the foreground, #puppet agent -t, or wait up to 30 minutes for puppet to run automatically.
   6. Verify that new keystore file is created. 
   7. Restart the collectd service with the command service collectd restart