Re-provision Netwitness Hosts Under Chef (11.X)
Process for removing and re-adding a host in order to change hostname, IP or Node-Zero IP.
This procedure will work on any appliance type.
Special thanks to Ken Pineiro for giving us the solution
References
000035662 - How to add hosts or services back to the UI in RSA NetWitness Logs & Packets 11.0
000035674 - How to manually re-add a host that has been removed from the RSA NetWitness
Remove host trusts
On the appliance (DO NOT DO THIS ON NODE-ZERO!!!):
- Take note of the existing salt minion ID
cat /etc/salt/minion
- Move all existing certs and trusts to tmp location
mkdir -p /tmp/cert-backup/ng
mv /etc/netwitness/platform /tmp/cert-backup/
mv /etc/pki/nw /tmp/cert-backup/
mv /etc/salt/pki/minion /tmp/cert-backup/
mv /etc/netwitness/ng/{appliance,concentrator,decoder,logcollector,broker,logdecoder,truststore,*_truststore} /tmp/cert-backup/ng/
mv /etc/salt/pki/minion/minion_master.pub /tmp/cert-backup/
mv /etc/netwitness/ng/*.pem /tmp/cert-backup/ng/
- Check the /etc/netwitness/ng folder afterwards to ensure there are no other folders containing certificates.
- If node-x is running a launch service (e.g. ESA), then you may also have to move the following:
mkdir /tmp/cert-backup/systemd
mv /etc/systemd/system/rsa-nw-xxxx-server.service.d/rsa-nw-xxxx-server-optsmanaged.conf /tmp/cert-backup/systemd/
Remove Node-Zero references to host
On the Admin Server:
- GUI Hosts page
Select the host and choose “Remove Host”
- Command line
- Check to see if the salt ID of the host is still in the system:
salt-key | grep <Salt Node ID>
- If the salt node ID of the removed host is still present, remove it
orchestration-cli-client --remove-key <Salt Node ID>
Make changes to host
At this point you will likely want to change something about the host, like the IP, hostname or even
the address of the Node0 (Admin server).
Changing the hostname
- Change it under the sysconfig settings
hostnamectl set-hostname newhostname
- Update /etc/hosts
vi /etc/hosts
Changing the IP
- Change it under sysconfig settings
vi /etc/sysconfig/network-scripts/ifcfg-<NAME>
Changing the Node0 IP
- Change the Admin server reference under the following
vi /etc/hosts
vi /etc/salt/minion
vi /etc/netwitness/ng/amqp.json
Re-add host
- Re-run the setup tui (choose the Install option)
nwsetup-tui
- Reboot the host
Re-Discover the host
- Go to the Web UI > Hosts page and click Discover
- Install the appropriate service onto the host (should not take long at all)
Troubleshooting
- Restart services to fix service connection issues
- Symptom:
Core services won't green up
H&W is showing alarms related to message bus failures
Stats page won't show all of the stats for the service
- Fix:
Most likely the services just need to be restarted.
Restart the applicable services on the NodeX having issues
systemctl restart nwconcentrator
systemctl restart nwappliance
systemctl restart collectd
systemctl restart rabbitmq-server
If you are still seeing message bus errors after doing this, try restarting rabbitmq on the Admin Server
systemctl restart rabitmq-server
Cleanup
- When verification is completed and system is working correctly
rm -rf /tmp/cert-backup
