NetWitness Community

JohnSnider · ‎2019-07-15

Updated for Version 12.x (backward compatible to 11.x)

Scenario -

Need to remotely backup your NetWitness hosts to a central location, to satisfy Disaster Recovery Requirements, perform a Tech Refreshes, or to be prepared for RMA replacement of a device.

Solution – A Wrapper for NRT

Building off the framework of the original nw-backup scripts written for 10.x backup/restore and migration to 11.x, a new set of version 11/12 scripts has been written as a "wrapper" to the built in NetWitness Recovery Tool (NRT) functionality of NetWitness since version 11.2 was released.

(Please note that this is not an officially supported solution by NetWitness Support, but can be used by customers as a possible backup solution, at their own risk)

Overview –

The solution consists of 5 scripts (all run from the NW Admin server (node-zero)), supporting files for custom feed backup fix and an example nw-base.nrt file (note: The '11' has been removed from the script names since this can support versions 11 or 12):

get-all-systems.sh – Generates an inventory file of all hosts in the NW server deployment.

ssh-propagate.sh – Generates an ECDSA key pair on the NW Server and propagates the public key to all hosts.
nw-backup.sh – The main backup script.
nw-restore.sh – Restore script for RMA/Tech refresh.
cf-fix.sh – Custom feed backup fix (used for applying fix to Warm Standby Server).
feedfix folder – Contains the backup.sh and restore.sh scripts to support the backup of custom
feed files on the admin server.
nw-base.nrt – An NRT script to add custom files to the backup directory as part of the backup process.

Process Walkthrough –

Copy the attached zip file to the NW Admin Server host (node-zero) and unzip:

mkdir /root/scripts

unzip nw-backup.zip -d /root/scripts

cd /root/scripts

chmod +x *.sh

Step 1: Run get-all-systems.sh

./get-all-systems.sh

Usage:

./get-all-systems.sh [ -u <USER> ][ -p <PASSWORD> ][-d /home/path ][ -b </backup/path> ][-n <hrs>][-h]

Configuration Options

Note: All command-line options are optional.

With no options selected, the script will use options set within the file itself.

-b </backup/path> : Path to store the all-systems file and logs. Default:(/var/netwitness/nw-backup)

-u <username> : User acct to use for non-root SSH access to hosts during backups. Default:(root)

-p <password> : Password for <username> Default: (ask)

-g <group> : Alternate Primary Group for entered username. Default: (username)

-d </home/path> : Base home directory path (/home) for <username> Default:(/var/netwitness/nw-backup)

-n <hrs> : Skip check for new systems if current new-systems is less than <hrs> old.

-h : Print this help information.

Run on the NW Admin Server. Creates the /var/netwitness/nw-backup directory (or the directory passed with the -b option), then using a combination of mongo and salt queries, will create the all-systems file in that directory. The all-systems file is used by the other scripts , with entries that contain the following in comma separated format:

DeviceType,Hostname,IPAddress,MinionID,SerialNumber

DeviceType = NRT Category Type (i.e. AdminServer,Broker,Concentrator,Decoder, etc.)

Hostname = Short hostname (not FQDN)

IPAddress = Management Interface IP address of host

MinionID = Unique Salt MinionID of host

SerialNumber = Device Serial Number for Reference and Support

Example:

AdminServer,nw-admin,192.168.1.129,70f95dc0-3cb6-4fd4-b9f2-ac923d0ba594,PK10T51

ESAPrimary,nw-esa,192.168.1.131,a598cb6b-4bd2-4ba2-af6a-79df3dab35e6,R9L8LNM

Broker+Search,nw-broker,192.168.1.130, 2a83b597-7970-4872-b76f-109cb591fa90,CSZ77X2

LogHybrid,nw-loghyb,192.168.1.133,87fc872c-68e3-45e3-9108-e30f847dc14e,PK10T0A

Malware,nw-malware,192.168.1.132,2c98e425-57a0-47d2-82d7-15795a6165f5,R90BCFWP

NetworkHybrid,nw-nethyb,192.168.1.134,9a99294e-3889-48b0-9555-11d3c21e2018,R90218K6

The script is designed to run from the cron on a regular basis (if you are in a dynamic environment were systems are added/removed on a regular basis), it has a 30 second timeout on the one question it asks.

If changes to the environment are detected, generation of new-systems and/or old-systems files will occur. The new-systems file can be used by other scripts for running specific targeted actions against the newly installed hosts. If a system is “offline” or has been removed from the UI, the old-systems file will have the entries for those hosts, so information about them is not lost.

Step 2: Run ssh-propagate.sh

./ssh-propagate.sh

Configuration options

Note: All command-line options are optional.

Run with no options, script uses the /var/netwitness/nw-backup/all-systems file to target

all hosts and copy the root users ecdsa-521 bit public ssh key to all hosts

Usage:

./ssh-propagate.sh [ -u <user> ] [-c] [ -b <path> ] [ -t <target> ]

-b <path> : path to the location of all-systems file. Default: (/var/netwitness/nw-backup)

-u <username> : User to propagate keys to on all nodes (must exist on NW Server). Default: (root)

-g <group> : Group associated with <username>. Default: (username)

-c : Used with -u <username> [ -g <group> ] for non-root user, creates user on remote hosts.

-t <target>: Target hosts (new-systems or anything grep-able from all-systems. Default: (ALL)

Run on the NW Admin Server. Performs the following (depending on options):

Verifies the designated user account already exists on the NW Server
Updates the /etc/hosts file with host shortnames using entries from all-systems file and hosts.custom file (if it exists in the same directory as the script)
Generates an ecdsa-521bit ssh key for root (or the specified user) if it does not exist
Iterates through the target list (default is ALL hosts in the all-systems file) and performs the following:
- Tests ssh connectivity to the host (host responding on port 22)
- Verifies the user exists on the remote host, if not and the -c option is selected
  - Creates remote user acct
  - Sets password to same as user acct on NW server
- Tests SSH key authentication to host as the user specified, if auth fails:
  - Copies the user ecdsa public key string to the ~/.ssh/authorized_keys file
  - Adds the target host fingerprint to the user's ~/.ssh/known_hosts file on the NW server
  - Verifies ssh connectivity via ssh-key authentication

Step 3: Modify the nw-base.nrt file

vi /root/scripts/nw-base.nrt

The default /etc/netwitness/recoverytool/nw-base.nrt file, distributed file with 11.x systems, only contains the following entries:

name nw-base

directory /etc/netwitness/platform/nodeinfo

file /etc/machine-id

# unmanaged files

stash /etc/fstab

stash /etc/hosts

stash /etc/sysconfig/iptables

stash /root/.ssh

# for azure

stash /etc/krb5.conf

stash /etc/logrotate.d/waagent.logrotate

stash /etc/mdadm.conf

stash /etc/waagent.conf

# only for nrt mode, run dnsclient to update the hosts configuration
after-import if [ "${NRT_MODE}" != 'standby' ]; then chef-client --config /var/lib/netwitness/config-management/client.rb --logfile /var/log/netwitness/config-management/chef-client.log --log_level info --runlist 'recipe[nw-dns-client]' > /dev/null 2>&1; fi

The “STASH” entries are NOT restored during an NRT import (recovery), but are available for reference in the /var/netwitness/backup/unmanaged folder, after the import. The included nw-base.nrt file has an expanded list of stash files and directories to include files used at several customer installations. To Fully restore functionality, these files need to be available after the restore.

Extended nw-base.nrt supplied with the scripts:

name nw-base

directory /etc/netwitness/platform/nodeinfo

file /etc/machine-id

# unmanaged files

stash /etc/fstab

stash /etc/hosts

stash /etc/resolv.conf

stash /etc/nsswitch.conf

stash /etc/passwd

stash /etc/shadow

stash /etc/group

stash /etc/sudo.conf

stash /etc/sudoers

stash /etc/sudoers.d

stash /etc/exports

stash /etc/krb5.conf

stash /etc/nfs.conf

stash /etc/ntp.conf

stash /etc/rsyslog.conf

stash /etc/logrotate.conf

stash /etc/sysconfig/network

stash /etc/sysconfig/nfs

stash /etc/sysconfig/iptables

stash /etc/sysconfig/iptables.bak

stash /etc/sysconfig/iptables-config

stash /etc/crontab

stash /etc/sysconfig/network-scripts/ifcfg-em1

stash /etc/sysconfig/network-scripts/ifcfg-em2

stash /etc/sysconfig/network-scripts/ifcfg-em3

stash /etc/sysconfig/network-scripts/ifcfg-em4

stash /etc/cron.hourly

stash /etc/cron.daily

stash /etc/cron.weekly

stash /etc/cron.daily

stash /etc/pam.d/netwitness

stash /etc/pam.d/securityanalytics

stash /etc/logrotate.d

stash /etc/logstash

stash /etc/multipath.conf

stash /etc/lvm

stash /etc/raddb

stash /etc/rsyslog.d

stash /etc/snmp

stash /etc/ssh

stash /var/ace

stash /home

stash /root (note: this backs up ALL file/folders in /root including /root/.ssh)

stash /var/netwitness/nw-backup/all-systems

# for azure

stash /etc/krb5.conf

stash /etc/logrotate.d/waagent.logrotate

stash /etc/mdadm.conf

stash /etc/waagent.conf

# only for nrt mode, run dnsclient to update the hosts configuration
after-import if [ "${NRT_MODE}" != 'standby' ]; then chef-client --config /var/lib/netwitness/config-management/client.rb --logfile /var/log/netwitness/config-management/chef-client.log --log_level info --runlist 'recipe[nw-dns-client]' > /dev/null 2>&1; fi

Edit the file to include any additional locations (files or directories) that contain customizations in your deployment, then save the file in the same directory as the nw-backup.sh script. The backup script will verify the file on each system matches your modified file and if not, will automatically copy the modified file to each host before running NRT on that host.

Step 4: Run nw-backup.sh

./nw-backup.sh

Configuration Options:

Note: All command-line options are optional.

With no options selected, the script backup ALL devices listed in the

/var/netwitness/nw-backup/all-systems file(except any commented(#) out)

and copy the backup files to the /var/netwitness/nw-backup/<date>/ directory

on the NW Server.

Usage:

./nw-backup.sh [-b <NRT path>] [-m <xfer mode>] [-l <NFS mount point)>] [-p <NW backup path>] [-s <remote server IP>] [-d <Dest path>] [-t <Target>] [-u <SCP user>] [-g <SCP group>] [-U <Rmt SCP user>] [-G <Rmt SCP group>] -M -L -R -I

Options:

-b <local NRT backup path> : path for NRT backup files. Default: (/var/netwitness/backup)

-m <mode> : remote transfer mode (scp or nfs). Default: (scp)

-p <NetWitness Server backup path> : Path on NW server for logs and location of all-systems

file. Default: (/var/netwitnes/nw-backup)

-d <Destination backup path> : Path on destination server to move completed backup files to

via nfs or scp. Default: (/var/netwitness/nw-backup)

-s <remote server IP> : Destination server IP address for storing completed backup files,

transferred via nfs or scp. Default: (NW Server IP)

-u <SCPUser> : User acct for SCP transfers of completed backups, user must exist on all

target systems and on Destination server. Default: (root)

-g <SCPgroup> : Group for assigning permission to files copied to remote host. Default (root)

-U <RmtSCPUser> : User acct on Remote system SCP transfers of completed backups, user must exist on
remote server adn have SSH-Key auth configured. Default: (root)

-G <RmtSCPgroup> : Group associated with Remote SCP user for transfer of completed backups, group must
exist on remote server. Default (root)

-l <mount_point> : local mount point for NFS share. Default: (/mnt/backup)

-t <Target> : backup ONLY specific Target(s), (can be anything greppable from all-systems file)
Default: (all)

Special Targets:

core (Broker, Concentrator, Decoder, LogDecoder, Archiver,

LogCollector(vlc), NetworkHybrid, LogHybrid)

nonw (all devices except AdminServer)

nwonly (AdminServer only)

esaonly (all ESA devices only)

endpoint (all endpoint devices EndpointHybrid, EndpointLogHybrid, Gateway)

Exclusions/Inclusions/Service Control:

-M : Exclude the Malware Analysis File Store.

-I : Include the Broker Index files for RMA/Tech Refresh situation. Default: (Exclude)

-L : DO NOT STOP LogCollector (nwlogcollector) service during backup. Default: (Stop Service)

-R : DO NOT STOP Reporting Engine (rsasoc_re) service during backup. Default: (Stop Service)

Note: NRT normal operation stops these services, not stopping will affect:

LogCollector - Will not have latest tracking data for some logs.

Reporting Engine - Some live chart data and alert status data may be lost.

NOTES –

If using SCP to a server other than the NW Admin Server, the copy uses the "-3" option and makes the transfer of the backup file via the NW Admin Server, so ONLY the NW Admin server needs to have SSH key authentication configured to all hosts.
Make sure the modified nw-base.nrt file is in the same directory you are running the nw-backup.sh script from, the script will hash that file, and verify that hash against the file on each server, if they do not match, it will copy the file in the script directory (where you ran the nw-backup.sh script from) to the remote host, before triggering the NRT backup on that host. If you do not want to make any modifications to the default file, don’t include the file in the script run directory.
Deploy Admin password is programmaticly called, so no exposure of password in the scripts
The /var/netwitness/nw-backup/all-systems file can be used for a myriad of other scripting calls, or for targeting a specific type of host, especially when using “salt” commands.

Restore: nw-restore.sh

Follow the published backup/restore document for the version of NW you are running
The only change will be that the file is archived in a tar file, so after copying the file to the /var/netwitness/backup directory on a NEW host, run tar -xvf <backup-file-name>.tar to extract it there.
Then follow the remaining instructions for running nwsetup-tui in recovery mode, install services, and upgrade host to same version as backup.
Then either manually run nw-recovery-tool --import or use the nw-restore.sh script from the NW server to finish the restore process.

nw-restore.sh

Usage: ./nw-restore.sh -t <hostname> [ -b <NRT path> ] [ -p <NW backup path> ] [ -I ] [ -h ]

Configuration Options:
Note: All command-line options except -t <hostname> are optional.
With no other options selected, the script will use default paths
and run against the targeted host.
General Options:
-b <local NRT backup path> : path to the location of NRT backup files on
machine being restored.
Default: (/var/netwitness/backup)
-p <NW Server backup path> : Base path on NW Server for logs and location of
all-systems file.
Default: (/var/netwitness/nw-backup)
-t <Hostname> : Hostname or IP of server being restored. (REQUIRED)
-I : Include the Broker Index files for RMA. Default: (Exclude)

Examples:
#1: ./nw-restore.sh -t nw-decoder-01
Would run the restore on 'nw-decoder-01' device with default path options:
- Backup files on host located @ /var/netwintess/backup
- all-systems file located on the NW server
@ @ /var/netwitness/nw-backup/all-systems
#2: ./nw-restore.sh -b /var/netwitness/localbackup -p /var/netwitness/nwbackups
-t nw-broker-02 -I
Would run a restore with the following options:
-b : NRT backup files are in /var/netwitness/localbackup on host being restored
-p : all-systems file is located in /var/netwitness/nwbackups on the NW Server
-t : get target host info for 'nw-broker-02' from the all-systems file
-I : If Index data exists in the backup files, restore the index files

CHANGE NOTES:

Device-Type Changes

Previous versions of the backup script were dependent on the Device-Type, listed in the all-systems file, being a single type (i.e. AdminServer, Broker, Decoder, etc.) and was based off the single "installed services" entry in mongo for each host.
NetWitness 11.4+ introduced the ability to add additional services to hosts (i.e. add New Health & Wellness(Search) to a Broker, or Endpoint Server to a LogHybrid), so now multiple "installed services" are listed for these hosts.
The get-all-systems.sh script, has been updated to handle this scenario by concatenating the installed services together, separated by a '+'. In the above examples the all-systems would show the Device Type listed as Broker+Search and Endpoint+LogHybrid. This allows continued use of the entries in all-systems to be used as a reference by other scripts when looking for specific services.

Non-root User for SCP

The backup scripts now support using a non-root user:group for the SCP copy of the backup files from the host being backed up, to the NW Server (or to an external destination server). If your environment does not allow direct login as root via ssh, or the host you are copying the backups too requires a non-root user, then this will allow for that scenario.
The backup user is required to have full access to "read" the files written by NRT during the backup, and the ability to create directories and files on the destination server.

get-all-systems.sh

Run with the -u <username> option, (assumes group name the same as username), if the user account/group does not exist on the NW Server, you will be prompted to have them created.
Default Settings for the account will be the following:

Example: using nwbackup as the username and not specifying an alternate home
directory (-d)

username: nwbackup

password:(you will be prompted to enter)

group: nwbackup (same name as the username)

home: /var/netwitness/nw-backup (or what the -b <bupath> option is set to)

shell: /bin/bash

Using the -d <homepath> will create and alternate home directory path (ex: /home)

Adding the -g <group> option will create/use a group that is different than the username
Adding the -d </home/path> option will create an alternate home directory.
If preferred, the user account can be created manually before running the script or use an existing account. If ONLY the username option is given, the Owner:Group of the backup directories will be set to username:username, if group option is also given they will be set to username:groupname.
If copying the backup to a remote (non-netwitness) destination server, the user MUST exist on that server, have the same primary group (or use the -U/-G options to designate a different user:group on the destination server), have write permissions to the destination path backup files will be stored and have ssh key authentication pre-configured.

ssh-propagate.sh

Run with the -u <username> option, create and propagate the ecdsa-521 ssh key for this user to all other nodes in the all-systems file.
If user does not exist on a node, adding the -c option will allow the script to create the remote user and set the password the same as the local user on the NW server, as well as copying the ssh key to that host.
If an alternate group name was used during the run of get-all-systems.sh, then you should run this with the -g <groupname> option also, so users are created with the same user:group on all systems.

nw-backup.sh

Run with the -u <username> uses designated username for all copying of backup files from nw nodes being backed up, to the designated destination (NW Server or external host).
If the group option was given during he previous setup scripts, then it should be included here also.
User MUST exist on ALL NW nodes and have read permissions to the backup files and write/create permission on the destination host.
If the destination host requires a different user:group for SCP, then use the -U and/or -G options to designate the remote user:group for transfers.

This does not change the fact that the actual backup scripts MUST be run as the root user, it just allows the use of a NON-ROOT user to do the file copies between systems.

/etc/hosts updates (ssh-propagate.sh)

Pre 11.5 - checks for IP entry in /etc/hosts for each host in the all-systems file, and if missing, adds an entry for <ipaddress> <hostname>.
Post 11.5 - /etc/hosts is controlled by a chef process to handle entries for known nw hosts, the new entries will look like this:
<IPaddress> <UUID> <UUID.netwitness>

Any entries manually added to /etc/hosts will be wiped out by the chef process. To add <hostname> (short hostname) to these entries, a new file /etc/hosts.user is available (it contains the localhost entry for each individual host to be merged with the /etc/hosts.netwitness file to create the new /etc/hosts. The hosts.user file (format: <ipaddress> <hostname> ) must be edited, so the chef recipe can update the /etc/hosts file with the hostnames.
ssh-propagate.sh will automatically update the /etc/hosts.user file from the entries in the all-systems file, and run the chef recipe to update the /etc/hosts, so the entries will read:
<IPaddress> <UUID> <UUID.netwitness> <hostname>
In addition, if you need to add other hosts to the /etc/hosts that are not part of the Netwitness Deployment (and not resolvable by DNS), you can create a hosts.custom file in the directory where the ssh-propagate.sh is run from, and it will append those entries to the /etc/hosts.user file before running the chef recipe.

nw-backup.sh updates

Previously, nw-backup.sh relied on the Device-Type entry in all-systems to match an NRT Category, NRT would then use the category.sequence file to get a list of components to backup for that Device Type. Post 11.4, with the ability added to install additional services on hosts, this process would no longer work effectively.
The script no longer depends on the Device-Type entry in the all-systems to run the backups, (it will still use it for filtering on specific types if the -t option is used, but is not used to determine what services are running on a particular host). It now queries each host for a list of enabled processes, and parses that list to determine what components are enabled on the host, and dynamically creates a component list to pass to the NRT script when launched.
Any changes to a host (adding services) will not require a change to the backup process, as the new services will be discovered automatically when the backup is run.
NFS use (for copying backup files to another host) has been updated to give better feed back if there are issues.
Addition of the -I option to backup Broker Index files for RMA/Tech Refresh so they can be restored using the nw-restore.sh script without losing their aggregation last session information (NOTE: this DOES STOP the broker service during the backup)
For versions less than 12.3, Improved extract of custom feed files to support restore of files for proper operation during RMA or transfer to warm standby server.

cf-fix.sh (and feedfix directory with backup.sh and restore.sh files)

Note: as of version 12.3, this fix is no longer needed, the functionality has been built into NRT.
NRT excludes excludes the /var/lib/netwitness/uax/scheduler directory in the sa-server NRT backup script, due to that directory holding all the job data from pcap/log extracts as well as the custom feed files (xml/csv). This causes an issue with transfer of the custom feed files to the warm standby server during sync, the jobs still exist, but the supporting files are missing.
To fix this issue, the nw-backup.sh script automatically adds the 2 scripts from the feedfix directory (backup.sh and restore.sh) to the /var/netwitness/uax/nwtools directory and modifies the sa-server.nrt file to run them pre-backup and post-restore.
The backup.sh script extracts the custom feed files (*.xml and *.csv) to a tar file and places it in the /var/netwitness/uax/backup directory so it is included in the backup of the sa-server files.
The restore.sh script extracts those files back to the proper location to support the custom feed jobs.
If you have a Warm Standby Admin server, copy the cf-fix.sh and the feedfix directory to the Standby server and run the cf-fix.sh script to configure the sa-server.nrt file with the new commands and add the 2 script files to the proper location. If you have run the nw-backup.sh script on the Primary Admin server these modifications are already made, so when the warm-standby sync job runs (basically an NRT run) and the backup file is copied to the standby server, during fail-over to the standby server, the files will be restored properly.

Version 7.5 updates

All-Scripts:
Several customers made modifications to the UI Admin -> Hosts entries due to obscure hostname entries that made it hard to dertermine the type of the server, so they added additional information to the
entry, like "XVTPC73 - Admin Server".
- The "Hosts" entries HAVE to be treated like actual hostnames, hence should not contain spaces, as this breaks several processes.
- Additions have been made in the get-all-systems.sh script to check for any spaces in the entries pulled from the Orchestration Mongo DB, if found the script will output a messages warning of the entries
  with Spaces and that they need to have the spaces removed from the name. In the previous example, just rename the Host entry to be: "XVTPC73-AdminServer" and everything will work correctly.
- In addition the ssh-propagate.sh and the nw-backup.sh have been modified to "check" for any entries with spaces in the "all-systems" file, in case it was manually created, or someone edited the file to add
  additional information in the file. The will similarly exit and print out an error showing the offending entry and how to fix it.

all references to the old v11 scripts (get-all-systems11.sh, ssh-propagate11.sh, nw-backup11.sh and nw-restore11.sh) have been removed, since scripts support both 11.x and 12.x versions.

ssh-propagate.sh

previously when running the script, the complete hosts entry process would run every time, even if no changes were needed (all "IP Host" entries already exist in the proper file).
added a quick cursory check, and if no additional hosts have been added, then the chef job will not run to re-create the /etc/host file. Speeding up the process.

JohnSnider · ‎2019-08-09

get-all-systems11.sh now supports NATed nodeX devices (all-systems will contain the NAT address seen by the NW node-zero server)

LucaBernabei · ‎2020-02-17

Hi, i'm thinking of using this solution for some customers that have just updated to version 11.3, so i'm trying it in my Lab.

When i use the "get-all-system" script, it ends with this errors:

2020-02-17 10:19:08 +0000 | 23843 | Warning: Generation of {GRB}all-systems completed with 3 errors, please review log file @ /var/netwitness/nw-backup/rsa-get-all-systems11-2020-02-17.log.
ERROR: ,ufccapsiem01,,u26e025b4-a88b-4377-a765-ee6fd104b013,uVMware-423e55f3ce27fabc-62d941d8ebdb1877 line in /var/netwitness/nw-backup/all-systems has invalid entries or empty spaces.
ERROR: ,ufccapsiem02,,u388e4a92-5ecd-429e-841b-d29b2ac8c5c2,uVMware-423ea34ce833842e-c972d941b0aa3d52 line in /var/netwitness/nw-backup/all-systems has invalid entries or empty spaces.
ERROR: ,ufccapsiem03,,u83948541-679e-439a-8079-2048bfc5901d,uVMware-423e891d57838a79-4592f3a9110bb7a1 line in /var/netwitness/nw-backup/all-systems has invalid entries or empty spaces.

And this is the file generated:

,ufccapsiem01,,u26e025b4-a88b-4377-a765-ee6fd104b013,uVMware-423e55f3ce27fabc-62d941d8ebdb1877
,ufccapsiem02,,u388e4a92-5ecd-429e-841b-d29b2ac8c5c2,uVMware-423ea34ce833842e-c972d941b0aa3d52
,ufccapsiem03,,u83948541-679e-439a-8079-2048bfc5901d,uVMware-423e891d57838a79-4592f3a9110bb7a1

As you can see there are some missing information, like the host type, and also i have a "u" at the beginning of every information.

Regarding the "u" problem i noticed that it appear when the script use the "--out=newline_values_only" function after the Gathering Host System info from Salt Grains.

Can you help me to solve this?

Thanks

JohnSnider · ‎2020-02-17

I just noticed this issue while testing on 11.4, a new version has been uploaded that fixes the issue. Please download the latest zip file and use it for your testing.

LucaBernabei · ‎2020-02-18

Thanks for the fast update, i confirm that the last version works also on 11.4

JohnSnider · ‎2020-02-18

Just noticed an error in the ssh-propagate script, so please download the latest copy with the fix.

JohnSnider · ‎2020-03-25

Minor update to get-all-systems and nw-backup to support some of the new device types in 11.4

ExileNjoka · ‎2020-09-09

It is my first time to use the script and am new to RSA Netwitness. John Snider‌ , Please help me understand the following:

1. By default, are the event sources, reports and other configurations automatically backuped?

2. On the other node other than NWAdmin, Do i need to to mount NFS share?

3. do all the backup get stored on the NWAdmin location or in each individual server?
3. in my case, with the below specification, how will i define my script to be able to backup to the remote server:
a. I have mounted //A.B.C.D/backups(my remote windows share folder) on /mnt/netwitness-backups on the NWADMIN
b. //A.B.C.D/backups contains netwitness folder which is where i want to keep my backup

JohnSnider · ‎2020-09-09

1. Yes everything is backed up using the built in NRT (Netwitness Recovery Tool) settings, I also add some additional items to the backup using the nwbase.nrt file stash settings for non netwitness files (PAM auth files, and other OS settings)

2. No need to mount storage on other hosts, the backup tar files will be transfered (SCP'd) from the remote hosts to the NW Admin server and placed in the location you specify (or by default in the /var/netwitness/nw-backup/<date folder>/

3. Both, there is a copy on the remote server as well as one copied to the NW Server (or to a remote host you specify)

4. If you have a mounted share on /mnt/netwitness-backups on the NWAdmin server, and want the backup stored in the netwitness folder on that mounted location, the command to run the backup would be:

nw-backup11.sh -d /mnt/netwitness-backups/netwitness

This will place the backup files in /mnt/netwitness-backups/netwitness/YYY-MM-DD/ folder for the date ran. and place the logs and the all-systems files in /var/netwitness/nw-backup/, if you want to keep everything in one place, you can specify:

nw-backup11.sh -p /mnt/netwitness-backups/netwitness -d /mnt/netwitness-backups/netwitness

This will put the all-systems files and the logs in /mnt/netwitness-backups/netwitness, and the backup files from each host in /mnt/netwitness-backups/netwitness/YYY-MM-DD folder.

Hope that helps

ExileNjoka · ‎2020-09-09

This is so insightful and helpful. Thank you so much.

Its working

KapitanAl · ‎2021-01-13

John Snider‌ - I have just deployed you solution on the environment and it works as a charm. Many thanks!

Are you considering updating the docs to reflect the newest changes (e.g.,-c and -u options)?

JohnSnider · ‎2021-01-13

Yes, a new version (and associated docs) will be coming out in the next couple of weeks.

JohnSnider · ‎2021-03-19

New Version released today, this supports up to 11.5.2.0 (tested) and should work on any version above 11.2

JohnSnider · ‎2021-03-22

To all users of these scripts:

If you downloaded the update that was posted the week of 3/15/2021, are running 11.5+, and executed the ssh-propagate11.sh, it may have changed the default 127.0.0.1 and ::1 lines in /etc/hosts, and the "localhost" entries are missing.

By default the /etc/hosts.user has the IPv4/IPv6 localhost lines already in it, and the script inadvertently overwrote them, the new script will not do that, but you will need to re-create the original /etc/hosts.user file and re-run the ssh-propagate11.sh script

To fix please do the following:

cat /etc/hosts | grep 127.0.0.1 > /etc/hosts.user

edit the /etc/hosts.user and modify the entry as follows

Note: <hostname> is the shortname of the host

<UUID> is the existing UUID of the host from the copied line (remove any xxx.netwitness entries)

127.0.0.1 <hostname> localhost localhost.localdomain localhost4 localhost4.localdomain4 <UUID> nw-node-zero
::1 <hostname> localhost localhost.localdomain localhost6 localhost6.localdomain6

download the new version of the nw-backup11.sh.zip file and extract the files

re-run the .ssh-propagate11.sh script and the entries should be corrected

I apologize for the error in the script, testing did not cause any errors since resolution of localhost always returned 127.0.0.1, so the backups did not fail when running mongodump, if your DNS does not resolve "localhost" to its default 127.0.0.1, then your backup of the NW Server would fail during the mongo backup.

GergelyLitavszk · ‎2022-11-14

Dear John Snider,

We tested the solution on the 12.0.0.0 version. (I know it is not a supported version so far, but we needed it for our infrastructure).

There was only a small problem in the script where it checks the version whether it should include node-infra or not in the comp list. (Line 722 if-case)

If the version is 12.0.0.0 -- it checks only the second part --, it does not inclue node-infra in the backup

With a small modification in the if statement condition, to check the first part as well (11.x or 12.x) it works properly. Something like this:
if [ $(OWB_FORCE_FIPS_MODE_OFF=1 salt-call nw_platform.version --out=newline_values_only | awk -F. '{print $2}') -ge 4 ] || [ $(OWB_FORCE_FIPS_MODE_OFF=1 salt-call nw_platform.version --out=newline_values_only | awk -F. '{print $1}') -eq 12 ];then

Many thanks for your hard work on this! It is really helpful.

Regards,
Gergely

JohnSnider · ‎2022-11-14

Thanks for the info, I'll get it fixed and a new version posted (I have a few additional updates to the scripts anyway that I have not posted yet)

JohnSnider · ‎2023-01-03

New version supporting 12.x (and 11.2+) is posted, please download and use these latest updates in your deployments.

LucaBufalieri · ‎2023-12-20

I am writing this comment in order to describe my problem with the script and how I solved it.
I use the script daily, and most of the time the script stops for something wrong. From recovery.log file I can read the following error,

ERROR: Minions returned with non-zero exit code

In order to understand the error, I added set -x at the begging of the script, in this way it was possibile to identify that the problem was in this command.

salt [salt] cmd.run 'nw-recovery-tool -e -c [component] -d [path] --archive-file [path]'

I tried to replicate the error, also in this case I had not 100% of failures and in case of failure it was possibile to read this message.

Minion did not return. [No Response]
The minions may not have all finished running and any remaining minions will return upon completion. To look up the return data for this job later, run the following command:

salt-run jobs.lookup_jid 20231208111810402244

Although the command fails, it was possible to find the backup in the requested path and run the proposed command it was possible to see that the return code was zero.
Based on what I described so far, I suspected that salt-master needed more time. After some researches, I found on https://salt-sproxy.readthedocs.io/en/latest/opts.html that it was possible to increase the timeout through the option --timeout.

To sum up, I changed line 1012 and line 1028 in the following way, in blue the added part.

OWB_FORCE_FIPS_MODE_OFF=1 salt ${HOST_ID} --timeout=[minutes] cmd.retcode "nw-recovery-tool -e ${COMPS} -d ${LCL_BUPATH} ${DEPLOYPW} --archive-file ${LCL_BUPATH}/${FILE_NM}" --out=quiet

OWB_FORCE_FIPS_MODE_OFF=1 salt ${HOST_ID} --timeout=[minutes] cmd.retcode "nw-recovery-tool -e ${COMPS} -d ${LCL_BUPATH} --archive-file ${LCL_BUPATH}/${FILE_NM}" --out=quiet

This changes allowed me to work without further errors.

JohnSnider · ‎2023-12-20

May I ask what the host type was that you were trying to run the backup on? if it on one of the nodes with a mongo DB, or the new H&W with elastic DB, the size of the DB may be the issue, if it's ESA and you do not have a retention setting configured for alerts/incidents the DB can get huge and take a long time to backup. What are you setting your [minutes] to for the timeout?

LorenzoMadonna · ‎2023-12-21

Thank's for your job.

on line 672 there is an error in the salt options.

Original:

OWB_FORCE_FIPS_MODE_OFF=1 salt ${HOST_ID} file.remove ${LCL_BUPATH}/${FILE_NM}.sha256 -out=newline_values_only > /dev/null

Edit:

OWB_FORCE_FIPS_MODE_OFF=1 salt ${HOST_ID} file.remove ${LCL_BUPATH}/${FILE_NM}.sha256 --out=newline_values_only > /dev/null

JohnSnider · ‎2023-12-21

@LorenzoMadonna thanks for the catch, While that would not cause the command to fail, it was a typo. I'll be posting a new version later today with that and some other fixes as well.

@LucaBufalieri It appears after doing some testing that the timeout you added is making up for an issue with communications to the host, is it over a slow network link? that timeout is not for the completion of the job, but for the minion to respond to the command send by the master. so for some reason the hosts with the issues are responding slowly to the command. I run this script as is on global installs of 100's of nodes, some with rather slow responses, but have no issues with the backups completing.

I would look into other issues on the hosts. you can test the latency using the linux "time" function to see how long it takes for a simple command to return
time salt <MINIONID> test.ping
running this in my current environment the highest "real" time for the command to complete is ~ 9 seconds with most of them returning in < 2 seconds.

NetWitness Community

Centralized Backup and Restore of NetWitness Version 11.2+ (A Wrapper Script for NRT)

Updated for Version 12.x (backward compatible to 11.x)

Scenario -

Solution – A Wrapper for NRT

Overview –

Process Walkthrough –

Step 1: Run get-all-systems.sh

Step 2: Run ssh-propagate.sh

Step 3: Modify the nw-base.nrt file

Step 4: Run nw-backup.sh

NOTES –

Restore: nw-restore.sh

nw-restore.sh