Disaster Recovery

[#1]

Disaster Recovery (Backup and Restore Instructions)

You can take a backup and restore of NetWitness Hosts using any of the following:

• (Recommended) NetWitness Recovery Wrapper Tool

• NetWitness Recovery Tool (NRT)

(Recommended) NetWitness Recovery Wrapper Tool

Note: NetWitness Recovery Wrapper tool is supported from NetWitness 11.6.1.4 and later. In case of host with large volume of data (>500GB), NetWitness recommends to use NetWitness Recovery Tool (nw-recovery-tool) for backup.

NetWitness Recovery Wrapper Tool (NRWT) provides centralized backup and restore that makes it easy for you to take a backup of all the supported installation options (Physical Host, Virtual Host, AWS, and Azure).

With NRWT you can:

• Backup (export) an individual, a specific, or all hosts at a time.

• Restore (import) an individual host at a time.

• Customize files or folders during backup and restore.

• Copy back up data to/from remote host location from/to Netwitness hosts provided:

• Remote host is reachable via SSH from each NetWitness hosts.

• The credentials are correct.

• The location specified has sufficient space to take a backup in case of export.

• The location specified should contain valid backup data in case of import.

• (For version 11.7.1 and later) Back up Mongo databases for Endpoint and ESA instances.

• (For version 11.7.1 and later) Include Broker index for NetWitness node in which Broker service is running.

• (For version 11.7.1 and later) Back up custom files and folders provided by user.

• (For version 12.3 and later) Avoid entering the password in the Command Line Interface (CLI) while exporting and importing the data.

• (Optional) (For version 12.3 and later) Log in to the NetWitness Server or any other component host systems as a non-root user and perform backup and recovery of data. You must use the following login credentials to log in to the NetWitness Server or any other component host systems.

  • Username: nwnrt

  • Password: netwitness

  Note: To log in as a non-root user to the NetWitness Server or any other component host systems, root users must use the username su nwnrt.

• (For version 12.3 and later) Back up Group Hosts and Category Hosts.

For details on previous run, check NRWT logs at /var/log/netwitness/recovery-tool/nw-recovery-wrapper.log on the Admin Server.

Basic Usage of the NetWitness Recovery Wrapper Tool

You can use the NRWT to back up data by using the export option. To restore data, use the import option. The basic usage of the tool is to run the following command from the root directory level:

nw-recovery-wrapper [command] [option]

The commands and options that you can use with this tool are described in the following tables.

Commands and Options	Description
-h --help	Display help on commands and option. For example, specify: nw-recovery-wrapper --help to get a list of supported operations and arguments.
-e, --export	Export data or configuration.
-i, --import	Import data or configuration.
-d, --dump-dir <path>	Path for the where data will be exported or imported from (for example, /var/netwitness/backup).
--host-key HOST_KEY [HOST_KEY ...]	Host IP, ID or display name.
--host-all	Specify for all hosts - supported only for export.
--category-group CATEGORY_GROUP [CATEGORY_GROUP ...]	Specify for host and service groups - supported only for export.
--host-group HOST_GROUP [HOST_GROUP ...]	Specify host group created on the UI hosts page - supported only for export.
--include CUSTOM_PATH [CUSTOM_PATH ...]	Custom path or file.
--remote-location REMOTE_LOCATION	Remote host path for remote host configuration.
--remote-ip REMOTE_IP	Remote host IP for remote host configuration.
--remote-password REMOTE_PASSWORD	Remote host password for remote host configuration.
--remote-user REMOTE_USER	User for remote host configuration. (Optional) user for remote host configuration. If not specified, defaults to root user.

Required Conditions

• Make sure that there is adequate disk space on dump directory to take the backup on each NetWitness Hosts.

• Valid Host key is entered. Host key can be Host ID, IP address or display name.

Back Up using NRT Wrapper:

1. Backup NetWitness Hosts and store on local dump directory of each hosts:
   nw-recovery-wrapper export --dump-dir <dir> --host-key <Host 1 IP/ID/Name> <Host 2 IP/ID/Name>......<Host N IP/ID/Name>

   nw-recovery-wrapper export --dump-dir <dir> --host-all

   Note: If you have logged in with the username nwnrt or su nwnrt, you must enter sudo before the commands you run while performing the backup and recovery actions on the NetWitness Server Host or any other Component Hosts using the NetWitness Recovery Wrapper Tool.
   For Example: To backup NetWitness hosts using the NetWitness Recovery Wrapper Tool, the first step is to run the following command after logging in.
   sudo nw-recovery-wrapper export --dump-dir <dir> --host-key <Host 1 IP/ID/Name> <Host 2 IP/ID/Name>......<Host N IP/ID/Name>

2. (Optional) Add custom files or folders during backup and restore other than what is predefined in recovery tool:

   Note: Make sure the custom files or directories are available on NetWitness Hosts, if not, the files or directories will be ignored.

   nw-recovery-wrapper export --dump-dir <dir> --include-file <custom files>/--include-dir --host-key <Host 1 IP/ID/Name> <Host 2 IP/ID/Name>......<Host N IP/ID/Name>
   nw-recovery-wrapper export --dump-dir <dir> --include-file <custom files>/--include-dir --host-all

3. (Optional) Copy backup data to remote Location:

   Note: Make sure that:
   - You specify valid values for --remote-ip, --remote-location arguments for remote copy operation.
   - Remote Host IP is valid and reachable via SSH from all NetWitness Hosts.
   - Remote Host location (--remote-location) has adequate space to take backup.

   nw-recovery-wrapper export --dump-dir <dir> --host-key <Host 1 IP/ID/Name> <Host 2 IP/ID/Name>......<Host N IP/ID/Name> --remote-ip <IP ADDRESS of remote host> --remote-location <remote-location-where-backups-should-be-copied-to>
   nw-recovery-wrapper export --dump-dir <dir> --host-all --remote-ip <IP ADDRESS of remote host> --remote-location <remote-location-where-backups-should-be-copied-to>

   Note:
   - Optional argument --remote-user defaults to root if you do not specify any value.
   - Optional argument --remote-password <remote-password> uses ssh keys if argument is not specified.

   Note: To perform password-less export, follow these steps on all the NetWitness nodes:
   1. ssh-keygen (Without passphrase)
   2. ssh-copy-id <remote - username>@<remote - ip>
   Confirm the ssh connection by performing step 3 and exit from the remote machine.
   3. ssh <remote - username>@<remote - ip>

   Example:

   For adminserver, the backup folder name will be adminserver-backup-2021-09-08-12:48:13

4. Backup (export) including custom files or folders and copy to remote location:

   Note: Make sure that:
   - the custom files or directories are available on NetWitness Hosts, if not, the files or directories will be ignored.
   - You specify valid values for --remote-ip, --remote-location arguments for remote copy operation.
   - Remote Host IP is valid and reachable via SSH from all NetWitness Hosts.
   - Remote Host location (--remote-location) has adequate space to take backup.

   nw-recovery-wrapper export --dump-dir <dir> --include <custom files/folder> --host-key <Host 1 IP/ID/Name> <Host 2 IP/ID/Name>......<Host N IP/ID/Name> --remote-ip <IP ADDRESS of remote host> --remote-location <remote-location-where-backups-should-be-copied-to>

   nw-recovery-wrapper export --dump-dir <dir> --include <custom files/folder> --host-all --remote-ip <IP ADDRESS of remote host> --remote-location <remote-location-where-backups-should-be-copied-to>

   Note:
   - Optional argument: --remote-user defaults to root if argument is not specified.
   - Optional argument --remote-password <remote-password> uses ssh keys if argument is not specified.

   Note: To perform password-less export, follow these steps on all the NetWitness nodes:
   1. ssh-keygen (Without passphrase)
   2. ssh-copy-id <remote - username>@<remote - ip>
   Confirm the ssh connection by performing step 3 and exit from the remote machine.
   3. ssh <remote - username>@<remote - ip>

Example:

For Admin server, the backup folder name will be adminserver-backup-2021-09-08-12:48:13

5. (For version 11.7.1 and later) (Optional) Include Mongo service.

   Note: Make sure that:
   - Mongo service is running on the NetWitness host.
   - --host-all and --host-key with multiple values are not supported for include Mongo operation.

   nw-recovery-wrapper export --dump-dir <dir> --host-key <Host 1 IP/ID/Name> --include-mongo

6. (For version 11.7.1 and later) (Optional) Include Broker index.

   Note: Make sure that:
   - Broker service is running on the NetWitness host.

   nw-recovery-wrapper export --dump-dir <dir> --host-key <Host 1 IP/ID/Name> <Host 2 IP/ID/Name>......<Host N IP/ID/Name> --include-broker-index

   nw-recovery-wrapper export --dump-dir <dir> --host-all --include-broker-index

7. (For version 11.7.1 and Later) (Optional) Backup (export) including Mongo and Broker index.

   Note: Make sure that:
   - Mongo service is running on the NetWitness host.
   - Broker service is running on the NetWitness host.
   - --host-all and --host-key with multiple values are not supported for include Mongo operation.

   nw-recovery-wrapper export --dump-dir <dir> --host-key <Host 1 IP/ID/Name> --include-mongo --include-broker-index

8. (For version 11.7.1 and later) (Optional) Backup (export) including custom files or folders, copying to remote location, Broker index and Mongo.

   nw-recovery-wrapper export --dump-dir <dir> --include-broker-index --include-mongo ---include-file <custom files>/--include-dir <custom folders> --host-key <Host 1 IP/ID/Name> --remote-ip <IP ADDRESS of remote host> --remote-location <remote-location-where-backups-should-be-copied-to>

   Note: To perform password-less export, follow these steps on all the NetWitness nodes:
   1. ssh-keygen (Without passphrase)
   2. ssh-copy-id <remote - username>@<remote - ip>
   Confirm the ssh connection by performing step 3 and exit from the remote machine.
   3. ssh <remote - username>@<remote - ip>

9. (For version 12.3 and later) Back up all the hosts specific to a given group created on the /admin/appliances page.

   nw-recovery-wrapper export --dump-dir <dump location> --host-group <UI host group>

   Example:

   nw-recovery-wrapper export --dump-dir /var/netwitness/Test-backup --host-group TestGroup

10. (For version 12.3 and later) Back up all the hosts specific to a given category such as Log Hybrid, Log Collector, and Standalone Broker in the environment.

    nw-recovery-wrapper export --dump-dir <dump location> --category-group <category name>

    Example:

    nw-recovery-wrapper export --dump-dir /var/netwitness/Test-backup --category-group LogDecoder

    Note: Make sure that:
    - Custom files or directories are present on NetWitness hosts to be backedup, if it is not present it skips the files or directory.
    - Fields such as --remote-ip, --remote-location are mandatory for remote copy operation.
    - Remote host IP credentials should be valid and reachable via SSH from all NetWitness hosts.
    - Remote host location (--remote-location) should have sufficient space to contain backups.
    - Mongo service is running on the NetWitness host.
    - Broker service is running on the NetWitness host.
    - --host-all,--host-key, --category-group, and --host-group with multiple values are not supported for include Mongo operation.
    - Optional argument --remote-password <remote-password> uses ssh keys if argument is not specified.

Restore (import) options supported in NRT Wrapper

Caution: Use import commands carefully as it performs system level changes.

1. Restore (import) single host at a time (using IP address, Host name, or Host ID).
   nw-recovery-wrapper import --dump-dir <dir> --host-key <Host IP/ID/Name>

2. Restore custom files or folders (if any).

   Note: Make sure the custom files or directories are available on NetWitness Hosts, if not, the files or directories will be ignored.

   nw-recovery-wrapper import --dump-dir <dir> --include-file <custom files>/--include-dir --host-key <Host IP/ID/Name>

3. Restore from a remote location.

   Note: Make sure that:
   - --remote-location contains remote host location in which data is backedup.
   - Remote Host IP is valid and reachable via SSH from all NetWitness Hosts.
   - Remote Host location (--remote-location) has adequate space to take backup.

   nw-recovery-wrapper import --remote-ip <IP address of remote host> --remote-location <location-of-backup-on-remote-host> --dump-dir <dir> --host-key <Host IP/ID/Name>

   Note:
   - Optional argument: --remote-user defaults to root if argument is not specified.
   - Optional argument --remote-password <remote-password> uses ssh keys if argument is not specified.

   Note: To perform password-less import, follow these steps on all the NetWitness nodes:
   1. ssh-keygen (Without passphrase)
   2. ssh-copy-id <remote - username>@<remote - ip>
   Confirm the ssh connection by performing step 3 and exit from the remote machine.
   3. ssh <remote - username>@<remote - ip>

   Example, for adminserver, the backup folder name should be adminserver-backup-2021-09-08-12:48:13
   nw-recovery-wrapper import --dump-directory <dir> --host-key <host-1> --remote-ip <remote-ip> --remote-location /home/adminserver-backup-2021-09-08-12:48:13

   Note:
   - Optional argument: --remote-user defaults to root if argument is not specified.
   - Optional argument --remote-password <remote-password> uses ssh keys if argument is not specified.

4. Note: To perform password-less import, follow these steps on all the NetWitness nodes:
   1. ssh-keygen (Without passphrase)
   2. ssh-copy-id <remote - username>@<remote - ip>
   Confirm the ssh connection by performing step 3 and exit from the remote machine.
   3. ssh <remote - username>@<remote - ip>

5. Restore data from remote location including custom files or folders.

   Note: Make sure that:
   - The custom files or directories are available on NetWitness Hosts, if not, the files or directories will be ignored.
   - --remote-location contains remote host location in which data is backedup.
   - Remote Host IP is valid and reachable via SSH from all NetWitness Hosts.
   - Remote Host location (--remote-location) has adequate space to take backup.

   nw-recovery-wrapper import --dump-dir <dir> --include <custom files/folder> --host-key <host1> --remote-ip <IP ADDRESS of remote host> --remote-location <remote-location-where-backups-should-be-copied-to>

   Example, for Admin Server, the backup folder name will be adminserver-backup-2021-09-08-12:48:13

   Note:
   - Optional argument: --remote-user defaults to root if argument is not specified.
   - Optional argument --remote-password <remote-password> uses ssh keys if argument is not specified.

6. Note: To perform password-less import, follow these steps on all the NetWitness nodes:
   1. ssh-keygen (Without passphrase)
   2. ssh-copy-id <remote - username>@<remote - ip>
   Confirm the ssh connection by performing step 3 and exit from the remote machine.
   3. ssh <remote - username>@<remote - ip>

7. (For version 11.7.1 and later) (Optional) Restore Mongo service.

   Note: Make sure that:
   - Mongo service is running on the NetWitness host.
   - --host-all and --host-key with multiple values are not supported for include Mongo operation.

   nw-recovery-wrapper import --dump-dir <dir> --host-key <Host 1 IP/ID/Name> --include-mongo

8. (For version 11.7.1 and later) (Optional) Restore Broker index.

   Note: Make sure that:
   - Broker service is running on the NetWitness host.
   - --host-all option is not support for include broker index operation.

   nw-recovery-wrapper import --dump-dir <dir> --host-key <Host 1 IP/ID/Name> --include-broker-index

9. (For version 11.7.1 and later) (Optional) Restore Mongo and Broker index.

   Note: Make sure that:
   - Mongo service is running on the NetWitness host.
   - Broker service is running on the NetWitness host.
   - --host-all and --host-key with multiple values are not supported for include Mongo operation.

   nw-recovery-wrapper import --dump-dir <dir> --host-key <Host 1 IP/ID/Name> --include-mongo --include-broker-index

10. (For version 11.7.1 and later) (Optional) Restore custom files or folders, copying to remote location, Broker index, and Mongo.

    Note: Make sure that:
    - Custom files or directories are present on NetWitness hosts to be backedup, if it is not present the files or directory is skipped for backup.
    - Fields such as --remote-ip, --remote-location are mandatory for remote copy operation.
    - Remote host IP credentials should be valid and reachable via SSH from all NetWitness hosts.
    - Remote host location (--remote-location) should have sufficient space to contain backups.
    - Mongo service is running on the NetWitness host.
    - Broker service is running on the NetWitness host.
    - --host-all and --host-key with multiple values are not supported for include Mongo operation.

    nw-recovery-wrapper import --dump-dir <dir> --include-file <custom files>/--include-dir <custom folders> --include-mongo --include-broker-index --host-key <host1> --remote-ip <IP ADDRESS of remote host> --remote-location <remote-location-where-backups-should-be-copied-to>

    Note:
    - Optional argument: --remote-user defaults to root if argument is not specified.
    - Optional argument --remote-password <remote-password> uses ssh keys if argument is not specified.

    Note: To perform password-less import, follow these steps on all the NetWitness nodes:
    1. ssh-keygen (Without passphrase)
    2. ssh-copy-id <remote - username>@<remote - ip>
    Confirm the ssh connection by performing step 3 and exit from the remote machine.
    3. ssh <remote - username>@<remote - ip>

Status Check

You can check the Backup or Restore status using the below command.

/var/log/netwitness/recovery-tool/recovery.log

Troubleshooting

Error Message	NRT Wrapper fails during backup or restore.
Solution	Do any one of the following: Log in to host where backup is failing and check /var/log/netwitness/recovery-tool/recovery.log. Run in debug mode (nw-recovery-wrapper -l debug) on Node 0 to get recovery logs of each host.

Error Message	NRT Wrapper fails due to incorrect password for remote copy operation (--remote-password).
Cause	NRWT fails if you enter wrong password multiple times during remote copy. Since SFTP uses SSH, it locks the system SSH for a while.
Solution	You must retry after sometime.

Error Message	NRT Wrapper fails after running for long hours for a particular host but the backup is still in progress. For example, Endpoint or ESA node.
Cause	NRT architecture is designed to take a backup of configuration and not user data. It uses salt to communicate between admin server(SA) and nodex. Due to large volume of data in the particular host, this issue is caused when salt communication times out.
Solution	SSH to the particular host and check the backup status at /var/log/netwitness/recovery-tool/recovery.log. If the data to be backed up is huge, recommend using "nw-recovery-tool" by logging into particular host.

NetWitness Recovery Tool (NRT)

You can use the NetWitness Recovery Tool (NRT) to back up and restore data from the NetWitness Server and component host systems. The NRT is a script that you run from the command line to back up and restore data on hosts for RMAs, hardware refreshes, and general backup and restore requirements. Refer to Disaster Recovery in Azure Deployment for specific steps on how to perform disaster recovery for hosts deployed in Azure VMs.

Note: You must run the NRT on each host system locally. You cannot run it from remote hosts or an external host.

The following types of hosts can be backed up and restored.

Note: In the NRT script, the following terms in bold are referred to as categories.

• NetWitness Admin Server (may include Broker, Investigate, Respond, Health and Wellness, and Reporting Engine)
• AnalystUI (may include Broker, Investigate, Respond, Reporting Engine)

• Archiver Log Archiver (Workbench and Archiver)
• Broker Stand-alone Broker
• Concentrator Network or Log Concentrator
• Decoder Network Decoder (Packets)
• Endpoint Endpoint Agents
• Endpoint Broker Endpoint Broker

• Endpoint Log Hybrid Log Collector, Log Decoder, Endpoint Server, and Concentrator

• Event Stream Analysis (ESA) Primary Contexthub, ESA Correlation, and Incident Management database
• ESA Secondary ESA Correlation
• Gateway Cloud Gateway
• Log Hybrid Retention Log Hybrid-Retention Optimized (for RSA Series 6 Hybrid hardware with Log Hybrid-Retention Optimization)
• Log Collector Log Collector including Virtual Log Collector if installed
• Log Decoder Log Decoder including Local Log Collector and Warehouse Connector, if installed
• Log Hybrid Log Collector, Log Decoder, and Concentrator
• Malware Malware Analysis and Broker
• Network Hybrid Concentrator and Decoder
• Search (for Health & Wellness Beta Host)
• UEBA User Entity and Behavior Analytics
• Warehouse Warehouse Connector

Basic Usage of the NetWitness Recovery Tool

You can use the NRT to back up data by using the export option. To restore data, use the import option. The basic usage of the tool is to run the following command from the root directory level:

nw-recovery-tool [command] [option]

The commands and options that you can use with this tool are described in the following tables.

Commands and Options	Description
-h, --help	Display help on commands and option. For example, specify: nw-recovery-tool --help-categories to get a list of all the valid category names.
-e, --export	Export data or configuration.
-i, --import	Import data or configuration.
-d, --dump-dir <path>	Path for the where data will be exported or imported from (for example, /var/netwitness/backup).
-C, --category <name>	Select components by category. Valid category names are AdminServer, AnalystUI, Archiver, Broker, Concentrator, Decoder, Endpoint, EndPointBroker, EndpointLogHybrid, ESAPrimary, ESASecondary, Gateway, LogHybridRetention, LogCollector, LogDecoder, LogHybrid, Malware, NetworkHybrid, Search, UEBA, and Warehouse. You can specify a single category or multiple categories if multiple categories are co-located on the same host. For example: --category AdminServer for the Admin Server exclusively. --category AdminServer --category Gateway for the Admin Server and the Cloud Gateway. --category ESAPrimary for the ESA Primary exclusively. --category Broker for the Broker exclusively. --category Broker --category EndpointBroker for the Broker and the Endpoint Broker.
--remote-location <path>	Remote backupfile path.
--remote-ip <IP>	Remote machine IP.
--remote-user <name>	Remote machine Username, (optional) user for remote host configuration. If not specified, defaults to root user.
--remote-password <pass>	Remote machine password.

Required Conditions

Make sure that the following conditions are met:

• Read the entire document before backing up any data. The document covers all deployment scenarios, so you want to make sure you have all the information required to back up and restore your implementation of NetWitness Platform before going through this process.
• Run the NRT for both backup and recovery locally, on each system being backed up or restored. You cannot run the NRT on an external host, or back up or restore several hosts simultaneously. However, you can back up several components on the same host system simultaneously.
• Export and import data on the same host. If a host fails and you need to build a new system, the new system must have the same identity parameters (i.e., the same IP address), and must be on the same version of NetWitness Suite.

• Make sure that there is adequate disk space in the backup location (/var/netwitness/backup is the recommended directory) before the export command in the nw-recovery tool is executed. Do not use a tmp directory because it fills up quickly and may cause the system to crash.

• Check the sizing of the Malware disks and adjust them before you back them up. The following table shows you the maximum size of Malware databases that you can back up by hardware type with the actions you can take to reduce them to the maximum size.

  Host	Source Hardware	Target Hardware	Database	Maximum Size for Backup	Actions to Reduce Size to Backup Maximum
  Malware	Series 4S Hybrid	Series 6 Core	/var/netwitness	2.5TB	Configure a rollover. Purge data that you do not need from the database.

• Restore to the exact ISO Image that each host had at the time of backup.
• If you have multiple services co-located on a single host, include all the services in a single command string for the import and export commands in the nw-recovery tool.

Note: When you run the NRT, the Malware , Reporting Engine, and Postgresql services are stopped and restarted during both the backup (export) and restore (import) processes

Disaster Recovery Workflow

The following diagram shows the high-level Disaster Recovery tasks.

Note: You only need to recover a host if it failed. This means that you can recover a single host, or any combination of hosts depending on which host or hosts failed.

The following diagram shows the tasks for:

• Backup (perform as soon as possible and as frequently as possible).
• Restore (only required if you need to restore your data).

Back Up and Restore Data for Hosts

The procedures for backing up and restoring data are different for NetWitness Server host systems and for component systems.

Caution: 1.) Do not remove component hosts (that is any host other than the NW Server host) from the Hosts View ((Admin) > Hosts) from the user interface when you are performing the following disaster recovery procedure. 2.) You must retain (restore) the ‘Host name’ that existed prior to performing the disaster recovery procedure.

Back Up and Restore Data on the NetWitness Server

Note: If you are using shared storage to export data from multiple hosts (for example, a shared mount or drive), use host-specific subfolders for the path to the location of the exported files for each host, to avoid overwriting one host’s exported data with another. For example, you could use a path similar to --dump-dir /mnt/storage/<host-specific-name> for the path to the location of the exported files.

Back Up Data on a NetWitness Server Host

Perform this procedure on an existing, functional NetWitness Server host system.

1. At the root level, type the following command:

   nw-recovery-tool --export --dump-dir /var/netwitness/backup --category AdminServer

   Note: If you have logged in with the username nwnrt or su nwnrt, you must enter sudo before the commands you run while performing the backup and recovery actions on the NetWitness Server Host or any other Component Hosts using NetWitness Recovery Tool.
   For Example: To backup the data on a NetWitness Server host using the NetWitness Recovery Tool, the first step is to run the following command after logging in.
   sudo nw-recovery-tool --export --dump-dir /var/netwitness/backup --category AdminServer

   Note: If a service is co-located with another category on the same host rather than on its own, dedicated host, you must include it in the command string. The Gateway and EndpointBroker can be co-located as show in the following examples:
   nw-recovery-tool--export --dump-dir /var/netwitness/backup --category AdminServer --category Gateway
   nw-recovery-tool--export --dump-dir /var/netwitness/backup --category Broker --category EndpointBroker

   nw-recovery-tool --export --dump-dir /tmp/dump --category ESAPrimary

2. Replace /var/netwitness/backup with the path to the location to which the data should be exported.

   1. Ensure that this location has sufficient space to store the backup data.
   2. The backup directory path should be located on the local host. However, the backup files could be located on a network mount or an external device.

The data is backed up on the NetWitness Server host in the location you set up in step 2.

3. Move the backed up data from the local host to an external server or a USB stick.

4. (Optional) Copy backup data to remote Location:

   Note: Make sure that:
   - You specify valid values for --remote-ip, --remote-location a rguments for remote copy operation.
   - Remote Host IP is valid and reachable via SSH from all NetWitness Hosts.
   - Remote Host location (--remote-location) has adequate space to take backup.

   nw-recovery-tool --export --dump-dir /var/netwitness/backup --category AdminServer --remote-ip <IP ADDRESS of remote host> --remote-location <remote-location-where-backupsshould-be-copied-to>

   Note:
   - Optional argument --remote-user defaults to root if you do not specify any value.
   - Optional argument --remote-password <remote-password> uses ssh keys if argument is not specified.

   Note: To perform password-less export, follow these steps on all the NetWitness nodes:
   1. ssh-keygen (Without passphrase)
   2. ssh-copy-id <remote - username>@<remote - ip>
   Confirm the ssh connection by performing step 3 and exit from the remote machine.
   3. ssh <remote - username>@<remote - ip>

Restore Data on a NetWitness Server Host

1. Re-image the NetWitness Server host using the same network configuration settings of the original host. For information about re-imaging the NetWitness Server host, see "Task 1 - Install 12.3 on the NetWitness Server Host" in the Physical Host Installation Guide for Version 12.3 Guide

   1. Optional If you need to establish network connectivity before you can fetch backup data, for example, if it is on a remote host, run the following script using the same IP address, subnet, gateway, DNS and domain information as the original host:

      netconfig --static --interface <name> --ip <address> --netmask <netmask> --gateway <gateway>

      For example:

      netconfig --static --interface eth0 --ip 192.168.1.100 --netmask 255.255.255.0 --gateway 192.168.1.1

      Optional: To specify DNS server(s), include the following additional parameter:

      --dns <address>

      Optional: To set the local domain name, include the following additional parameter:

      --domain <name>

   2. (Optional) If you are using DHCP, run the following script:

      netconfig --dhcp --interface <name>

      For example:

      netconfig --dhcp --interface eth0

   3. Add the backup data to the backup directory path on the local host, for example:

      /var/netwitness/backup

2. Run the nwsetup-tui command. This initiates the Setup program.

   Note: During the Setup program, when you are prompted for the network configuration of the host, be sure to specify the same identical network configuration that was used for the original installation of the host.

3. When you are prompted, select install type option 2: Recover (Reinstall), click OK, and then enter the path to the backup directory containing the backup data.
   
   
   
   The file path provided in the recovery path location should be in folder structure. If it is tar.gz file or any compressed file, it needs to be extracted. The tar.gz file can be extracted using the command “tar -zxvf /root/backup.tar.gz” if required.
   The path can be /root/ or /var/netwitness/backup or any similar path.
   The backup file for admin-server will have folders as listed below.
   

4. After the installation completes successfully, ensure that the host is running the exact same release and patch version of the data that was backed up:

   • If the data was on an system that was updated to a later patch release, update the host by following the instructions for updating systems offline in the update guide for the same patch version as what was previously running on the host (the exact release or patch version for which data was backed up).

   • If the data was on a major release version (for example, 12.3) that had not been updated to a later patch version, you do not need to update the host system.

5. When the host is running at the correct version, run the following command on the NetWitness Server to restore data:

   nw-recovery-tool --import --dump-dir /var/netwitness/backup --category AdminServer

   Note: If a service is co-located with another category on the same host rather than on its own, dedicated host, you must include it in the command string. The Gateway and EndpointBroker can be co-located as show in the following examples:
   nw-recovery-tool--import --dump-dir /var/netwitness/backup --category AdminServer --category Gateway
   nw-recovery-tool--import --dump-dir /var/netwitness/backup --category Broker --category EndpointBroker

6. (Optional) Restore backup from a remote location:

   Note: Make sure that:
   - You specify valid values for --remote-ip, --remote-location arguments for remote copy operation.
   - Remote Host IP is valid and reachable via SSH from all NetWitness Hosts.
   - Remote Host location (--remote-location) has adequate space to take backup.

   nw-recovery-tool --import --dump-dir /var/netwitness/backup --category AdminServer --remote-ip <IP ADDRESS of remote host> --remote-location <location-of-backup-on-remote-host>

   Note:
   - Optional argument --remote-user defaults to root if you do not specify any value.
   - Optional argument --remote-password <remote-password> uses ssh keys if argument is not specified.

   Note: To perform password-less export, follow these steps on all the NetWitness nodes:
   1. ssh-keygen (Without passphrase)
   2. ssh-copy-id <remote - username>@<remote - ip>
   Confirm the ssh connection by performing step 3 and exit from the remote machine.
   3. ssh <remote - username>@<remote - ip>

7. (Conditional) If you use custom firewall rules or custom entries in /etc/hosts:
   1. (Conditional) If you use custom firewall rules (that is, replied "Yes" to the "Disable Firewall" nwsetup-tui prompt during installation), restore the /etc/sysconfig/iptables file from the backup copy located in the <dump-dir>/unmanaged/etc/sysconfig/iptables file.
   2. (Conditional) If you use custom entries in /etc/hosts, restore the /etc/hosts.users file from the backup copy located at <dump-dir>/unmanaged/etc/hosts.user to /etc on the host.
   3. If you performed step 7a or 7b, refresh the host by running the following command:
      nw-manage --refresh-host --host-key <ID, IP, hostname or display name of host>
8. Reboot the NetWitness Server host.

Note: If you want to add any more custom entries to /etc/host, you must add them to the /etc/hosts.users file and then refresh the host as described in step 6c.

Back Up and Restore Data on Other Component Hosts

Perform these procedures on each existing, functional component host system.

Back Up Data on a Component Host

1. At the root level, type the following command:
   nw-recovery-tool --export --dump-dir /var/netwitness/backup --category <category name>

   where the category name is one of the following:
   AdminServer, AnalystUI, Archiver, Broker, Concentrator, Decoder, Endpoint, EndPointBroker, EndpointLogHybrid, ESAPrimary, ESASecondary, Gateway, LogHybridRetention, LogCollector, LogDecoder, LogHybrid, Malware, NetworkHybrid, Search, UEBA,or Warehouse

   Note: If you have logged in with the username nwnrt or su nwnrt, you must enter sudo before the commands you run while performing the backup and recovery actions on the NetWitness Server Host or any other Component Hosts using NetWitness Recovery Tool.
   For Example: To backup the data on a component host using the NetWitness Recovery Tool, the first step is to run the following command after logging in.
   sudo nw-recovery-tool --export --dump-dir /var/netwitness/backup --category <category name>

Note: 1.) Use the category that matches the host type. 2.) If services are co-located on a Component Host rather than on its own dedicated host, you must include it in the command string. For example, a Warehouse Connector resides on a Log Decoder host. The following is an example of this command string.
nw-recovery-tool--export --dump-dir /var/netwitness/backup --category LogDecoder --category Warehouse

2. (Optional) Copy backup data to remote Location:

Note: Make sure that:
- You specify valid values for --remote-ip, --remote-location arguments for remote copy operation.
- Remote Host IP is valid and reachable via SSH from all NetWitness Hosts.
- Remote Host location (--remote-location) has adequate space to take backup.

nw-recovery-tool --export --dump-dir /var/netwitness/backup --category <category name> --remote-ip <IP ADDRESS of remote host> --remote-location <remote-location-where-backupsshould-be-copied-to>

Note:
- Optional argument --remote-user defaults to root if you do not specify any value.
- Optional argument --remote-password <remote-password> uses ssh keys if argument is not specified.

Note: To perform password-less export, follow these steps on all the NetWitness nodes:
1. ssh-keygen (Without passphrase)
2. ssh-copy-id <remote - username>@<remote - ip>
Confirm the ssh connection by performing step 3 and exit from the remote machine.
3. ssh <remote - username>@<remote - ip>

3. (Optional) Replace /var/netwitness/backup with the path to the location to which the data should be exported:
   1. Ensure that this location has sufficient space to store the backup data.
   2. The backup directory path should be located on the local host. However, the backup files could be located on a network mount or an external device.
4. For Endpoint Log Hybrid and ESA Primary hosts, you can export application data that is stored in the database by running the following command:
   nw-recovery-tool --export --dump-dir /var/netwitness/backup --component mongo
   You can replace /var/netwitness/backup with the path to the location to which the data should be exported.

Note: 1.) Make sure that there is enough space in the export location for the files from the Mongo database. 2.) You can back up the Endpoint Log Hybrid or ESA Primary host data and Mongo database in a single command string. For example, nw-recovery-tool --export --dump-dir /var/netwitness/backup --category EndpointLogHybrid --component mongo

5. For Malware, you can export application data from the Malware application database by running the following command:
   nw-recovery-tool --export --dump-dir /var/netwitness/backup --component postgresql
   You can replace /var/netwitness/backup with the path to the location to which the data should be exported.

Note: Ensure that there is enough space in the export location for the files from the Malware database.

6. Move the backed up data from the local host to an external server or a USB stick.

Restore Data on a Component Host

1. Re-image the component host using the same network configuration settings of the original host. For information about re-imaging a component host, see "Task 2 - Install 12.3 on Other Component Hosts" in the Physical Host Installation Guide for Version 12.3 Guide

2. Optional If you need to establish network connectivity before you can fetch backup data, for example, if it is on a remote host, run the following script using the same IP address, subnet, gateway, DNS and domain information as the original host:
   netconfig --static --interface <name> --ip <address> --netmask <netmask> --gateway <gateway>
   For example:
   netconfig --static --interface eth0 --ip 192.168.1.100 --netmask 255.255.255.0 --gateway 192.168.1.1
   Optional: To specify DNS server(s), include the following additional parameter:
   --dns <address>
   Optional: To set the local domain name, include the following additional parameter:
   --domain <name>

   1. (Optional) If you are using DHCP, run the following script:
      netconfig --dhcp --interface <name>
      For example:
      netconfig --dhcp --interface eth0
   2. Add the backup data to the backup directory path on the local host, for example, /var/netwitness/backup.
3. Run the nwsetup-tui command. This initiates the Setup program.

Note: During the Setup program, when you are prompted for the network configuration of the host, be sure to specify the same identical network configuration that was used for the original installation of the host.

4. When you are prompted, select install type option 2: Recover (Reinstall), click OK, and then enter the path to the directory containing the backup data.
   The file path provided in the recovery path location should be in folder structure. If it is tar.gz file or any compressed file, it needs to be extracted. The tar.gz file can be extracted using the command “tar -zxvf /root/backup.tar.gz” if required.
   The path can be /root/ or /var/netwitness/backup or any similar path.

5. After you complete the nwsetup-tui command setup, you must re-install the appropriate services on the host using the Install command from the Hosts View in the NetWitness Platform User Interface.

6. After the service installation completes, ensure that the host is running the exact same release and patch version of the data that was backed up:
   • If the data was on system that was updated to a later patch release, update the host by following the instructions for updating systems offline for the same patch version as what was previously running on the host (the exact release or patch version for which data was backed up).
   • If the data was on a major release version (for example, 12.3) that had not been updated to a later patch version, you do not need to update the host system.
7. When the host is running at the correct version, return to the root level of the component host and run the following command to restore data:
   nw-recovery-tool --import --dump-dir /var/netwitness/backup --category <category name>

   Note: If services are co-located on a Component Host rather than on its own dedicated host, you must include it in the command string. For example, a Warehouse Connector resides on a Log Decoder host. The following is an example of this command string.
   nw-recovery-tool--import --dump-dir /var/netwitness/backup --category LogDecoder --category Warehouse

8. (Optional) Restore backup data from remote Location:

Note: Make sure that:
- You specify valid values for --remote-ip, --remote-location a rguments for remote copy operation.
- Remote Host IP is valid and reachable via SSH from all NetWitness Hosts.
- Remote Host location (--remote-location) has adequate space to take backup.

nw-recovery-tool --import --dump-dir /var/netwitness/backup --category <category name> --remote-ip <IP ADDRESS of remote host> --remote-location <location-of-backup-on-remote-host>

Note:
- Optional argument --remote-user defaults to root if you do not specify any value.
- Optional argument --remote-password <remote-password> uses ssh keys if argument is not specified.

Note: To perform password-less export, follow these steps on all the NetWitness nodes:
1. ssh-keygen (Without passphrase)
2. ssh-copy-id <remote - username>@<remote - ip>
Confirm the ssh connection by performing step 3 and exit from the remote machine.
3. ssh <remote - username>@<remote - ip>

9. For EndpointLogHybrid and ESAPrimary systems, you can import application data to be restored by running the following command:
   nw-recovery-tool --import --dump-dir /var/netwitness/backup --component mongo
10. For Malware, you can import application data from the Malware application database to be restored by running the following command:
    nw-recovery-tool --import --dump-dir /var/netwitness/backup --component postgresql
11. For a Decoder, Log Decoder , Concentrator, Archiver, Network Hybrid, or Log Hybrid configured with external storage (that is, DAC, SAN, Unity or Powervault):
    1. Scan the <dump-dir>/unmanaged/etc/fstab file for devices with mount points that do not exist in the system /etc/fstab file.

    IMPORTANT: If you are migrating to new host hardware (that is a new Decoder, Log Decoder, Concentrator, Archiver, Network Hybrid, or Log Hybrid host), before you proceed to the next step you must:
    1. Power off the old hardware host and the external storage device attached to it.
    2. Attach the external storage device to the new host hardware.
    3. Power on both the new host hardware and the external storage device attached to it.

    1. Complete the following steps for each device in the backup copy of <dump-dir>/unmanaged/etc/fstab.
       1. Verify that the corresponding device is present and attached. If it not attached, attach it. If the device is no longer applicable, skip it and go to the next device.

       2. Verify that the mount point directory exists on the file system. If it does not exist, create the directory with the mkdir <path> command.

       3. Add the fstab entry from the backup copy to the system /etc/fstab file.

          Caution: For a Series 6 Hybrid, you must restore backed up data to the /etc/fstab directory according to the instructions in Appendix A. Modify fstab for Series 6 Hybrid Storage After Recovery.

    2. Run the following command on each host.
       mount -a
12. (Conditional) If you use custom firewall rules or custom entries in /etc/hosts:
    1. (Conditional) If you use custom firewall rules (that is, replied "Yes" to the "Disable Firewall" nwsetup-tui prompt during installation), restore the /etc/sysconfig/iptables file from the backup copy located in the <dump-dir>/unmanaged/etc/sysconfig/iptables file.
    2. (Conditional) If you use custom entries in /etc/hosts, restore the /etc/hosts.users file from the backup copy located at <dump-dir>/unmanaged/etc/hosts.user to /etc on the host.
    3. If you performed step 12a or 12b, refresh the host by running the following command:
       nw-manage --refresh-host --host-key <ID, IP, hostname or display name of host>
13. Reboot the component host.

Hardware Refresh Only - Use Additional Space in New Hardware Hosts

Refer to the Core Database Tuning Guide for NetWitness Platform for instructions on how to use all the space you have available on your new hardware. Go to the NetWitness All Versions Documents page and find NetWitness Platform guides to troubleshoot issues.