Deployment Optional Setup Procedures
You can deploy NetWitness with the following options.
New Health and Wellness Search
Hybrid Categories on Series 6 (R640) Hardware
NW Server Deployment on ESA Hardware
Introduction to ESA Primary Disaster Recovery Failover
Perform ESA Primary Disaster Recovery Failover (Make Active)
Troubleshoot ESA Primary Disaster Recovery Failover Issues
Analyst User Interface
The Analyst User Interface (UI) gives you access to a subset of features in the NetWitness Platform UI that you can set up in individual locations when you deploy NetWitness Platform in multiple locations. It is designed to reduce latency and improve the performance that can occur when accessing all functionality from the Primary User Interface on the NW Server Host (Primary UI).
You can have multiple Analyst UI instances provisioned in the same manner as the other NW component hosts.
Features and Limitations
Each Analyst UI host:
- Can be deployed to specific organizational groups. For example: the Americas, EMEA, APAC, Tier 1 Analysts, Tier 3 Analysts.
- If Analyst UI hosts are deployed regionally, you have the capability of querying those regional brokers directly (less latency), instead of than having to route through the Primary UI.
- Helps distribute load off the Primary UI.
- Has its own Reporting Engine (RE).
- If it becomes unavailable for any planned or unplanned reason, it will not affect the Primary UI or any other Analyst UI instances.
- Provides the same pre-query filter verification, Data Privacy protection, and RBAC functionality as the Primary UI.
- Points back to the primary NW Server for authentication and configuration.
- Does not have access to any administrative functions. All administration functions take place on the Primary UI.
- Does not allow you to create or manage Content (that is, ESA rules, app rules, feeds). All Content creation and management takes place on the Primary UI.
Use Case
Large environments that include Geo distribution with a single data center and multiple NW Servers require Analyst UI instances in all their NetWitness locations or managed entities.
For example, if an Analyst UI is deployed for the EMEA SOC team, analysts can query their EMEA NetWitness Platform hosts directly. If the EMEA team has Broker hosts and Concentrator hosts within the region, the Analyst UI can connect and query them instead of connecting back to Primary user Interface (Primary UI).
Deployment
You must install the Analyst UI service category on a dedicated host, and you install it in the same manner as any component service category on a host.
See the "Task 2 - Install the latest version on Other Component Hosts" in the NetWitness Platform Installation Guides for instructions on how to install any component service. Go to the NetWitness All Versions Documents page and find NetWitness Platform guides to troubleshoot issues.
After you provision the Analyst UI host (that is after you run the nwsetup-tui for the component host designated for the Analyst UI), complete the following steps to install the Analyst UI service category on the provisioned host.
- Log in to NetWitness and go to
(Admin) > Hosts.
The New Hosts dialog is displayed with the Hosts view grayed out in the background.Note: If the New Hosts dialog is not displayed, click Discover in the Hosts view toolbar.
- Select the host in the New Hosts dialog and click Enable.
The New Hosts dialog closes and the host is displayed in the Hosts view. - Select that host in the Hosts view (for example, Analyst UI) and click
.
The Install Services dialog is displayed. - Select Analyst UI in Category and click Install.
- Configure NetWitness Platform for each Analyst UI instance.Go to the NetWitness All Versions Documents page and find NetWitness Platform guides to troubleshoot issues.
- Make sure that each Analyst UI instance is connected to the correct local Reporting Engine and has the appropriate Investigation parameters set. The Getting Started Guide for NetWitness Platform describes the default Analyst UI Dashboard and how you manage dashboards.
Note: You must add data sources to each Reporting Engine instance to execute Reports and Charts on an Analyst UI. See "Configure the Data Sources" in the Reporting Engine Configuration Guide for NetWitness Platform for instructions.
-
Configure whether to normalize alerts for any Respond Server (NW Server or Analyst UI) by enabling or disabling alert normalization. "Configure Analyst UI for Respond Server Alert Normalization" in the NetWitness Respond Configuration Guide for NetWitness Platform tells you how to configure Respond Server alert normalization for the Analyst UI.
- Make sure that each Analyst UI instance is connected to the correct local Reporting Engine and has the appropriate Investigation parameters set. The Getting Started Guide for NetWitness Platform describes the default Analyst UI Dashboard and how you manage dashboards.
Group Aggregation
You use Group Aggregation to configure multiple Archiver or Concentrator services as a group and share the aggregation tasks between them. You can configure multiple Archiver services or Concentrator services to efficiently aggregate from multiple Log Decoder services to improve query performance on the data:
- Stored in the Archiver.
- Processed through the Concentrator.
RSA Group Aggregation Deployment Recommendations
RSA recommends the following deployment for Group Aggregation:
- 1 - 2 Log Decoders
- 3 - 5 Archivers or Concentrators
Advantages of Using Group Aggregation
- Increases the speed of NetWitness queries.
- Improves the performance of aggregate queries (Count and Sum) on the environment.
- Enhances investigation service performance.
- Gives you the option of storing data for a longer duration for investigation purposes.
The following diagram illustrates Group Aggregation.
You can have any number of Archivers or Concentrators grouped together and form an aggregation group. The Archiver or Concentrator services in the group divide all the aggregated sessions between them based on the number of sessions defined in the Aggregate Max Sessions parameter.
For example, in an aggregation group containing two Archiver services or two Concentrator services with the Aggregate Max Sessions parameter set to 10,000, the services would divide the session between themselves as illustrated in the following table.
| Archiver 0 or Concentrator 0 | Archiver 1 or Concentrator 1 |
| 1 - 9,999 | 10,000 - 19,999 |
| 20,000 - 29,999 | 30,000 - 39,999 |
| 40,000 - 49,999 | 50,000 - 59,999 |
Configure Group Aggregation
Complete this procedure to configure multiple Archiver or Concentrator services as a group and share the aggregation tasks between them.
Prerequisites
Plan the network design for group aggregation. The following figure is an example of a group aggregation setup.
Ensure that you understand the Group aggregation parameters in the following table, and create a group aggregation plan.
| Parameter | Description |
|---|---|
|
Group Name |
It determines the group to which the Archiver or Concentrator belongs. |
|
Size |
It determines the number of Archiver or Concentrator services in the aggregation group. |
|
Member Number |
It determines the position of the Archiver or Concentrator in the aggregation group. For a group of size N, member number from 0 to N-1 must be set on each of the Archiver or Concentrators services in the aggregation group. |
|
Membership Mode |
There are two membership modes:
|
Note: The Membership Mode parameter has an effect only when no sessions have been aggregated from the service. After some sessions are aggregated, this parameter has no effect.
Set up Group Aggregation
This workflow shows the procedures you complete to configure group aggregation.
Complete the following steps to set up group aggregation.
- Configure multiple Archiver or Concentrator services in your environment. Make sure that you add the same Log Decoder as data source to all the services.
-
Perform the following on all the Archiver or Concentrator services that you want to be part of aggregation group:
- Go to
-
Select the Archiver or Concentrator service, and select
> View > Config.
The Service Config view of the Archiver or Concentrator is displayed.
- In the Aggregate Services section, select Log Decoder.
- Click
to change the status of the Log Decoder to offline if it is online.
-
Click
.
The Edit Aggregate Service dialog is displayed.
-
Click
.
The Edit Group Aggregation dialog is displayed.
-
Select the Enabled checkbox and set the following parameters:
- In the Group Name field, type the group name.
- In the Size field, select the number of Archiver or Concentrator services in the aggregation group.
- In the Member Number field, select the position of the Archiver or Concentrator in the aggregation group.
- In the Membership Mode drop-down menu, select the mode.
- Click Save.
- In the Service Config view, click Apply.
- Perform Step b to Step i on all other Archiver or Concentrator services that need to be part of group aggregation.
- Go to
-
In the Aggregation Configuration section, set the Aggregate Max Sessions parameter set to 10000.
New Health and Wellness Search
New Health and Wellness is an advanced monitoring and alerting system that provides insights on the operational state of the host and services in your deployment and helps identify potential issues.
System Requirements
The following tables list the memory, disk, and CPU recommended for the New Health and Wellness based on the size of the deployment.
Note: The recommended values might differ when you install and try the new features and enhancements.
Caution: If the New Health and Wellness node is on a different subnet, you must open the respective NetWitness Platform hosts port. For more information, see "New Health and Wellness on Different Subnet" section in the Network Architecture and Ports.
Standalone Virtual Host
Minimum memory for a standalone virtual host is 16 GB.
Each NetWitness platform host writes 150 MB of New Health and Wellness metrics data into Elasticsearch data per day. For example, if you have 45 NetWitness Platform hosts then 6.6 GB of metrics data is written to Elasticsearch per day.
| CPU | Memory |
|---|---|
| 4 cores | 16 GB |
Physical Host
| Deployment Size | Memory | CPU | DISK per day |
|---|---|---|---|
| Small (~5-10 hosts / 20-40 services) | 16 GB | 15% | 1.5 GB |
| Medium (~150-200 hosts / 300- 400 services) | 18 GB | 15% | 29 GB |
| Large (~250-300 hosts / 500-600 services) | 22 GB | 15% | 44 GB |
Based on the resources available, you can deploy the New Health and Wellness feature on any one of the following, listed in the order of preferred deployment method with most preferred first:
- Standalone virtual host (Most preferred recommendation to ensure no performance impact on any other functionality of deployed nodes)
- Physical host:
- Broker
- Admin Server
- ESA
Installing New Health and Wellness enables all hosts in your deployment to start sending metrics to monitor New Health and Wellness. After you deploy New Health and Wellness , see the "Monitor New Health and Wellness " topic in the System Maintenance Guide for instructions on how to configure and use this feature.
Please direct any New Health and Wellness feedback to nw.health.wellness.feedback@netwitness.com.
After you provision the New Health and Wellness host, complete the following steps to install the New Health and Wellness service category on the provisioned host.
- Log in to NetWitness and go to
The New Hosts dialog is displayed with the Hosts view grayed out in the background.Note: If the New Hosts dialog is not displayed, click Discover in the Hosts view toolbar.
Note: If you are not installing New Health and Wellness on a standalone virtual host ignore step 2.
- Select the host in the New Hosts dialog and click Enable.
The New Hosts dialog closes and the host is displayed in the Hosts view. - Select that host on which New Health and Wellness should be installed in the Hosts view (for example, New Health and Wellness) and click
The Install Services dialog is displayed. - Select New Health and Wellness in Category and click Install.
-
Refresh all hosts to write to elastic search.
-
SSH to the NW Server host.
- Run the following commands.
nw-manage --refresh-host --host-all
This may take few minutes for the changes to take effect based on the number of hosts in your deployment.
-
Note: (For Standalone Virtual host only) After you review your initial datastore configuration, you may determine that you need to add a new volume. For information on adding a new volume see “Add New Volume and Extend Existing File Systems” topic in the Virtual Host Installation Guide.
Note: After you have installed New Health and Wellness, for some reason, if you want to uninstall New Health and Wellness, you must refer to "Uninstall New Health and Wellness" in the Upgrade guide.
Hybrid Categories on Series 6 (R640) Hardware
You can install Hybrid Categories such as Log Hybrid and Network (Packet) Hybrid service categories on a Series 6 (R640) Physical host. This gives you the ability to attach multiple PowerVault external storage devices to the Series 6 (R640) Physical host.
NW Server Deployment on ESA Hardware
You now have the option to deploy the NW Server host on Series 6 Analytics hardware. The Series 6 Analytics Hardware has more memory and storage capacity than the standard Core appliance on which NW Server has typically been deployed. This results in better overall responsiveness and larger retention capacity for Report Engine.
Note: You can install the NW Server on ESA hardware, but you cannot co-locate any ESA services (categories) with the NW Server on this hardware.
Second Endpoint ServerSecond Endpoint Server
Complete the following procedure to deploy a second Endpoint Server.
- Set up a new host in NetWitness Platform.
- For a physical host, complete steps 1 to 16 in "Install NetWitness Platform" under "Installation Tasks" in the Physical Host Installation Guide for NetWitness Platform 12.5.
- For a virtual host, complete steps 1 to 6 in "Step 4. Install NetWitness Platform" in the Virtual Host Installation Guide for NetWitness Platform 12.5.
Go to the NetWitness All Versions Documents page and find NetWitness Platform guides to troubleshoot issues.
- SSH to the host that you set up in step 1.
- Submit the following command string.
mkdir -p /etc/pki/nw/nwe-caNote: You do not need to modify permissions.
- Copy the following two files from the previously deployed endpoint server to the new/second endpoint server:
/etc/pki/nw/nwe-ca/nwerootca-cert.pem
/etc/pki/nw/nwe-ca/nwerootca-key.pem - nstall Endpoint on the host.
-
Log in to NetWitness Platform and go to
The New Hosts dialog is displayed with the Hosts view grayed out in the background.Note: If the New Hosts dialog is not displayed, click Discover in the Hosts view toolbar.
-
Select the new host in the New Hosts dialog and click Enable.
The New Hosts dialog closes and the host is displayed in the Hosts view. -
Select that host in the Hosts view (for example, Endpoint Server II) and click
.
The Install Services dialog is displayed. - Select Endpoint in Host Type and click Install.
-
Warm Standby NW Server Host Warm Standby NW Server Host
The Warm Standby NW Server duplicates the critical components and configurations of your active NW Server host to increase reliability.
A secondary NW Server remains in the standby role and, when configured, receives backups of the primary NW Server in the active role at regular intervals. If the primary NW Server fails (goes offline), the fail-over procedure must be executed allowing the secondary NW Server to assume the active role.
In the latest version, we support a secondary NW Server that has a different IP address from the primary NW Server. Having the same IP address for both the primary NW Server and secondary NW Server is no longer necessary.
When you set up a secondary NW Server as a Warm Standby, a failure or scheduled switch from the primary NW Server to the secondary NW Server is referred to as a fail-over. You fail back to return to the normal operating state (that is, primary NW Server in the active role and the secondary NW Server in the standby role).
The following diagram illustrates the fail-over and fail-back process.
| 1 | Set up secondary NW Server as standby (initial setup). This is the normal operating state. |
| 2 | The primary NW Server fails over to the secondary NW Server. After the fail-over, get the primary NW Server back online and set it up in the standby role. This is a temporary operating state. |
| 3 | Fail the secondary NW Server back to the primary. The primary NW Server is back to the active role and secondary is back to the standby role. This is the normal operating state. |
Note: When you set up the secondary NW Server, follow the same administrative procedures, for example, for upgrade and maintenance, as the procedures for the primary active NW Server.
Procedures Procedures
Complete the following task to set up a secondary NW Server in the standby role for fail-over:
Complete the following tasks when required to maintain high availability:
Planned Fail-Over Scenario
This scenario occurs when you schedule a fail over (see Planned Fail-Over under step 3 in the Fail Over primary NW Server to Secondary NW Server procedure). You should not need do anything after the fail-over completes.
Required Fail-Over Scenario without Hardware Replacement
This scenario occurs when the primary NW Server fails (see Required Fail-Over under step 3 in the Fail Over Primary NW Server to Secondary NW Server topic), but you are able to recover it easily without re-imaging (for example, the active NW Server has corrupt or insufficient RAM). You do not need to run the nwsetup-tui and you do not need to contact NetWitness Customer Support to reestablish correct licensing when:
- The active (primary NW Server) fails over to the Standby (secondary NW Server) and that secondary host temporarily assumes the role of the active NW Server.
- You fix the problem with the primary NW Server (for example, install new RAM) and fail back to it from the secondary host.
Required Fail-Over Scenario with Hardware Replacement
This scenario occurs when the active NW Server completely fails and the hardware requires replacement, for example you receive a Return Merchandise Authorization (RMA). You need to run reconfigure the host with the nwsetup-tui and contact NetWitness Customer Support to reestablish licensing. If you choose to rebuild the replacement host as a temporary standby (for example, until your scheduled fail-back occurs), you must answer "Yes" to the Standby Host Recovery Mode nw-setup-tui prompt when configuring this temporary standby for failing back (see step 4 in the Set Up Secondary NW Server in Standby Role procedure for the context of this prompt).
Set Up Secondary NW Server in Standby RoleSet Up Secondary NW Server in Standby Role
- Before you install a secondary NW Server host for the standby role, make sure that:
- The primary NW Server is running 12.5.
- All component hosts are running 12.5
If you are:
- Installing NetWitness Platform 12.5, follow the instructions in the NetWitness Platform Physical Host Installation Guide for Version 12.5.
- Create a base image on the secondary NW Server:
- Attach media (ISO) to the host.
See the NetWitness Platform Build Stick Instructions for more information.- Physical media - use the ISO to create bootable flash drive media the Etcher® or another suitable imaging tool etch an Linux file system on the USB drive. See the NetWitnessBuild Stick Instructions for information on how to create a build stick from the ISO. Etcher is available at: https://etcher.io.
- iDRAC installations - the virtual media type is:
- Virtual Floppy for mapped flash drives.
- Virtual CD for mapped optical media devices or ISO file.
- Log in to the host and reboot it.
- Select F11 (boot menu) during reboot to select a boot device and boot to the connected media.
After some system checks during booting, the following Welcome to NetWitness Platform 12.5 installation menu is displayed. The menu graphics will render differently if you use a physical USB flash media. - Select Install NetWitness Platform 12.5 (default selection) and press Enter.
The Installation program runs and stops at the Enter (y/Y) to clear drives prompt that asks you to format the drives.
Caution: You must respond y or Y to this prompt even if the host does not have an internal RAID configuration or the installation will fail.
- Type Y to continue.
The default action is No, so if you ignore the prompt and it will select No in 30 seconds and will not clear the drives. The Press enter to reboot prompt is displayed.
The system displays all installation tasks it is performing. This can take a minute or so.After it completes the tasks, the installation program reboots the host.Caution: Do not reboot the attached media (media that contains the ISO file, for example a build stick).
- Log in to the host with the root credentials.
- Attach media (ISO) to the host.
- Run the nwsetup-tui command.
Note: 1.) When you navigate through the Setup program prompts, use the down and up arrows to move among fields, use the Tab key to move to and from commands (such as <Yes>, <No>, <OK>, and <Cancel>. Press Enter to register your command response and move to the next prompt.
2.) The Setup program adopts the color scheme of the desktop or console you use access the host.
3.) During the Setup program, when you are prompted for the network configuration of the host, be sure to specify the same network configuration that was used for the original installation of 11.x on this host (it must be exactly the same).This initiates the nwsetup-tui (Setup program) and the EULA is displayed.
- Tab to Accept and press Enter.
The Is this the host you want for your 12.5 NW Server prompt is displayed.
Your response to this prompt identifies a host as either the primary or secondary during a fresh install (and the selected response stays constant regardless of the current or future role, that is active or standby of the host). - Tab to Yes and press Enter.
The Install or Recover prompt is displayed. - Tab to 3 Install (Warm Standby) and press Enter.
The Standby Host Recovery Mode prompt is displayed.
- Tab to:
- No and press Enter to set up a secondary NW Server with the standby role (most common scenario).
- Yes and press Enter to set up a host that was previously used as a primary NW Server with the standby role so you can execute a fail-over and fail-back (less common scenario).
The NW Active Server IP Address prompt is displayed.
- Type the IP Address of the NW Server in the active role, tab to OK, and press Enter.
The Host Name prompt is displayed
Caution: If you include "." in a host name, the host name must also include a valid domain name.
- Press Enter if want to keep this name. If not edit the host name, tab to OK, and press Enter to change it.
The Master Password prompt is displayed.
Note: You must use the same Master and Deploy Admin credentials fot the Warm Standby NW Server Host that you used for the Active NW Server Host.
The following list of characters are supported for Master Password and Deployment Password:
- Symbols: ! @ # % ^ +
- Lowercase Characters: a-z
- Uppercase Characters: A-Z
No ambiguous characters are supported for Master Password and Deployment Password. For example: space { } [ ] ( ) / \ ' " ` ~ ; : .< > -
- Type the Password, down arrow to Verify, retype the password, tab to OK, and press Enter.
The Deployment Password prompt is displayed. - Type in the Password, down arrow to Verify, retype the password, tab to OK, and press Enter.
One of the following conditional prompts is displayed.- If the Setup program finds a valid IP Address for this host, the following prompt is displayed.
Press Enter if you want to use this IP and avoid changing your network settings. Tab to Yes and press Enter if you want to change the IP configuration found on the host. - If you are using an SSH connection, the following warning is displayed.
Note: If you connect directly from the host console, the following warning will not be displayed.
Press Enter to close warning prompt. - If the Setup Program found an IP configuration and you chose to use it, the Update Repository prompt is displayed. Go to step 12 to and complete the installation.
- If the Setup Program did not find an IP configuration or if you chose to change the existing IP configuration, the Network Configuration prompt is displayed.
- If the Setup program finds a valid IP Address for this host, the following prompt is displayed.
- Tab to OK and press Enter to use Static IP.
If you want to use DHCP, down arrow to 2 Use DHCP and press Enter.
The Network Configuration prompt is displayed. - Down arrow to the network interface you want, tab to OK, and press Enter. If you do not want to continue, tab to Exit.
The following Static IP Configuration prompt is displayed. - Type the configuration values (using the down arrow to move from field to field), tab to OK, and press Enter. If you do not complete all the required fields, an All fields are required error message is displayed (secondary DNS Server and Local Domain Name fields are not required). If you use the wrong syntax or character length for any of the fields, an Invalid <field-name> error message is displayed.
Caution: If you select DNS Server, make sure that the DNS Server is correct and the host can access it before proceeding with the installation.
The Update Repository prompt is displayed.
- Press Enter to choose the Local Repo on the NW Server.
If you want to use an external repo, down arrow to External Repo, tab to OK, and press Enter.- If you select 1 The Local Repo (on the NW Server) in the Setup program, make sure that you have the appropriate media attached to the host (media that contains the ISO file, for example a build stick) from which it can install NetWitness 12.5
- .0.0. If the program cannot find the attached media, you receive the following prompt.
- If you select 2 An External Repo (on an externally-managed server), the UI prompts you for a URL. The repositories give you access to NetWitness updates and CentOS updates. Refer to "Appendix B. Create an External Repo" in the Physical Host Installation Guide for instructions on how to create this repo and its external repo URL so you can enter it in the following prompt.
Enter the base URL of the NetWitness external repo and click OK. The Start Install prompt is displayed.
See "Set Up an External Repository with RSA and OS Updates" under "Hosts and Services Procedures" in the NetWitness Platform Hosts and Services Getting Started Guide for instructions. Go to the NetWitness All Versions Documents page and find NetWitness Platform guides to troubleshoot issues.
The Disable firewall prompt is displayed.
- Tab to No (default), and press Enter to use the standard firewall configuration. Tab to Yes, and press Enter to disable the standard firewall configuration.
If you select Yes, confirm your selection(select Yes again) or select No to use the standard firewall configuration.
The Start Install/Upgrade prompt is displayed. - Press Enter to install 12.5 on the NW Server.
When Installation complete is displayed, you have installed the 11.7 NW Server on this host.Note: Ignore the hash code errors similar to the errors shown in the following figure that are displayed when you initiate the nwsetup-tui command. Yum does not use MD5 for any security operations so they do not affect the system security.
- License the secondary NW Server.
-
Log in to the secondary NW Server User Interface, click
- SSH to the primary NW Server.
- Edit the /opt/netwitness/flexnetls/local-configuration.yaml file and add the back up hostid (that is, the License Server ID ).
This is an example of the section of the local-configuration.yaml file before you add the License Server ID.
# Hostid of the backup server, if in fail over configuration.
#backup-hostid:
This is an example of the section of the local-configuration.yaml file after you add the MAC address (for example, 000c2918c80d) of the Warm Standby NW Server Host.
# Hostid of the backup server, if in fail over configuration.
backup-hostid: "000c2918c80d"
- Restart the fneserver service.
systemctl restart flexnetls-RSALM
- (Conditional) If your NetWitness Platform deployment is prohibited from accessing the Internet (Air Gap), you must:
- Download the capability request from NetWitness Platform User Interface.
- Upload the request to FNO.
- Upload the response from FNO to the NetWitness Platform User Interface.
-
- Schedule the backup of the primary NW Server and the copying of this backed-up data to the secondary NW Server.
- SSH to the primary NW Server.
- Submit the following commands.
/opt/rsa/saTools/bin/schedule-standby-admin-data-sync -d -i <warm-standby-admin-server-ip>
Note: If the Warm Standby server becomes active, and if it will be accessed by any NW host with a NAT IP address, the NAT IP address must be added to the command:
/opt/rsa/saTools/bin/schedule-standby-admin-data-sync -d -i <warm-standby-admin-server-ip>-p<warm-standby-NAT-admin-server-ip>This backs up the primary NW Server data and copies the backup archive file to the secondary NW Server daily for future fail-over use. It also schedules the backup and copy to execute on a daily basis. You can display help for the schedule-standby-admin-data-sync script with the following command string.
/opt/rsa/saTools/bin/schedule-standby-admin-data-sync –-help
This returns the following help to which you can refer to customize the host data backup (such as backup frequency).
Schedule Data Synch between AdminServer and Standby AdminServer
Script also executes a synchronization each time.
Usage:
schedule-standby-admin-data-sync command [options]
Commands:-h, --help Display Help -d, --daily Schedule daily data synchronization -w, --weekly Schedule weekly data synchronization -c, --custom <crontab formatted> Schedule custom data synchronization i.e. to schedule for midnight on 1st and 10th of the month: '0 0 1,10 * *' -i, --standby-ip <IP address> (required) IP address of standby NW Server -p, --standy-ip-public <NAT IP address> (optional) NAT-based IP address of standby NW Server -v, --verbose Enable verbose output
Fail Over Primary NW Server to Secondary NW Server with Same IP AddressFail Over Primary NW Server to Secondary NW Server with Same IP Address
Initially, the primary NW Server fails over to the secondary NW Server. When the primary NW Server is back up, the secondary NW Server fails over to the primary NW Server, and that is referred to as a fail-back. When it is possible for the secondary NW Server to have the same IP address as the primary NW Server after failover, complete the following procedure to fail over from the primary NW Server to the secondary NW Server.
- SSH to the secondary NW Server.
IMPORTANT: If you have installed New Health & Wellness on Admin Server, perform the following:
- Install New Health & Wellness on Standby NW server.
- Back up the Search category (New Health and Wellness ) from Primary NW server to standby NW server using the NRT:
nw-recovery-tool --export --dump-directory <Backup_directory_path> --category Search
You must manually copy files from Primary NW server to Standby NW server after running the above command.
- Run the nw-failover script with the --make-active argument:
nw-failover --make-active - Complete the following steps for planned and required fail-over.
- SSH to the primary NW Server.
- Run the fail-over script with the --make-standby argument to assign the standby role to the primary NW Server:
nw-failover --make-standby - Shut down the primary NW Server.
Note: If you have a catastrophic failure, you may need to provision a new host or re-image the primary NW Server and complete the Set Up secondary NW Server in Standby Role procedure for this host to create a new primary NW Server, so that you can fail back to it.
-
Validate that the component hosts have connected to the new active NW Server by running the following command on the new active NW Server:
nw-manage --check-hosts-status
Note: Do not continue to the next step until all component hosts have successfully connected to the new active NW Server. Rerun nw-manage --check-hosts-status as needed to continue verification.
-
Restart the Respond-server service once the failover operation has been successfully completed.
-
Follow the IP address change procedures for the now-active NW Server in "Change Host Network Configuration" in the System Maintenance Guide. Go to the NetWitness All Versions Documents page and find NetWitness Platform guides to troubleshoot issues.
Note: If the primary NW Server is going to remain online, you must update its IP address so that it no longer conflicts with the Warm Standby NW Server IP address when it is changed. Follow the IP address change procedures for the now-active NW Server in "Change Host Network Configuration" in the System Maintenance Guide. Go to the NetWitness All Versions Documents page and find NetWitness Platform guides to troubleshoot issues.
- Shut down the primary NW Server, or reboot the primary NW Server and set it up as the new secondary NW Server by completing the steps in Set Up Secondary NW Server in Standby Role for this host to create a new primary NW Server, so that you can fail back to this host.
IMPORTANT: After successful fail over to Standby NW server, restore the Search category (New Health & Wellness) using NRT:
nw-recovery-tool --import --dump-directory <Backup_directory_path> --category Search
Fail Over Primary NW Server to Secondary NW Server with Different IP AddressFail Over Primary NW Server to Secondary NW Server with Different IP Address
If your secondary NW Server will maintain a different IP address from your primary NW Server after a fail-over event, for example, if the secondary NW Server is located in a different data center from the primary NW Server, follow these steps to fail over from the primary NW Server to the secondary NW Server.
- SSH to the secondary NW Server.
IMPORTANT: If you have installed New Health & Wellness on Admin Server, perform the following:
- Install New Health & Wellness on Standby NW server.
- Back up the Search category (New Health and Wellness ) from Primary NW server to standby NW server using the NRT:
nw-recovery-tool --export --dump-directory <Backup_directory_path> --category Search
You must manually copy files from Primary NW server to Standby NW server after running the above command.
- Run the failover script:
nw-failover --make-active - Complete the following steps for planned and required fail-over.
- SSH to the primary NW Server.
- Run the fail-over script with the --make-standby argument to assign the standby role to the primary NW Server:
nw-failover --make-standby - Shut down the primary NW Server.
Note: If you have a catastrophic failure, you may need to provision a new host or re-image the primary NW Server and complete the Set Up secondary NW Server in Standby Role procedure for this host to create a new primary NW Server, so that you can fail back to it.
- If you are running in a mixed-version environment, log in to each component host that is still on a version prior to 11.7 and execute the following:
- netconfig --update-dns --dns <active-nw-server-ip-address>
- sed -Ei 's/^master:.*/master: <active-nw-server-ip-address>/g' /etc/salt/minion
- systemctl restart salt-minion
Note: If you have a component host that uses a NAT IP address to reach the NW Server, substitute the <NAT-IP-address> for the <active-nw-server-ip-address> in both bullet items above.
- Restart the Respond-server service once the failover operation has been successfully completed.
- Shut down the primary NW Server, or reboot the primary NW Server and set it up as the new secondary NW Server by completing the steps in Set Up Secondary NW Server in Standby Role for this host to create a new primary NW Server, so that you can fail back to this host.
IMPORTANT: After successful fail over to Standby NW server, restore the Search category (New Health & Wellness) using NRT:
nw-recovery-tool --import --dump-directory <Backup_directory_path> --category Search
Note: If the secondary NW Server had any custom certificates, you must re-apply them after failover. For information about how to re-apply custom certificates, see "(Optional) Use a Custom Server Certificate" in the System Security and User Management Guide. Go to the NetWitness All Versions Documents page and find NetWitness Platform guides to troubleshoot issues.
Note: After upgrading the NW Server host or a component host to the latest version, review the contents of the /etc/hosts.user file for any obsolete host entries. The /etc/hosts.user file contains system and user-generated entries that are not managed by NetWitness Platform. However, entries from /etc/hosts.user are merged with NetWitness Platform-generated host mappings to create and update /etc/hosts. To avoid conflicts with NetWitness Platform-generated mappings, and to avoid generating connectivity errors resulting from an IP address change, NetWitness recommends that you remove any entries in /etc/hosts.user that include a non-loopback IP address of a NetWitness Platform host. After updating /etc/hosts.user, you must refresh the system by running the following command:
nw-manage --refresh-host --host-key <ID, IP, hostname or display name of host>
Note: While changing a host's IP address or during failover, component hosts can become disconnected from NW Server hosts. Follow these steps to reconnect a host system to its NW Server system.
1. Log in to the component host using SSH or the console.
2. Run the command nw-manage --override-nws-ip --ipv4 <current IP address of the NW Server.
When this command completes, the component host is reconnected to the NW Server at the specified IP address.
Note:
- Before performing failover, make sure that the deploy_admin password of the Standby NW server is not expired. You can verify this by logging in to the Standby NW server nw-shell.
nw-shell
offline » login
user: deploy_admin
password: <deploy_admin_password>
WARNING: Login failure: User credentials have expired (The following warning message confirms that your deploy_admin credentials have expired)
Next step, update the deploy_admin password using nw-manage --update-deploy-admin-pw
After the command is successfully executed, verify the deploy_admin password using security-cli-client --quiet --get-config-prop --prop-hierarchy nw.security-client --prop-name platform.deployment.password
- Before performing failover, make sure that your Standby NW server certificates are up to date.
Follow the steps in the sections that apply to your environment.
- SSO
- Reporting Engine
- UCF
- PAM
- ECAT
- RSA NetWitness Orchestrator (By Demisto)
- Audit Logging
- Health and Wellness
- Malware Analysis
- Windows Legacy Collection
SSO
Update Configuration for Single Sign-On
Note: You must disable SSO configurations ONLY when NW Server IP is changed.
When the host network is configured with a new IP address, the SSO configurations also must be updated.
To do this:
- Disable the SSO configuration using nw-shell after failover from new IP.
To resolve this issue you must disable SSO manually, using the following commands:
- SSH to admin server node.
- Connect to nw-shell.
- Connect to admin server service using the connect --service admin-server command.
- Log in to admin server using the login command.
- Enter the admin username and password.
- Execute the following commands:
- cd /rsa/security/authentication/web/saml/sso-enabled
- set false
- logout
- exit
- systemctl restart rsa-nw-admin-server
-
Change the host IP address to the new IP.
- Generate the new metadata and reupload it in ADFS. For more information, see the "Configure SAML 2.0 provider settings for portals" topic in the Microsoft documentation.
For more information, see the "Troubleshooting" topic in the System Security and User Management Guide.
Reporting Engine
Update Configuration for Reporting Engine
Note: You must update the Reporting Engine configurations ONLY when NW Server IP is changed.
When the host network is configured with a new IP address, you must update and verify the Reporting Engin configurations. The hostname for NetWitness configurations under the Output Actions must be updated with the new IP.
To manually configure the new IP, perform the following steps:
- Log in to NetWitness Platform.
-
Navigate to
- Click the Output Actions tab.
- Add the new IP address in the Hostname field.
- Click Apply.
UCF
To enable UCF to communicate with NetWitness Platform:
-
On the UCF server, execute the runConnectionManager.bat file (the same file that is used for adding connection details).
-
Select Option #2, Edit endpoints.
-
Select the NW Server connection from the options that are displayed.
-
When you are prompted for Host Address (the old IP address is shown in parentheses) enter the new IP address.
Note: Do not change any other setting.
PAM
If you have PAM configured, after the failover, you must configure the system again using the instructions in the "Configure PAM Login Capability" topic in the System Security and User Management Guide. Go to the NetWitness All Versions Documents page and find NetWitness Platform guides to troubleshoot issues.
ECAT
Update the following services:
- Log in to the NetWitness Endpoint user interface and go to Configure > Monitoring and External Components Configuration > Incident Message Broker.
- Update the server Hostname and IP Address to the current active server and test the settings.
- Log in to the NetWitness Endpoint user interface and go to Configure > Monitoring and External Components Configuration > NetWitness Suite.
- Update the server Hostname and IP address to the current active server and test settings.
If you are forwarding syslog messages to a NetWitness Platform Log Decoder, update the syslog server settings to point to the new IP address of the Log Decoder host.
- Log in to the NetWitness Endpoint user interface and go to Configure > Syslog Server.
- Select logdecoder, and in Server Hostname/IP, enter the new IP address of the Log Decoder host.
RSA NetWitness Orchestrator (By Demisto)
Update the Current Active NW Server to Fetch Respond Incidents and Alerts
- Log in to Orchestrator and go to Settings > server&services.
- Edit the NetWitness instance by updating the server URL to the current active NW Server to fetch respond incidents and alerts.
Update Component Hosts Acting as Data Sources
If you change the IP address of a component host, for example, a Concentrator, Network or Log Decoder, or Broker, that is acting as data source to the Orchestrator, update the following settings to point to the new IP address of the host.
- Log in to Orchestrator and go to Settings > server&services and select the component host.
- Enter the new IP address of the component host in Server URL and click Done.
Audit Logging
If you have changed the IP address of the NW Server, you must reconfigure audit logging. For instructions, see "Configure Global Audit Logging" in the System Configuration Guide.
Health and Wellness
If you have any Health and Wellness rules that contain IP addresses that have been changed, you must update those rules with the new IP addresses. For information about managing Health and Wellness rules, see "Monitor Health and Wellness using NetWitness Platform UI" in the System Maintenance Guide.
Malware Analysis
Source host IP address changes are not updated in the NetWitness user interface for Malware Analysis continuous scan configurations. You must manually update this configuration in the Malware Analysis Config view > General > Continuous Scan Configuration and update the source host IP address to the new host IP address.
Windows Legacy Collection
On occasion, you may need to change the IP address of your Windows Legacy Collector. You may also need to edit any Destination Groups that you have configured.
Change WLC IP Address
The following procedure describes how to change the IP address for your system.
- Log onto the Windows Legacy Collector system and manually change the IP address on the system.
- In the UI, confirm that the Log Collector service corresponding to the WLC system shows up in error (Red). It might take some time for it to reflect the changed status.
-
On the NetWitness Server, use the nw-manage utility to view the host information for the WLC using the following command:
nw-manage --list-hosts
Sample output from running the command is shown here:
{
"id" : "fdb8150c-e040-459e-8cc5-3c60ec2c65ae",
"displayName" : "WLC-HOST-104",
"hostname" : "10.101.216.102",
"ipv4" : "10.101.216.102",
"ipv4Public" : null
} ]You use the value of "id" from your output in the following step.
-
Use the nw-manage utility to change the IP address of the WLC. For the host-id argument, use the value for the "id" that you noted from step 3. For the ipv4 value, use the new IP Address to which you are changing.
nw-manage --update-host --host-id "fdb8150c-e040-459e-8cc5-3c60ec2c65ae" --ipv4 10.101.216.105
- After you see the message that the previous command ran successfully, go to the NetWitness Server UI and verify that the WLC service is running without any errors.
Edit Destination Groups For Log Collectors and VLCs
The Windows Legacy Collector is often configured with Destination Groups to forward events to Log Collectors or Virtual Log Collectors. If the IP address of any such Destination LC or VLC is changed, the Windows Legacy Collector can no longer forward events. To remediate this, you must edit the Destination groups for the WLC, making sure to select the new LC or VLC IP Address.
Note: If you added any content to the /etc/hosts file on the primary server, the contents of that file are available under /var/netwitness/standby-data/unmanaged/etc on the failover server. You can manually copy those files to the /etc/hosts file on the failover server after the failover is complete.
Fail Back Secondary NW Server to Primary NW ServerFail Back Secondary NW Server to Primary NW Server
After a fail-over from the primary NW Server to the secondary NW Server, you need to fail back to your original setup of the primary NW Server in the active role and the secondary NW Server in the standby role.
Essentially, you follow the same steps described under Fail Over Primary NW Server to Secondary NW Server to fail back to your original setup (that is primary NW Server-active and secondary NW Server-standby). The difference is that you now need to fail over from the secondary NW Server to the primary NW Server.
Note: The secondary SA becomes active in case of a fail over. You should re-run the schedule-standby-admin-data-sync from the secondary SA before failing back to the primary SA.
Introduction to ESA Primary Disaster Recovery Failover
NetWitness platform enables administrators to perform smooth failovers with minimal processes and tools involved in the entire design, making it easier for the clients to minimize downtime during service interruptions.
Administrators can set up an ESA Primary StandBy Node in the event of a disaster or unplanned outage of the original active ESA Primary node. Recovery involves switching from the active ESA Primary node to ESA Primary Standby node by taking Mongo and configuration backups of the active ESA Primary system and restoring them to the ESA Primary Standby with the required configurations to ensure uninterrupted access to ESA correlation and context hub services.
The new ESA Primary Standby node should be configured with its unique IP and host details, ensuring a seamless setup process not hindered by prior configurations.
Note: More than 1 ESA Primary Standby is not supported.
Prerequisites
-
Active ESA Primary System
-
ESA Primary Standby System
-
The system configuration of the ESA Primary Standby System must be the same as the system configuration of the active ESA Primary System.
Workflow
The following figure is a high-level workflow illustrating the step-by-step tasks you can perform to complete ESA Primary Disaster Recovery Failover and Failback process.
-
Install the ESA Primary Standby Node (initial setup) and set up the data sync from Active ESA Primary to ESA Primary Standby Node.
-
Perform ESA Primary Disaster Recovery Failover to make ESA Primary Standby node as the active node.
-
If the older Active ESA Primary is recovered after a disaster, set up the data sync from the current Active ESA Primary to the older Active ESA Primary (ESA Primary Standby Node) and then perform Failback. This is an optional step.
For more information on the workflow, see ESA Primary Disaster Recovery Failover Use Case Example.
Install ESA Primary Standby
You must install a new VM / HW appliance on a dedicated host in the same manner as you install any component service category on a host.
See the "Install NetWitness Platform" in the NetWitness Platform Documentation for instructions on how to install any component service. Follow the steps to deploy for ESA Primary Standby component.
After you provision the VM / HW appliance (that is after you run nwsetup-tui for the component host designated for the ESA Primary Standby UI), complete the following steps to setup the ESA Primary Standby service category on the provisioned host.
-
Log in to NetWitness and go to
(Admin) > Hosts.
The New Hosts dialog is displayed with the Hosts view grayed out in the background.
If the New Hosts dialog is not displayed, click Discover in the Hosts view toolbar.
- Select the host in the New Hosts dialog and click Enable.
The New Hosts dialog closes and the host is displayed in the Hosts view.
-
Select that host in the Hosts view (for example, ESA Primary Standby) and click
.
The Install Services dialog is displayed
- Select ESA Primary Standby in Category and click Install.
- During the installation, the ESA Primary Standby services are enabled. Proceed with the installation once the ESA Primary Standby services are disabled.
Note: - The ESA Primary Standby services can be activated only after the failover. At any point of time, only one ESA Primary node should be active and running.
Note: - At any point of time if you want to know which ESA Primary node is active, you must run the following command.
nw-failover-esa -- show-state
Caution: In CCM, the Correlation servers from both active ESA Primary node and ESA Primary Standby node will be available. However, you must only configure your deployments, policies, and groups with the Correlation server service running on the current active ESA primary node.
