Post Upgrade Tasks for 11.7.2

[#1]

Post Upgrade Tasks

This topic is divided into two sections. Complete the tasks in one of the following sections based on your upgrade path:

• Post Upgrade Tasks for Customers Upgrading from version 11.7.x.x
• Post Upgrade Tasks for Customers Upgrading From 11.5.3.2 or 11.6.x.x

Post Upgrade Tasks for Customers Upgrading from version 11.7.x.x

Complete the tasks that apply to the hosts in your environment.

• General
• Event Stream Analysis (ESA)
• Respond
• Reference Log Decoder
• User Entity Behavior Analytics

General

(Conditional) Configure NAT-Based IP Addresses

If you have a host, such as a VLC, that requires a NAT-based IP address in order to connect to the NW Server host, you must update the host configuration with the following steps.

1. Log in to the host that requires the use of NAT IP addresses, using the console or SSH.
2. Run the following command:
   nw-manage --enable-nat-usage
3. To set the NAT address for the NW Server:
   1. Log into the NW Server using the console or SSH.
   2. Run the following command:
      nw-manage -update-host --host-id <UUID of NW Server> --ipv4-public <NAT IP of NW Server>

   Note: You can find the UUID and view the current NAT IP address of the host by running nw-manage --list-hosts.

(Conditional - For Warm-Standby Hosts Only) Register the Secondary IP Address of Warm-Standby Hosts

The Warm-Standby server must be upgraded to 11.5 or later before completing the following steps.

1. Log in to the NW Server using the console or SSH.
2. Run the following command:
   nw-manage --add-nws-secondary-ip --ipv4 <ip address of Warm/Standby Server>

Note: If the Warm-Standby server requires a NAT-based IP address (IPv4-public) for any host to access it during failover, the NAT IP address must also be registered by running the following command: nw-manage --add-nws-secondary-ip --ipv4 <NAT-based IP address of Warm Standby Server>

3. Verify the correct Warm Standby host IP address value by running the following command:
   nw-manage --get-nws-secondary-ip

Review Contents of /etc/hosts.user for Obsolete Host Entries

After upgrading the NW Server host or a component host, review the contents of the /etc/hosts.user file for any obsolete host entries. The /etc/hosts.user file contains system and user-generated entries that are not managed by NetWitness Platform. However, entries from /etc/hosts.user are merged with NetWitness Platform-generated host mappings to create and update /etc/hosts. To avoid conflicts with NetWitness Platform-generated mappings, and to avoid generating connectivity errors resulting from an IP address change, NetWitness recommends that you remove any entries in /etc/hosts.user that include a non-loopback IP address of a NetWitness Platform host.

After updating /etc/hosts.user, you must refresh the system by running the following command:
nw-manage --refresh-host --host-key <ID, IP, hostname or display name of host>

Jetty Configuration

For Jetty Configuration and related information, see Manage Custom Host Entries topic in the System Maintenance Guide.

Make Sure Services Have Restarted and Are Capturing and Aggregating Data

Make sure that services have restarted and are capturing data (this depends on whether or not you have auto-start enabled).

If required, restart data capture and aggregation for the following services:

• Decoder
• Log Decoder
• Broker
• Concentrator
• Archiver

To Start Network Capture:

1. In the NetWitness Platform menu, go to (Admin) > Services.
   The Services view is displayed.
2. Select each Decoder service.

3. Under (actions), select View > System.

4. In the toolbar, click 

To Start Log Capture:

• 1. In the NetWitness Platform menu, go to (Admin) > Services.
     The Services view is displayed.
  2. Select each Log Decoder service.
  3. Under (actions), select View > System.
  4. In the toolbar, click 

To Start Aggregation:

1. In the NetWitness Platform menu, go to (Admin) > Services.

   The Services view is displayed.

2. For each Concentrator, Broker, and Archiver service:

   1. Select the service.
   2. Under (actions), select View > Config.
   3. In the toolbar, click 

Event Stream Analysis (ESA)

Mixed mode is not supported for ESA hosts in NetWitness Platform version 11.5 and later. The NetWitness server, ESA primary host, and ESA secondary host must all be on the same NetWitness Platform version.

There are no post-upgrade tasks required for ESA. For ESA troubleshooting, see ESA Troubleshooting Information.

If you want to add support for Endpoint, UEBA, and Live content rules, you must update the multi-valued and single-valued parameter meta keys on the ESA Correlation service to include all the required meta keys. It is not necessary to make these adjustments during the upgrade; you can make the adjustments later at a convenient time. For detailed information and instructions, see "Update Your ESA Rules for the Required Multi-Value and Single-Value Meta Keys" in the ESA Configuration Guide

Show Updates to an ESA Rule Deployment

You can view changes to an ESA rule deployment, such as adding or removing rules. When there is a change to a deployment, the update icon ( ) appears next to the name of the deployment in the Rules tab options panel.

To show updates to an ESA rule deployment:

1. Go to (Configure) > ESA Rules.The Rules tab is displayed.
2. In the options panel, under Deployments click Show Updates on the far right.
3. Click Deploy Now.

If you are unable to deploy the ESA rule, see Known Issues for the workaround.

Respond

The Primary ESA server must be upgraded to 11.7.2.0 before you can complete these tasks.

Note: After upgrading the primary NW Server (including the Respond Server service), the Respond Server service is not automatically re-enabled until after the Primary ESA host is also upgraded to 11.7.2.0. The Respond post-upgrade tasks only apply after the Respond Server service is upgraded and is in the enabled state.

(Conditional) Restore Any Respond Service Custom Keys in the Aggregation Rule Schema

Note: If you did not manually customize the incident aggregation rule schema, you can skip this task.

If you added custom keys in the var/lib/netwitness/respond-server/data/aggregation_ rule_schema.json file for use in the groupBy clause for 11.x, modify the /var/lib/netwitness/respond-server/data/aggregation_rule_schema.json file and add the custom keys from the automatic backup file.

The backup file is located in /var/lib/netwitness/respond-server/data and it is in the following format:
aggregation_rule_schema.json.bak-<time of the backup>

Reference Log Decoder

Make sure your reference Log Decoder is upgraded to 11.5 or later versions. If you never set up a reference Log Decoder, no action is required. For details, see the Log Parser Customization Guide.

User Entity Behavior Analytics

IMPORTANT: Before the upgrade, if you encountered and resolved the task failure issues, then after the upgrade, you must replace the authentication.json file before you run the post-upgrade tasks. The task failure issues in Airflow and their solutions are described in the 'Troubleshooting' topic of the UEBA Configuration Guide.

IMPORTANT: Every UEBA deployment when upgraded requires additional steps to complete the upgrade process. When you upgrade from 11.5.x to 11.5.x.x or 11.6.x to 11.6.x.x, you must follow UEBA instructions in the Upgrade Guide for 11.5.x.x or 11.6.x.x, before you upgrade to 11.7.x.

1. (For Virtual Machines Only) Update the airflow parallelism on VM.
   If the UEBA system is running on VM, update the airflow parallelism to be 64 by running the following command as root from the UEBA host.

   sed -i "s|parallelism = 256|parallelism = 64|g" /var/netwitness/presidio/airflow/airflow.cfg

   Note: Copy this command in a single line.

2. Update the UEBA configuration using the following command from the UEBA machine.
   source /etc/sysconfig/airflow

   source $AIRFLOW_VENV/bin/activate

   OWB_ALLOW_NON_FIPS=on python /var/netwitness/presidio/airflow/venv/lib/python2.7/site-packages/presidio_workflows-1.0-py2.7.egg/presidio/resources/rerun_ueba_server_config.py

3. (Optional) Update the UEBA processing schema, if needed.

   NetWitness recommends that the UEBA start date is set to 28 days earlier than the current date. For UEBA systems that intend to process TLS data, you must make sure that the start date is set to no later than 14 days earlier than the current date.

   For more information, see the "reset-presidio script" section in the UEBA Configuration Guide.

4. Run the airflow upgrade DAG.

   • Go to Airflow main page https://<UEBA-host-name>/admin

   • Enter the admin username and password.

   • Click the Play in presidio_upgrade_dag_from_<previous_version> to_11.7.2.0.
     

     Note: A light green circle will appear next to the upgrade DAG row during the upgrade. If the upgrade process is completed successfully the light green circle changes to green. If the upgrade process fails, the light green circle changes to red.

5. Set the appropriate "Boot Jar Pools" slots:

   • Physical Appliance: Update the spring_boot_jar_pool slot value be 18.

   • Virtual Appliance: Update the spring_boot_jar_pool slot value to 22.
     To update the Spring Boot Jar Pools slots, go to the Airflow main page, tap the Admin tab at the top bar and tap Pools.
   1. To access the Airflow UI, go to https://<UEBA_host>/admin and enter the credentials.
      User: admin
      Password: The environment deploy admin password.
   2. Click on the pencil mark of the Pools to update the slot values.
      
6. Edit the spring_boot_jar_pool and update the slots amount to 22.
   

Post Upgrade Tasks for Customers Upgrading From 11.5.3.2 or 11.6.x.x

Complete the tasks that apply to the hosts in your environment.

• General
• Event Stream Analysis (ESA)
• Respond
• Reference Log Decoder
• Update UEBA Configurations

General

(Conditional) Configure NAT-Based IP Addresses

If you have a host, such as a VLC, that requires a NAT-based IP address in order to connect to the NW Server host, you must update the host configuration with the following steps.

1. Log in to the host that requires the use of NAT IP addresses, using the console or SSH.
2. Run the following command:
   nw-manage --enable-nat-usage
3. To set the NAT address for the NW Server:
   1. Log into the NW Server using the console or SSH.
   2. Run the following command:
      nw-manage -update-host --host-id <UUID of NW Server> --ipv4-public <NAT IP of NW Server>

   Note: You can find the UUID and view the current NAT IP address of the host by running nw-manage --list-hosts.

(Conditional - For Warm-Standby Hosts Only) Register the Secondary IP Address of Warm-Standby Hosts

The Warm-Standby server must be upgraded to 11.5 or later before completing the following steps.

1. Log in to the NW Server using the console or SSH.
2. Run the following command:
   nw-manage --add-nws-secondary-ip --ipv4 <ip address of Warm/Standby Server>

Note: If the Warm-Standby server requires a NAT-based IP address (IPv4-public) for any host to access it during failover, the NAT IP address must also be registered by running the following command: nw-manage --add-nws-secondary-ip --ipv4 <NAT-based IP address of Warm Standby Server>

3. Verify the correct Warm Standby host IP address value by running the following command:
   nw-manage --get-nws-secondary-ip

Review Contents of /etc/hosts.user for Obsolete Host Entries

After upgrading the NW Server host or a component host, review the contents of the /etc/hosts.user file for any obsolete host entries. The /etc/hosts.user file contains system and user-generated entries that are not managed by NetWitness Platform. However, entries from /etc/hosts.user are merged with NetWitness Platform-generated host mappings to create and update /etc/hosts. To avoid conflicts with NetWitness Platform-generated mappings, and to avoid generating connectivity errors resulting from an IP address change, NetWitness recommends that you remove any entries in /etc/hosts.user that include a non-loopback IP address of a NetWitness Platform host.

After updating /etc/hosts.user, you must refresh the system by running the following command:
nw-manage --refresh-host --host-key <ID, IP, hostname or display name of host>

Make Sure Services Have Restarted and Are Capturing and Aggregating Data

Make sure that services have restarted and are capturing data (this depends on whether or not you have auto-start enabled).

If required, restart data capture and aggregation for the following services:

• Decoder
• Log Decoder
• Broker
• Concentrator
• Archiver

To Start Network Capture:

1. In the NetWitness Platform menu, go to (Admin) > Services.
   The Services view is displayed.
2. Select each Decoder service.

3. Under (actions), select View > System.

4. In the toolbar, click 

To Start Log Capture:

• 1. In the NetWitness Platform menu, go to (Admin) > Services.
     The Services view is displayed.
  2. Select each Log Decoder service.
  3. Under (actions), select View > System.
  4. In the toolbar, click 

To Start Aggregation:

1. In the NetWitness Platform menu, go to (Admin) > Services.

   The Services view is displayed.

2. For each Concentrator, Broker, and Archiver service:

   1. Select the service.
   2. Under (actions), select View > Config.
   3. In the toolbar, click 

Event Stream Analysis (ESA)

Mixed mode is not supported for ESA hosts in NetWitness Platform version 11.5 and later. The NetWitness server, ESA primary host, and ESA secondary host must all be on the same NetWitness Platform version.

There are no post-upgrade tasks required for ESA. For ESA troubleshooting, see ESA Troubleshooting Information.

If you want to add support for Endpoint, UEBA, and Live content rules, you must update the multi-valued and single-valued parameter meta keys on the ESA Correlation service to include all the required meta keys. It is not necessary to make these adjustments during the upgrade; you can make the adjustments later at a convenient time. For detailed information and instructions, see "Update Your ESA Rules for the Required Multi-Value and Single-Value Meta Keys" in the ESA Configuration Guide

Show Updates to an ESA Rule Deployment

You can view changes to an ESA rule deployment, such as adding or removing rules. When there is a change to a deployment, the update icon () appears next to the name of the deployment in the Rules tab options panel.

To show updates to an ESA rule deployment:

1. Go to (Configure) > ESA Rules.The Rules tab is displayed.
2. In the options panel, under Deployments click Show Updates on the far right.
3. Click Deploy Now.

If you are unable to deploy the ESA rule, see Known Issues for the workaround.

Respond

The Primary ESA server must be upgraded to 11.7.2.0 before you can complete these tasks.

Note: After upgrading the primary NW Server (including the Respond Server service), the Respond Server service is not automatically re-enabled until after the Primary ESA host is also upgraded to 11.7.2.0. The Respond post-upgrade tasks only apply after the Respond Server service is upgraded and is in the enabled state.

(Conditional) Restore Any Respond Service Custom Keys in the Aggregation Rule Schema

Note: If you did not manually customize the incident aggregation rule schema, you can skip this task.

If you added custom keys in the var/lib/netwitness/respond-server/data/aggregation_ rule_schema.json file for use in the groupBy clause for 11.x, modify the /var/lib/netwitness/respond-server/data/aggregation_rule_schema.json file and add the custom keys from the automatic backup file.

The backup file is located in /var/lib/netwitness/respond-server/data and it is in the following format:
aggregation_rule_schema.json.bak-<time of the backup>

Reference Log Decoder

Make sure your reference Log Decoder is upgraded to 11.5 or later versions. If you never set up a reference Log Decoder, no action is required. For details, see the Log Parser Customization Guide.

Update UEBA Configurations

IMPORTANT: In order to complete an upgrade process, every UEBA upgrade needs to be followed by upgrade steps.
In case of gradual upgrade, follow the instruction of each upgrade’s guide before continuing to the next upgrade.

Note: The Modeled Behaviors functionality is added to UEBA in 11.5.2. For any reason if you need to disable this functionality for your organization, see "Enable or Disable the Modeled Behaviors for Users" topic in the UEBA Configuration Guide for NetWitness Platform XDR.

1. (For Virtual Machines Only) Update the airflow parallelism on VM.

   If the UEBA system is running on VM, update the airflow parallelism to be 64 by running the following command as root from the UEBA host.

   sed -i "s|parallelism = 256|parallelism = 64|g" /var/netwitness/presidio/airflow/airflow.cfg

2. Update the UEBA configuration using the following command as root from the UEBA machine.

   python /var/netwitness/presidio/airflow/venv/lib/python2.7/site-packages/presidio_workflows-1.0-py2.7.egg/presidio/resources/rerun_ueba_server_config.py

3. (Optional) Update the UEBA processing schema, using the following script:
   python /var/netwitness/presidio/airflow/venv/lib/python2.7/sitepackages/presidio_workflows-1.0-py2.7.egg/presidio/utils/airflow/reset_presidio.pys

   NetWitness recommends that the UEBA start date is set to 28 days earlier than the current date. For UEBA systems that intend to process TLS data, you must make sure that the start date is set to no later than 14 days earlier than the current date.

   For more information about the script, see "reset-presidio script" section in the UEBA Configuration Guide for NetWitness Platform XDR.

4. Run the airflow upgrade DAG.

   Note: An error message may appear at the top of the Airflow home page until the post upgrade process is complete.

   1. Go to Airflow main page https://<UEBA-host-name>/admin and enter the credentials.

      User: admin

      Password: The environment deploy admin password.

   2. Click the Play in presidio_upgrade_dag_from_<previous_version> to_11.7.2.0.

      Note: A light green circle will appear next to the upgrade DAG row during the upgrade. If the upgrade process is completed successfully the light green circle changes to green. If the upgrade process fails, the light green circle changes to red.

5. Set the appropriate Boot Jar Pools according to the setup.

   • Physical Appliance: Update the spring_boot_jar_pool slot value be 18.

   • Virtual Appliance: Update the spring_boot_jar_pool slot value to 22.

   To update the number of Spring Boot Jar Pools:

   1. Go to the Airflow main page https://<UEBA_host>/admin and enter the credentials.

      User: admin

      Password: The environment deploy admin password.

   2. Click the Admin > Pools.
   3. Edit the spring_boot_jar_pool and update the slots amount.