Post Upgrade Tasks

After you upgrade to 11.7, NetWitness has several new features in the user interface. Complete the tasks that apply to the hosts in your environment.

General

(Conditional) Configure NAT-Based IP Addresses

If you have a host, such as a VLC, that requires a NAT-based IP address in order to connect to the NW Server host, you must update the host configuration with the following steps.

  1. Log in to the host that requires the use of NAT IP addresses, using the console or SSH.
  2. Run the following command:
    nw-manage --enable-nat-usage
  3. To set the NAT address for the NW Server:
    1. Log into the NW Server using the console or SSH.
    2. Run the following command:
      nw-manage -update-host --host-id <UUID of NW Server> --ipv4-public <NAT IP of NW Server>

      3. Note: You can find the UUID and view the current NAT IP address of the host by running nw-manage --list-hosts.

(Conditional - For Warm-Standby Hosts Only) Register the Secondary IP Address of Warm-Standby Hosts

The Warm-Standby server must be upgraded to 11.7 before completing the following steps.

  1. Log in to the NW Server using the console or SSH.
  2. Run the following command:
    nw-manage --add-nws-secondary-ip --ipv4 <ip address of Warm/Standby Server>

    3. Note: If the Warm-Standby server requires a NAT-based IP address (IPv4-public) for any host to access it during failover, the NAT IP address must also be registered by running the following command: nw-manage --add-nws-secondary-ip --ipv4 <NAT-based IP address of Warm Standby Server>

  3. Verify the correct Warm Standby host IP address value by running the following command:
    nw-manage --get-nws-secondary-ip

Review Contents of /etc/hosts.user for Obsolete Host Entries

After upgrading the NW Server host or a component host, review the contents of the /etc/hosts.user file for any obsolete host entries. The /etc/hosts.user file contains system and user-generated entries that are not managed by NetWitness Platform. However, entries from /etc/hosts.user are merged with NetWitness Platform-generated host mappings to create and update /etc/hosts. To avoid conflicts with NetWitness Platform-generated mappings, and to avoid generating connectivity errors resulting from an IP address change, RSA recommends that you remove any entries in /etc/hosts.user that include a non-loopback IP address of a NetWitness Platform host.

After updating /etc/hosts.user, you must refresh the system by running the following command:
nw-manage --refresh-host --host-key <ID, IP, hostname or display name of host>

Jetty Configuration

For Jetty Configuration and related information, see Manage Custom Host Entries topic in the System Maintenance Guide.

Reconfigure DNS Servers

By default, a component host upgraded from 11.4 or earlier is configured with the same system DNS server as the NW Server. If this component host requires a different system DNS address, see "Change Host Network Configuration" in the System Maintenance Guide for instructions.

Make Sure Services Have Restarted and Are Capturing and Aggregating Data

Make sure that services have restarted and are capturing data (this depends on whether or not you have auto-start enabled).

If required, restart data capture and aggregation for the following services:

  • Decoder
  • Log Decoder
  • Broker
  • Concentrator
  • Archiver

Start Network Capture

  1. In the NetWitness Platform menu, go to netwitness_adminicon_25x22.png (Admin) > Services.
    The Services view is displayed.
  2. Select each Decoder service.

  3. Under netwitness_actions_button.png (actions), select View > System.

  4. In the toolbar, click netwitness_startcapturedr_140x33.png

Start Log Capture

    1. In the NetWitness Platform menu, go to netwitness_adminicon_25x22.png (Admin) > Services.
      The Services view is displayed.
    2. Select each Log Decoder service.
    3. Under netwitness_actions_button.png (actions), select View > System.
    4. In the toolbar, click netwitness_startcapturedr_140x33.png

    • Start Aggregation

    1. In the NetWitness Platform menu, go to netwitness_adminicon_25x22.png (Admin) > Services.

      The Services view is displayed.

    2. For each Concentrator, Broker, and Archiver service:

      1. Select the service.
      2. Under netwitness_actions_button.png (actions), select View > Config.
      3. In the toolbar, click netwitness_startaggr.png
    3. Event Stream Analysis (ESA)

    Note: Mixed mode is not supported for ESA hosts in NetWitness Platform version 11.6 and later. The NetWitness server, ESA primary host, and ESA secondary host must all be on the same NetWitness Platform version.

    There are no required post-upgrade tasks for ESA. For ESA troubleshooting, see ESA Troubleshooting Information.

    If you want to add support for Endpoint, UEBA, and Live content rules, you must update the multi-valued and single-valued parameter meta keys on the ESA Correlation service to include all the required meta keys. It is not necessary to make these adjustments during the upgrade; you can make the adjustments later at a convenient time. For detailed information and instructions, see "Update Your ESA Rules for the Required Multi-Value and Single-Value Meta Keys" in the ESA Configuration Guide

    Upgrade Considerations for ESA Analytics

    The Event Stream Analytics Server (ESA Analytics) service is not supported or available in NetWitness version 11.6 and later. The Whois Lookup Configuration and ESA Analytics Mapping panels are no longer in the user interface (Admin > System).

    Note: Event Stream Analysis (ESA) is not end of life. ESA Correlation rules and the ESA Correlation service are supported. ESA Analytics, which is used for Automated Threat Detection, is different from ESA Correlation Rules and is EOL. In its place, you can use ESA Correlation as it offers more functional capabilities and better performance.

    New Health and Wellness

    Note: New Health and Wellness in 11.5 replaces Next GEN Health and Wellness (BETA) in 11.4.x.x.

    Deploy the New Health and Wellness Content from Live

    After you upgrade from version 11.4.x.x to 11.7, New Health and Wellness content is not updated. To use the latest (default) content, you must deploy the content through NetWitness Live Services.

    Note: RSA recommends you to take a copy of 11.4.x.x Health and Wellness content before you deploy the content from NetWitness Live Services, as it overwrites the existing content.

    1. Log in to NetWitness Platform UI.

    2. Click netwitness_configureicon_27x24.png (CONFIGURE) > LIVE CONTENT.

    3. In the Search Criteria panel, select the Resource Types as:

      • Health and Wellness Dashboards
      • Health and Wellness Monitors

    4. Click Search.

    5. In the Matching Resources view, select the checkbox to the left of the resources that you want to deploy.

    6. In the Matching Resources toolbar, click netwitness_deploybtn.png .

    7. In the Deployment Wizard > Resources tab, click Next.

    8. In the Services tab, select the Metrics Server service.

    9. Click Next.

    10. Click Deploy.
      The Deploy page is displayed. The Progress bar turns green when you have successfully deployed the resources to the selected services.

    11. Click Close.

    (Optional) Update UUID of New Health and Wellness Host to Update Service Configuration Documents

    If you have configured services for New Health and Wellness from nw-shell using set-config API and upgrade NetWitness Platform version from 11.4.x.x to 11.7, you must update IP with UUID for a host on which New Health and Wellness is installed.

    1. SSH to Admin Server.

    2. Check the UUID of a host on which New Health and Wellness is installed using the command:
      orchestration-cli-client --list-hosts

      This lists NetWitness Platform hosts along with the respective UUIDs. Make a note of the UUID of host on which New Health and Wellness is installed.

    3. Identify the services on which set-config is invoked using the command:
      mongo localhost/metrics-server -u deploy_admin -p <deployment_password> --authenticationDatabase admin --eval 'db.metric_config.find({ "createdBy": { $ne: "system" }})'
      This will list the configuration documents of the services on which set-config is invoked.

    Note: If no service documents are listed which means no services are configured before the upgrade, so you can ignore the remaining steps.

    1. In the configuration file, update the service document “host” field by replacing IP with the UUID of the host on which New Health and Wellness is installed.
      For example, update the host IP with UUID as below:

    "host" : “e086511c-121c-4e66-95a3-e87e27b12acb”

    1. Log in to nw-shell using the command:
      nw-shell
    2. Connect to metrics-server service using the command:
      connect --service metrics-server

    3. Enter the log in command:
      login

    4. Enter the admin username and password.
    5. Go to /rsa/metrics/elastic/set-config and invoke configuration files using the command:
      invoke --file /<absolute_path_of_service_config_file>

    Investigate

    (Conditional - For Custom Roles Only) Adjust investigate-server Permissions for Custom User Roles

    After upgrading to Version 11.7, the built-in user roles for analysts (and others) using Investigate have the investigate-server.event.filter permission enabled, but the upgrade process does not enable the permission for custom user roles. Users who are assigned a custom user role that does not have this permission enabled cannot see the Filter Events panel, a new panel in 11.7 where they can drill into metadata.

    Note: The built-in user roles for analysts using Investigate have three additional permissions added in Version 11.4 enabled, but the upgrade process does not enable the permissions for custom user roles. Users who are assigned a custom user role that does not have these permissions cannot see the Navigate view and Legacy Events view in the Investigate menu. The three permissions that need to be enabled for custom user roles are:
    investigate-server.columngroup.read, investigate-server.metagroup.read, and investigate-server.profile.read

    To enable the permissions for a user role:

    1. Go to netwitness_adminicon_25x22.png (Admin) > Security and click the Roles tab.
    2. Select the custom user role that needs to be edited and click netwitness_edit.png (edit icon).
    3. In the Edit Role dialog, ensure that these four permissions are enabled:
      investigate-server.event.filter
      investigate-server.columngroup.read
      investigate-server.metagroup.read
      investigate-server.profile.read
    4. Click Save to save your changes. When analysts with the custom user role log in to the NetWitness Platform, the changes are in effect.

    Respond

    The Primary ESA server must be upgraded to 11.7 before you can complete these tasks.

    Note: After upgrading the primary NW Server (including the Respond Server service), the Respond Server service is not automatically re-enabled until after the Primary ESA host is also upgraded to 11.7. The Respond post-upgrade tasks only apply after the Respond Server service is upgraded and is in the enabled state.

    (Conditional) Restore Any Respond Service Custom Keys in the Aggregation Rule Schema

    Note: If you did not manually customize the incident aggregation rule schema, you can skip this task.

    If you added custom keys in the var/lib/netwitness/respond-server/data/aggregation_ rule_schema.json file for use in the groupBy clause for 11.x, modify the /var/lib/netwitness/respond-server/data/aggregation_rule_schema.json file and add the custom keys from the automatic backup file.

    The backup file is located in /var/lib/netwitness/respond-server/data and it is in the following format:
    aggregation_rule_schema.json.bak-<time of the backup>

    (Conditional) Restore any Customized Respond Service Normalization Scripts

    Note: If you did not manually customize any alert normalization scripts, you can skip this task.
    This procedure applies to upgrades from 11.3.x to 11.7. (For upgrades from 11.4.x to 11.7, there are no backups of the normalization script files since customizations are in separate custom_normalize script files, which are not overwritten during the upgrade.

    To prevent overwriting future customizations, custom normalization script files are available in NetWitness Platform 11.4 and later. Add any custom logic to the custom_normalize_<alert type>.js files.

    1. Locate any custom logic from the backup Respond normalization scripts located in the /var/lib/netwitness/respond-server/scripts.bak-<timestamp> directory, where <timestamp> is the time that the backup completed:
      data_privacy_map.js
      normalize_alerts.js
      normalize_core_alerts.js
      normalize_ecat_alerts.js
      normalize_ma_alerts.js
      normalize_ueba_alerts.js
      normalize_wtd_alerts.js
      utils.js

    2. Edit the new 11.4 or later script files in the /var/lib/netwitness/respond-server/scripts directory to include any logic from the back up files. If you have any customizations in the normalization files, add them to the normalization files with the custom prefix.
      data_privacy_map.js
      custom_normalize_alerts.js
      custom_normalize_core_alerts.js
      custom_normalize_ecat_alerts.js
      custom_normalize_ma_alerts.js
      custom_normalize_ueba_alerts.js
      custom_normalize_wtd_alerts.js
      utils.js

      For Example, the custom_normalize_core_alerts.js is the normalization script for ESA to add any custom logic. This java script file has a function ‘normalizeAlert’ with the parameters headers, rawAlert, and normalizedAlert. The variable ‘normalized’ is an immutable copy object, which has an embedded object of a list of normalized events. So if you have any custom meta keys configured for the events then you have to iterate through the ‘normalized.events’ to populate the appropriate meta keys with values from the ‘rawAlert.events’ object. Below is the sample code.

      normalizeAlert = function (headers, rawAlert, normalizedAlert) {

      // normalizedAlert is the immutable copy of ootb normalizer alert, make sure you use

      // normalized object to update/set the values in your scripts

      var normalized = Object.assign(normalizedAlert);

      var custom_events;

      if(normalized.events !== undefined) {

      custom_events = normalized.events;

      } else {

      custom_events = new Array([]);

      }

      for (var i = 0; i < rawAlert.events.length; i++) {

      custom_events[i].legalentity=Utils.stringValue(rawAlert.events[i].isgs_legalentity);

      custom_events[i].companycode=Utils.stringValue(rawAlert.events[i].isgs_companycode);

      }

      if(normalized.events === undefined){

      normalized.events = custom_events;

      }

      return normalized;

      };

      Update the custom_normalize_alerts.js script to normalize Detect AI events

      To prevent the custom_normalize_alerts.js script being overwritten during the upgrade process, edit and update the custom_normalize_alerts.js script to include the function required for normalizing Detect AI events.

      1. Locate the file vi /var/netwitness/respond-server/scripts/custom_normalize_alerts.js.

      2. Add the following code in line number 12:
      "custom_normalize_detectai_alerts"

      3. Add the following code in line number 41:
      else if(headers.deviceProduct == "Detect AI")

      { transformer = transformers["custom_normalize_detectai_alerts"]; }


      You can also look at the built-in Respond normalization script files for reference, such as normalize_alerts.js. For more information, see "Configure Custom Respond Server Alert Normalization" in the NetWitness Respond Configuration Guide.

    Reference Log Decoder

    For full functionality, make sure your reference Log Decoder is at 11.6 or later. If you never set up a reference Log Decoder, there is no need to take action. For details, see the Log Parser Customization Guide.

    Windows Log Collector

    Update the Windows Log Collector UUID

    After upgrading to 11.6, for each Windows Log Collector configured in your environment, run the following command on the NW Server:

    wlc-cli-client --update-to-uuid --host <WLC host address>

    User Entity Behavior Analytics

    IMPORTANT: Before the upgrade, if you encountered and resolved the task failure issues, then after the upgrade, you must replace the authentication.json file before you run the post-upgrade tasks. The task failure issues in Airflow and their solutions are described in the 'Troubleshooting' topic of the UEBA Configuration Guide.

    IMPORTANT: Every UEBA deployment when upgraded requires additional steps to complete the upgrade process. When you upgrade from 11.5.0.0 or 11.5.1.0 to 11.7.0.0, you must follow UEBA instructions in the Upgrade Guide for 11.5.1.0, before you upgrade to 11.7.0.0.

    Note: When you upgrade to 11.7.0.0 from 11.4.x.x, you don't need to rerun the UEBA system for the last 28 days, if you don't update the current processing schemas. When you upgrade to 11.7.0.0 from a version prior to 11.4.x, the UEBA system runs a rerun automatically.

    1. (For Virtual Machines Only) Update the airflow parallelism on VM.
      If the UEBA system is running on VM, update the airflow parallelism to be 64 by running the following command as root from the UEBA host.

      sed -i "s|parallelism = 256|parallelism = 64|g" /var/netwitness/presidio/airflow/airflow.cfg

    2. Update the UEBA configuration using the following command from the UEBA machine.
      python /var/netwitness/presidio/airflow/venv/lib/python2.7/site-packages/presidio_workflows-1.0-py2.7.egg/presidio/resources/rerun_ueba_server_config.py

    1. (Optional) Update the UEBA processing schema, using the following script:

      python /var/netwitness/presidio/airflow/venv/lib/python2.7/site-packages/presidio_workflows-1.0-py2.7.egg/presidio/utils/airflow/reset_presidio.py script.

      RSA recommends that the UEBA start date is set to 28 days earlier than the current date. For UEBA systems that intend to process TLS data, you must make sure that the start date is set to no later than 14 days earlier than the current date.

      For more information, see the "reset-presidio script" section in the UEBA Configuration Guide.

    2. Run the airflow upgrade DAG.

      • Go to Airflow main page https://<UEBA-host-name>/admin

      • Enter the admin username and password.

      • Click the Play in presidio_upgarde_dag_from_<previous_version> to_11.7.0.0.
        netwitness_airflow1_1193x652.png

        Note: A light green circle will appear next to the upgrade DAG row during the upgrade. If the upgrade process is completed successfully the light green circle changes to green. If the upgrade process fails, the light green circle changes to red.

    3. Set the appropriate "Boot Jar Pools" slots:

      • Physical Appliance: Update the spring_boot_jar_pool slot value be 18.

      • Virtual Appliance: Update the spring_boot_jar_pool and the retention_spring_boot_jar_pool slot values to 22.
        To update the “Spring Boot Jar Pools” slots, Go to the Airflow main page, tap the “Admin” tab at the top bar and tap “Pools”.
      1. To access the Airflow UI, go to https://<UEBA_host>/admin and enter the credentials.
        User: admin
        Password: The environment deploy admin password.
      1. Click on the pencil mark of the Pools to update the slot values.
        netwitness_airflowtbs.png
    4. Edit the spring_boot_jar_pool and update the slots amount to 5.
      netwitness_airflowslt_1116x193.png