Perform Post Upgrade Tasks

This topic lists the tasks you must perform after upgrading NetWitness Platform. Complete the tasks that apply to the hosts in your environment.

General

You must configure Jetty, restore the core services contents, and also start Network capture, Log capture, and aggregation after upgrading NetWitness Platform.

Configure Jetty

For Jetty Configuration and related information, see Manage Custom Host Entries topic in the System Maintenance Guide.

Make Sure Services Have Restarted and Are Capturing and Aggregating Data

Make sure that services have restarted and are capturing data (this depends on whether or not you have auto-start enabled).

If required, restart data capture and aggregation for the following services:

  • Decoder
  • Log Decoder
  • Broker
  • Concentrator
  • Archiver

To Start Network Capture:

  1. In the NetWitness Platform menu, go to AdminIcon_25x22.png(Admin) > Services.
    The Services view is displayed.
  2. Select each Decoder service.

  3. Under actions_button.png(actions), select View > System.

  4. In the toolbar, click StartCapturedr_140x33.png

To Start Log Capture:

    1. In the NetWitness Platform menu, go to AdminIcon_25x22.png(Admin) > Services.
      The Services view is displayed.
    2. Select each Log Decoder service.
    3. Under actions_button.png(actions), select View > System.
    4. In the toolbar, click StartCapturedr_140x33.png

To Start Aggregation:

  1. In the NetWitness Platform menu, go to AdminIcon_25x22.png(Admin) > Services.

    The Services view is displayed.

  2. For each Concentrator, Broker, and Archiver service:

    1. Select the service.
    2. Under actions_button.png(actions), select View > Config.

    3. In the toolbar, click StartAggr.png

  3. For Event Stream Analysis (ESA):

    Note: Mixed mode is not supported for ESA hosts in NetWitness Platform version 11.6 and later. The NetWitness server, ESA primary host, and ESA secondary host must all be on the same NetWitness Platform version.

    There are no required post-upgrade tasks for ESA. For ESA troubleshooting, see ESA Troubleshooting Information.

    If you want to add support for Endpoint, UEBA, and Live content rules, you must update the multi-valued and single-valued parameter meta keys on the ESA Correlation service to include all the required meta keys. It is not necessary to make these adjustments during the upgrade; you can make the adjustments later at a convenient time. For detailed information and instructions, see Update Your ESA Rules for the Required Multi-Value and Single-Value Meta Keys in the ESA Configuration Guide.

    Restore the Core Services Contents

    Once you upgrade to 12.3.1.0, the Core services Contents such as Configuration files (.cfg), Feeds, Parsers, and Log Devices are copied to the .tar location of the respective components such as Decoder, Log Hybrid, Network Hybrid, and Log Decoder.

    The following table lists the Core Services Contents paths and the .tar location of the respective components where the Core Services Contents are copied.

    Core Services Contents Paths Components .tar location of the Components
    /etc/netwitness/ng/feeds (Feeds) Decoder /var/netwitness/decoder/decoder_backupcontent_ccm.tar
    /etc/netwitness/ng/parsers (Parsers) Log Hybrid /var/netwitness/logdecoder/logdecoder_backupcontent_ccm.tar
    /etc/netwitness/ng/envision/etc/devices (Log Devices) Network Hybrid /var/netwitness/decoder/decoder_backupcontent_ccm.tar
    /etc/netwitness/ng/NwDecoder.cfg (Configuration files (.cfg)) Log Decoder /var/netwitness/logdecoder/logdecoder_backupcontent_ccm.tar

    By default, CCM option is disabled. After upgrading to 12.3.1.0, if you enable CCM and encounter the loss of the Core Services Contents, you can use the backup tar files to recover the lost data. For more information, see https://community.netwitness.com/t5/netwitness-knowledge-base/automatic-backup-of-core-service-content-after-upgrading-core/ta-p/694527.

Event Stream Analysis (ESA)

After upgrading to the 12.3.1.0 version, all the ESA deployments will be migrated to ConfigureIcon_14x12.png(CONFIGURE) > Policies page. Each deployment will be converted into a policy and group and will be available to manage only after the upgrade of the Correlation servers to the 12.3.1.x version. Make sure that you plan the upgrade process so that Correlation servers are upgraded immediately after the Admin Server is done. The deployments will not be accessible until the corresponding Correlation servers are upgraded. However, the correlation servers will still continue to process the Alerts and Events. Verify if all the ESA deployments are in a healthy state. For more information, see View a Deployment topic in the Live Services Management Guide.

Note: Analysts must have appropriate permissions to view the ESA rules under ConfigureIcon_13x11.png(CONFIGURE) > ESA Rules and ConfigureIcon_12x10.png(CONFIGURE) > Policies pages. For more information, see the Source-server section in the Role Permissions topic in the System Security and User Management Guide.

The pre-upgrade and post-upgrade states of deployments are represented in the following table.

 SlNo Pre-upgrade Deployment State Post-upgrade Deployment State
Creates Policy Creates Group The policy will be Published
1 Healthy deployment

Yes

Yes

Yes
2 Deployment with errors Yes Yes Yes
3 Deployment with only rules

Yes

No

No
4 Deployment with no rules No No No

(Optional) Using the Merge Policy button, you can merge a policy having ESA content with a policy with no ESA content. For more information, see Merge Policy with ESA Content topic in the Live Services Management Guide.

Manage ESA Deployments and Data Sources

In 12.1 and later versions, you can only manage the ESA deployments and Data Sources through Centralized Content Management. Go to ConfigureIcon_12x10.png(CONFIGURE) > Policies > Content > Event Stream Analysis page to manage the ESA deployments and Data Sources. You can only manage the ESA Rules in the ESA Rules page. Refer the following figures.

ESA_DEP_UpG_12.3.png

esa_rules_tab_configure_1929x910.png

You must upgrade the ESA hosts immediately after upgrading the Admin Server.

For more information on Centralized Content Management and managing the deployments, see Centralized Content Management Guide for NetWitness.

Respond

The Primary ESA server must be upgraded to 12.3.1.0 before you can complete the following task.

Note: After upgrading the primary NW Server (including the Respond Server service), the Respond Server service is not automatically re-enabled until after the Primary ESA host is also upgraded to 12.3.1.0. The Respond post-upgrade tasks only apply after the Respond Server service is upgraded and is in the enabled state.

(Conditional) Restore Any Respond Service Custom Keys in the custom_normalize_alerts.js and support new datasource

Note: If you did not manually customize the custom_normalize_alerts.js, you can skip this task. We attempt to automatically migrate the custom keys. However in case of failures, use this step to verify the custom data's integrity.

If you added custom keys in the /var/netwitness/respond-server/scripts/custom_normalize_alerts.js file for use in custom normailization, modify the /var/netwitness/respond-server/scripts/custom_normalize_alerts.js file and add the custom normalized keys from the automatic backup file. The backup file is located in /var/netwitness/respond-server/scripts and it is in the following format:

custom_normalize_alerts.js.bak-<time of the backup>

In case of automatic update of the script fails, add support for Netwitness Core and NetWitness Insight by updating the custom_normalize_alerts.js file manually to support these new sources in respond.

User Entity Behavior Analytics

Complete the following tasks after upgrading UEBA to 12.3.1.0.

IMPORTANT: After you upgrade the UEBA host to version 12.3.1.0, you must first install the hotfix of 12.3.1.0 and then proceed with the post-upgrade tasks. For more information on installing the hotfix related to the UEBA host in 12.3.1.0, please contact the NetWitness Customer Support team..

IMPORTANT: Every UEBA deployment when upgraded requires additional steps to complete the upgrade process. When you upgrade from 11.6.x to 11.6.x.x, you must follow UEBA instructions in the Upgrade Guide for 11.6.x.x, before you upgrade to 11.7.x.

  1. Update the UEBA configuration using the following command from the UEBA machine.
    source /etc/sysconfig/airflow

    source $AIRFLOW_VENV/bin/activate

    OWB_ALLOW_NON_FIPS=on python /var/netwitness/presidio/airflow/venv/lib/python2.7/site-packages/presidio_workflows-1.0-py2.7.egg/presidio/resources/rerun_ueba_server_config.py

  1. (Optional) Update the UEBA processing schema, if needed.

    NetWitness recommends that the UEBA start date is set to 28 days earlier than the current date. For UEBA systems that intend to process TLS data, you must make sure that the start date is set to no later than 14 days earlier than the current date.

    For more information, see the "reset-presidio script" section in the UEBA Configuration Guide.

  2. Run the airflow upgrade DAG.

    • Go to Airflow main page https://<UEBA-host-name>/admin

    • Enter the admin username and password.

    • Click the Play in presidio_upgrade_dag_from_<previous_version> to_12.3.1.0.
      Airflow1_1193x652.png

      Note: A light green circle will appear next to the upgrade DAG row during the upgrade. If the upgrade process is completed successfully the light green circle changes to green. If the upgrade process fails, the light green circle changes to red.

  3. Set the appropriate "Boot Jar Pools" slots:

    • Physical Appliance: Update the spring_boot_jar_pool slot value be 18.

    • Virtual Appliance: Update the spring_boot_jar_pool slot value to 22.
      To update the Spring Boot Jar Pools slots, go to the Airflow main page, tap the Admin tab at the top bar and tap Pools.
    1. To access the Airflow UI, go to https://<UEBA_host>/admin and enter the credentials.
      User: admin
      Password: The environment deploy admin password.
    1. Click on the pencil mark of the Pools to update the slot values.
      AirflowTbs.png

  4. Edit the spring_boot_jar_pool and update the slots amount to 22.
    AirflowSlt_1116x193.png

  5. Import the Elasticsearch presidio data after upgrading the UEBA host from 12.0.0.0 or older versions to 12.3.1.0. Make sure the following prerequisites are met before you import the Elasticsearch presidio data:

    • Elasticsearch version must be upgraded to 7.15.2 from 5.5.0.

    • UEBA host must be upgraded to 12.3.1.0

    • UEBA rpm version must be 12.3.1.0.

    • Elasticsearch data in 12.0.0.0 or older versions must be exported and stored in the Elasticsearch data backup folder located in the /root/ directory.

      To import the Elasticsearch data:

      1. Go to cd ueba_es_migration_tool. Run the following command.

        sh elk-migration-script.sh

        The Elasticsearch migration tool guide is displayed.

        import_docs_to_elasticsearch_7.15.2.png

      2. Select Import documents to elasticsearch 7.15.2 from backup.

      3. In the next step, select Fresh Import to import the backup data.

        fresh_import_2109x1070.png

      4. Restart the Presidio UI service once the Import operation is completed. Run the following command.

        systemctl restart presidio-ui

      5. Go to the NetWitness Platform Users tab and verify if all the Elasticsearch data is imported.

        Note:
        - Go to <backup_directory_path>/log/log/es-migration-import.log if you want to view the log for any exceptions.
        - If the Import operation fails due to some technical issue, select Resume Import once the issue is resolved, to resume the Import operation.

Legacy Windows Log Collector

Update the Legacy Windows Log Collector UUID

After upgrading to 12.3.1.0, for each Legacy Windows Log Collector configured in your environment, run the following command on the NW Server:

wlc-cli-client --update-to-uuid --host <WLC host address>

Refresh Legacy Windows Log Collector Certificates with Updated SA Certificates

Post Upgrade Steps:

  1. Execute the following command in SA:

    1. wlc-cli-client --host-display-name hostDisplayName --service-display-name serviceDisplayName --host WLChostIPAddress --port 50101 --use-ssl false

      Enter following information:

      1. Legacy Windows Log Collector REST Username and Legacy Windows Log Collector REST Password: Enter the admin credentials for the Legacy Windows Log Collector.

      2. Security Server Username and Security Server Password: Enter admin credentials for NetWitness.

  2. Restart the system.

Perform Sanity Checks After Upgrade

You must perform the following sanity checks after upgrading to NetWitness 12.3.1.0.

  1. Go to AdminIcon_25x22.png(Admin) > Services view to verify that all the services are active (appearing in green) after upgrade.

  2. Verify that the services are upgraded to match the host version. The service version in AdminIcon_25x22.png(Admin) > Services view must match the host version in AdminIcon_25x22.png(Admin) > Hosts view after upgrade.

  3. In the AdminIcon_25x22.png(Admin) > Services view, do the following.

    • Select a Log Collector service and go to actions_button_30x19.png(actions) > View > System view to verify if the required logs collection is started. You should click the collection_log_collector.PNGdrop-down option and go to the right collection protocol to check if the logs collection is started. If the required collection is not started, select start_right_collection_86x21.pngnext to the required collection protocol from the list to start the collection.

    • Select a Log Decoder service and go to actions_button_30x19.png(actions) > View > System view to verify if the Log Decoder is capturing the logs properly.

    • Select a Packet Decoder service and go to actions_button_30x19.png(actions) > View > Config view to check if the capture interface is configured under Decoder Configuration section. If the capture interface is not configured, you must select the required capture interface from the drop-down list to configure it. If the capture interface is already configured, go to the actions_button_30x19.png(actions) > View > System view of the Packet Decoder and check if the capture is started. If the capture is not started, click StartCapturedr_109x25.pngto start the packet capture.

  4. Go to AdminIcon_25x22.png(Admin) > Services > Select a Log Decoder or Packet Decoder service > actions_button_30x19.png(actions) > View > Stats > General view to analyze the current capture rate.

  5. Verify that the Concentrators, Archivers, and Brokers are aggregating the data. Make sure that you can investigate from each Concentrator, Archiver, and Broker to validate that it is operational.

  6. Go to Respond > Alerts view to verify if the alerts are triggering from different sources.

  7. Go to AdminIcon_25x22.png(Admin) > Health & Wellness > Alarms view and verify if the SMS server is up and running.

  8. Go to AdminIcon_25x22.png(Admin) > Event Sources > Monitoring Policies view and verify if the policies configured before upgrade are appearing.

  9. Go to AdminIcon_25x22.png(Admin) > Health & Wellness > New Health & Wellness > Pivot to Dashboard > Elastic > Dashboard view and ensure the following.

    • The visualizations you created before upgrade still exist.

    • The metric server is up and running.

    • Alerts are generated properly for the monitors you have configured before upgrade.