Post Upgrade Tasks for 12.3

[#1]

After you upgrade to 12.3.0.0, NetWitness has several new features in the user interface. Complete the tasks that apply to the hosts in your environment.

• General
• Event Stream Analysis (ESA)
• Respond
• Legacy Windows Log Collector
• User Entity Behavior Analytics

General

Jetty Configuration

For Jetty Configuration and related information, see Manage Custom Host Entries topic in the System Maintenance Guide.

Make Sure Services Have Restarted and Are Capturing and Aggregating Data

Make sure that services have restarted and are capturing data (this depends on whether or not you have auto-start enabled).

If required, restart data capture and aggregation for the following services:

• Decoder
• Log Decoder
• Broker
• Concentrator
• Archiver

Start Network Capture

1. In the NetWitness Platform menu, go to  (Admin) > Services.
   The Services view is displayed.
2. Select each Decoder service.

3. Under (actions), select View > System.

4. In the toolbar, click 

Start Log Capture

1. In the NetWitness Platform menu, go to (Admin) > Services.
   The Services view is displayed.
2. Select each Log Decoder service.
3. Under  (actions), select View > System.
4. In the toolbar, click  

•  

Start Aggregation

1. In the NetWitness Platform menu, go to (Admin) > Services.

   The Services view is displayed.

2. For each Concentrator, Broker, and Archiver service:

   1. Select the service.
   2. Under  (actions), select View > Config.
   3. In the toolbar, click 
3. Event Stream Analysis (ESA)

Note: Mixed mode is not supported for ESA hosts in NetWitness Platform version 11.6 and later. The NetWitness server, ESA primary host, and ESA secondary host must all be on the same NetWitness Platform version.

There are no required post-upgrade tasks for ESA. For ESA troubleshooting, see ESA Troubleshooting Information.

If you want to add support for Endpoint, UEBA, and Live content rules, you must update the multi-valued and single-valued parameter meta keys on the ESA Correlation service to include all the required meta keys. It is not necessary to make these adjustments during the upgrade; you can make the adjustments later at a convenient time. For detailed information and instructions, see "Update Your ESA Rules for the Required Multi-Value and Single-Value Meta Keys" in the ESA Configuration Guide

Restore the Core Services Contents

Once you upgrade to 12.3, the Core services Contents such as Configuration files (.cfg), Feeds, Parsers, and Log Devices are copied to the .tar location of the respective components such as Decoder, Log Hybrid, Network Hybrid, and Log Decoder.

The following table lists the Core Services Contents paths and the .tar location of the respective components where the Core Services Contents are copied.

Core Services Contents Paths	Components	.tar location of the Components
/etc/netwitness/ng/feeds (Feeds)	Decoder	/var/netwitness/decoder/decoder_backupcontent_ccm.tar
/etc/netwitness/ng/parsers (Parsers)	Log Hybrid	/var/netwitness/logdecoder/logdecoder_backupcontent_ccm.tar
/etc/netwitness/ng/envision/etc/devices (Log Devices)	Network Hybrid	/var/netwitness/decoder/decoder_backupcontent_ccm.tar
/etc/netwitness/ng/NwDecoder.cfg (Configuration files (.cfg))	Log Decoder	/var/netwitness/logdecoder/logdecoder_backupcontent_ccm.tar

By default, CCM option is disabled. After upgrading to 12.3, if you enable CCM and encounter the loss of the Core Services Contents, you can use the backup tar files to recover the lost data. For more information, see https://community.netwitness.com/t5/netwitness-knowledge-base/automatic-backup-of-core-service-content-after-upgrading-core/ta-p/694527.

Event Stream Analysis (ESA)

After upgrading to the 12.3 version, all the ESA deployments will be migrated to (CONFIGURE) > Policies page. Each deployment will be converted into a policy and group and will be available to manage only after the upgrade of the Correlation servers to the 12.3 version. Make sure that you plan the upgrade process so that Correlation servers are upgraded immediately after the Admin Server is done. The deployments will not be accessible until the corresponding Correlation servers are upgraded. However, the correlation servers will still continue to process the Alerts and Events. Verify if all the ESA deployments are in a healthy state. For more information, see "View a Deployment" topic in the Live Services Management Guide.

Note: Analysts must have appropriate permissions to view the ESA rules under (CONFIGURE) > ESA Rules and (CONFIGURE) > Policies pages. For more information, see the Source-server section in the "Role Permissions" topic in the System Security and User Management Guide.

The pre-upgrade and post-upgrade states of deployments are represented in the following table.

SlNo	Pre-upgrade Deployment State	Post-upgrade Deployment State
		Creates Policy	Creates Group	The policy will be Published
1	Healthy deployment	Yes	Yes	Yes
2	Deployment with errors	Yes	Yes	Yes
3	Deployment with only rules	Yes	No	No
4	Deployment with no rules	No	No	No

(Optional) Using the Merge Policy button, you can merge a policy having ESA content with a policy with no ESA content. For more information, see "Merge Policy with ESA Content" topic in the Live Services Management Guide.

Manage ESA Deployments and Data Sources

In 12.1 and later versions, you can only manage the ESA deployments and Data Sources through Centralized Content Management. Go to (CONFIGURE) > Policies > Content > Event Stream Analysis page to manage the ESA deployments and Data Sources. You can only manage the ESA Rules in the ESA Rules page. Refer the following screenshots.

You must upgrade the ESA hosts immediately after upgrading the Admin Server.

For more information on Centralized Content Management and managing the deployments, see https://community.netwitness.com/t5/netwitness-platform-online/centralized-content-management-guide-for-12-1-1/ta-p/694426.

Respond

The Primary ESA server must be upgraded to 12.3.0.0 before you can complete these tasks.

Note: After upgrading the primary NW Server (including the Respond Server service), the Respond Server service is not automatically re-enabled until after the Primary ESA host is also upgraded to 12.3.0.0 The Respond post-upgrade tasks only apply after the Respond Server service is upgraded and is in the enabled state.

(Conditional) Restore Any Respond Service Custom Keys in the Aggregation Rule Schema

Note: If you did not manually customize the incident aggregation rule schema, you can skip this task.

If you added custom keys in the var/lib/netwitness/respond-server/data/aggregation_rule_schema.json file for use in the groupBy clause for 12.1.0.0, modify the /var/lib/netwitness/respond-server/data/aggregation_rule_schema.json file and add the custom keys from the automatic backup file. The backup file is located in /var/lib/netwitness/respond-server/data and it is in the following format:

aggregation_rule_schema.json.bak-<time of the backup>

(Conditional) Restore Any Respond Service Custom Keys in the custom_normalize_alerts.js and support new datasource

Note: If you did not manually customize the custom_normalize_alerts.js, you can skip this task. We attempt to automatically migrate the custom keys. However in case of failures, use this step to verify the custom data's integrity.

If you added custom keys in the /var/netwitness/respond-server/scripts/custom_normalize_alerts.js file for use in custom normailization, modify the /var/netwitness/respond-server/scripts/custom_normalize_alerts.js file and add the custom normalized keys from the automatic backup file. The backup file is located in /var/netwitness/respond-server/scripts and it is in the following format:

custom_normalize_alerts.js.bak-<time of the backup>

In case of automatic update of the script fails, add support for Netwitness Core and NetWitness Insight by updating the custom_normalize_alerts.js file manually to support these new sources in respond.

Legacy Windows Log Collector

Perform the following tasks after upgrading the Legacy Windows Log Collector to 12.3.

Update the Legacy Windows Log Collector UUID

After upgrading to 12.3.0.0, for each Legacy Windows Log Collector configured in your environment, run the following command on the NW Server:

wlc-cli-client --update-to-uuid --host <WLC host address>

Refresh Legacy Windows Log Collector Certificates with Updated SA Certificates

Post Upgrade Steps:

1. Execute the following command in SA:

   1. wlc-cli-client --host-display-name hostDisplayName --service-display-name serviceDisplayName --host WLChostIPAddress --port 50101 --use-ssl false

      Enter following information:

      1. Legacy Windows Log Collector REST Username and Legacy Windows Log Collector REST Password: Enter the admin credentials for the Legacy Windows Log Collector.

      2. Security Server Username and Security Server Password: Enter admin credentials for NetWitness.

2. Restart the system.

User Entity Behavior Analytics

IMPORTANT: Every UEBA deployment when upgraded requires additional steps to complete the upgrade process. When you upgrade from 11.6.x to 11.6.x.x, you must follow UEBA instructions in the Upgrade Guide for 11.6.x.x, before you upgrade to 11.7.x.

1. Update the UEBA configuration using the following command from the UEBA machine.
   
   source /etc/sysconfig/airflow

   source $AIRFLOW_VENV/bin/activate

   OWB_ALLOW_NON_FIPS=on python /var/netwitness/presidio/airflow/venv/lib/python2.7/site-packages/presidio_workflows-1.0-py2.7.egg/presidio/resources/rerun_ueba_server_config.py

2. (Optional) Update the UEBA processing schema, if needed.

   NetWitness recommends that the UEBA start date is set to 28 days earlier than the current date. For UEBA systems that intend to process TLS data, you must make sure that the start date is set to no later than 14 days earlier than the current date.

   For more information, see the "reset-presidio script" section in the UEBA Configuration Guide.

3. Run the airflow upgrade DAG.

   • Go to Airflow main page https://<UEBA-host-name>/admin

   • Enter the admin username and password.

   • Click the Play in presidio_upgrade_dag_from_<previous_version> to_12.3.0.0.

     

     Note: A light green circle will appear next to the upgrade DAG row during the upgrade. If the upgrade process is completed successfully the light green circle changes to green. If the upgrade process fails, the light green circle changes to red.

4. Set the appropriate "Boot Jar Pools" slots:

   • Physical Appliance: Update the spring_boot_jar_pool slot value be 18.

   • Virtual Appliance: Update the spring_boot_jar_pool slot value to 22.
     To update the Spring Boot Jar Pools slots, go to the Airflow main page, tap the Admin tab at the top bar and tap Pools.
   1. To access the Airflow UI, go to https://<UEBA_host>/admin and enter the credentials.
      User: admin
      Password: The environment deploy admin password.
   2. Click on the pencil mark of the Pools to update the slot values.
      

      

5. Edit the spring_boot_jar_pool and update the slots amount to 22.
   

   

6. Import the Elasticsearch presidio data after upgrading the UEBA host from 12.0 and older versions to 12.3.0.0. Make sure the following prerequisites are met before you import the Elasticsearch presidio data:

• Elasticsearch version must be upgraded to 7.17.6 from 5.5.0.

• UEBA host must be upgraded to 12.3.0.0

• UEBA rpm version must be 12.3.0.0.

• Elasticsearch data in 12.0.0.0 or older versions must be exported and stored in the Elasticsearch data backup folder located in the /root/ directory.

  To import the Elasticsearch data:

1. Go to cd ueba_es_migration_tool. Run the following command.

   sh elk-migration-script.sh

   The Elasticsearch migration tool guide is displayed.

   

2. Select Import documents to elasticsearch 7.x from backup.

3. In the next step, select Fresh Import to import the backup data.

   

4. Restart the Presidio UI service once the Import operation is completed. Run the following command.

   systemctl restart presidio-ui

5. Go to the NetWitness Platform XDR Users tab and verify if all the Elasticsearch data is imported.

   Note:
   - Go to <backup_directory_path>/log/log/es-migration-import.log if you want to view the log for any exceptions.
   - If the Import operation fails due to some technical issue, select Resume Import once the issue is resolved, to resume the Import operation.