Post Upgrade Tasks for 12.5.1.0

[#1]

Perform Post Upgrade Tasks

This topic lists the tasks you must perform after upgrading NetWitness Platform. Complete the tasks that apply to the hosts in your environment.

• General
• Event Stream Analysis (ESA)
• Respond
• User and Entity Behavior Analytics

• Legacy Windows Log Collector

• Warehouse Connector

• Setting Recovery Password for Lockbox

General

You must configure Jetty, restore the core services contents, and also start Network capture, Log capture, and aggregation after upgrading NetWitness Platform.

Configure Jetty

For Jetty Configuration and related information, see Manage Custom Jetty Configuration section in Manage Custom Host Entries topic in the System Maintenance Guide.

Make Sure Services Have Restarted and Are Capturing and Aggregating Data

Make sure that services have restarted and are capturing data (this depends on whether or not you have auto-start enabled).

If required, restart data capture and aggregation for the following services:

• Decoder
• Log Decoder
• Broker
• Concentrator
• Archiver

To Start Network Capture:

1. In the NetWitness Platform menu, go to  (Admin) > Services.
   The Services view is displayed.
2. Select each Decoder service.

3. Under  (actions), select View > System.

4. In the toolbar, click 

To Start Log Capture:

1.  In the NetWitness Platform menu, go to  (Admin) > Services. The Services view is displayed.
2. Select each Log Decoder service.
3. Under  (actions), select View > System.
4. In the toolbar, click 

To Start Aggregation:

1. In the NetWitness Platform menu, go to  (Admin) > Services.

   The Services view is displayed.

2. For each Concentrator, Broker, and Archiver service:

   1. Select the service.
   2. Under  (actions), select View > Config.

   3. In the toolbar, click 

3. For Event Stream Analysis (ESA):

   Note: Mixed mode is not supported for ESA hosts. The NetWitness server, ESA primary host, and ESA secondary host must all be on the same NetWitness Platform version.

   There are no required post-upgrade tasks for ESA. For ESA troubleshooting, see ESA Troubleshooting Information.

   If you want to add support for Endpoint, UEBA, and Live content rules, you must update the multi-valued and single-valued parameter meta keys on the ESA Correlation service to include all the required meta keys. It is not necessary to make these adjustments during the upgrade; you can make the adjustments later at a convenient time. For detailed information and instructions, see Update Your ESA Rules for the Required Multi-Value and Single-Value Meta Keys in the ESA Configuration Guide.

   Restore the Core Services Contents

   Once you upgrade to 12.5.1.0, the Core services Contents such as Configuration files (.cfg), Feeds, Parsers, and Log Devices are copied to the .tar location of the respective components such as Decoder, Log Hybrid, Network Hybrid, and Log Decoder.

   The following table lists the Core Services Contents paths and the .tar location of the respective components where the Core Services Contents are copied.

   Core Services Contents Paths	Components	.tar location of the Components
   /etc/netwitness/ng/feeds (Feeds)	Decoder	/var/netwitness/decoder/decoder_backupcontent_ccm.tar
   /etc/netwitness/ng/parsers (Parsers)	Log Hybrid	/var/netwitness/logdecoder/logdecoder_backupcontent_ccm.tar
   /etc/netwitness/ng/envision/etc/devices (Log Devices)	Network Hybrid	/var/netwitness/decoder/decoder_backupcontent_ccm.tar
   /etc/netwitness/ng/NwDecoder.cfg (Configuration files (.cfg))	Log Decoder	/var/netwitness/logdecoder/logdecoder_backupcontent_ccm.tar

   By default, CCM option is disabled. After upgrading to 12.5.1.0, if you enable CCM and encounter the loss of the Core Services Contents, you can use the backup tar files to recover the lost data. For more information, see https://community.netwitness.com/t5/netwitness-knowledge-base/automatic-backup-of-core-service-content-after-upgrading-core/ta-p/694527.

Event Stream Analysis (ESA)

After upgrading to the 12.5.1.0 version, all the ESA deployments will be migrated to  (CONFIGURE) > Policies page. Each deployment will be converted into a policy and group and will be available to manage only after the upgrade of the Correlation servers to the 12.5.1.0 version. Make sure that you plan the upgrade process so that Correlation servers are upgraded immediately after the Admin Server is done. The deployments will not be accessible until the corresponding Correlation servers are upgraded. However, the correlation servers will still continue to process the Alerts and Events. Verify if all the ESA deployments are in a healthy state. For more information, see View a Deployment topic in the Live Services Management Guide.

Note: Analysts must have appropriate permissions to view the ESA rules under  (CONFIGURE) > ESA Rules and  (CONFIGURE) > Policies pages. For more information, see the Source-server section in the Role Permissions topic in the System Security and User Management Guide.

The pre-upgrade and post-upgrade states of deployments are represented in the following table.

SlNo	Pre-upgrade Deployment State	Post-upgrade Deployment State
		Creates Policy	Creates Group	The policy will be Published
1	Healthy deployment	Yes	Yes	Yes
2	Deployment with errors	Yes	Yes	Yes
3	Deployment with only rules	Yes	No	No
4	Deployment with no rules	No	No	No

(Optional) Using the Merge Policy button, you can merge a policy having ESA content with a policy with no ESA content. For more information, see Merge Policy with ESA Content topic in the Live Services Management Guide.

Manage ESA Deployments and Data Sources

You can only manage the ESA deployments and Data Sources through Centralized Content Management.

Go to (CONFIGURE) > Policies > Content > Event Stream Analysis page to manage the ESA deployments and Data Sources. You can only manage the ESA Rules in the ESA Rules page. Refer the following figures.

You must upgrade the ESA hosts immediately after upgrading the Admin Server.

Note: To generate ESA alerts with Mitre tactics and techniques, the ESA deployment associated with the ESA rule must be redeployed mandatorily after upgrading to the 12.5.1.0 version.

Migrate custom scripts for notifications

• Since there are a wide range of changes regarding the file permissions and ownership attributes of custom script files of the Script Notifications feature (  (Admin) > System > Global Notifications > Script), Netwitness suggests having a backup of the custom scripts before the system is upgraded to 12.5.1.0.

• Once the upgrade is completed, each script needs to be revisited for syntactic/semantic changes that have to be done.

• Even if the custom script in NetWitness versions prior to 12.5.1.0 is accessing any file resources in the /tmp or /var/tmp folder, they cannot be accessed further since the ownership with which the custom script executes has been changed. The suggestion for this scenario is to tweak/modify the custom script to create/read from a new file in /tmp or /var/tmp directory.

For more information on Centralized Content Management and managing the deployments, see Centralized Content Management Guide for NetWitness.

Respond

The Primary ESA server must be upgraded to 12.5.1.0 before you can complete the following task.

Note: After upgrading the primary NW Server (including the Respond Server service), the Respond Server service is not automatically re-enabled until after the Primary ESA host is also upgraded to 12.5.1.0. The Respond post-upgrade tasks only apply after the Respond Server service is upgraded and is in the enabled state.

(Conditional) Restore Any Respond Service Custom Keys in the custom_normalize_alerts.js and support new datasource

Note: If you did not manually customize the custom_normalize_alerts.js, you can skip this task. We attempt to automatically migrate the custom keys. However, in case of failures, use this step to verify the custom data's integrity.

If you added custom keys in the /var/netwitness/respond-server/scripts/custom_normalize_alerts.js file for use in custom normailization, modify the /var/netwitness/respond-server/scripts/custom_normalize_alerts.js file and add the custom normalized keys from the automatic backup file. The backup file is located in /var/netwitness/respond-server/scripts and it is in the following format:

custom_normalize_alerts.js.bak-<time of the backup>

In case of automatic update of the script fails, add support for Netwitness Core and NetWitness Insight by updating the custom_normalize_alerts.js file manually to support these new sources in respond.

 

User and Entity Behavior Analytics

Complete the following tasks after upgrading UEBA to 12.5.1.0.

Important: After completing the upgrade, NetWitness recommends users to manually update any configuration values (e.g., xmx value) in the workflows-default.jsonfile instead of replacing the backup file taken before the upgrade. Replacing the backup file (workflows-default.json) could result in displaying outdated features or configurations that are no longer available, leading to potential issues.

1. Update the UEBA configuration using the following command from the UEBA machine.

   • source /etc/sysconfig/airflow

   • source $AIRFLOW_VENV/bin/activate

   • python /var/netwitness/presidio/airflow/venv39/lib/python3.9/site-packages/presidio_workflows-1.0-py3.9.egg/presidio/resources/rerun_ueba_server_config.py

   • deactivate

2. (Optional) Update the UEBA processing schema, if needed.

   NetWitness recommends that the UEBA start date is set to 28 days earlier than the current date. For UEBA systems that intend to process TLS data, you must make sure that the start date is set to no later than 14 days earlier than the current date.

   For more information, see the "reset-presidio script" section in the UEBA Configuration Guide.

3. Run the airflow upgrade DAG.

   • Go to Airflow main page https://<UEBA-host-name>/admin

   • Enter the admin username and password.

   • Click the Play in presidio_upgrade_dag_from_<previous_version> to_12.5.1.0.

     

     Note: A light green circle will appear next to the upgrade DAG row during the upgrade. If the upgrade process is completed successfully the light green circle changes to green. If the upgrade process fails, the light green circle changes to red.

4. Set the appropriate "Boot Jar Pools" slots:

   • Physical Appliance: Update the spring_boot_jar_pool slot value be 18.

   • Virtual Appliance: Update the spring_boot_jar_pool slot value to 22.
     To update the Spring Boot Jar Pools slots, go to the Airflow main page, tap the Admin tab at the top bar and tap Pools.
   1. To access the Airflow UI, go to https://<UEBA_host>/admin and enter the credentials.
      User: admin
      Password: The environment deploy admin password.
   2. Click on the pencil mark of the Pools to update the slot values.

      

     5. Edit the spring_boot_jar_pool and update the slots amount to 22.

         

Legacy Windows Log Collector

Refresh Legacy Windows Log Collector Certificates with Updated SA Certificates

Post Upgrade Steps:

1. Execute the following command in Admin Server:

   1. wlc-cli-client --host-display-name hostDisplayName --service-display-name serviceDisplayName --host WLChostIPAddress --port 50101 --use-ssl false

      Enter following information:

      1. Legacy Windows Log Collector REST Username and Legacy Windows Log Collector REST Password: Enter the admin credentials for the Legacy Windows Log Collector.

      2. Security Server Username and Security Server Password: Enter admin credentials for NetWitness.

2. Restart the system.

Warehouse Connector

The Warehouse connector uses a lockbox to store credentials securely for data integration sources and destinations. However, users upgrading from earlier versions to the 12.5.1.0 version cannot start the configured streams without migrating their existing credentials in the new lockbox. As a result, users must manually create a new lockbox key and then refresh the password for their sources and destinations configured in Warehouse Connector, wherever applicable, using the following steps :

1. Log in to the NetWitness Platform.

2. Navigate to  (Admin) > Services.

3. In the Services view, select the added Warehouse Connector service, and select > View > Config.

4. In the Services Config view of Warehouse Connector, click the Lockbox Settings tab and create a fresh lockbox key.

   

5. Reauthorize the user account in source configurations using Explore or REST API. Reauthorization of source user account is not available in UI. The command to reauthorize the user account from Explore is given below:

   > /warehouseconnector/sources/<source:port> ---> setPass property with password=<password of the configured user in source>

6. Reauthorize the user account in SFTP destination configuration from UI, Explore or REST API. The command to reauthorize the password from Explore is given below:

   > /warehouseconnector/destinations/<sftp_destination> ---> setPass property with password=<password of the configured user in SFTP destination>

7. If NFS directory mount was removed as part of pre-upgrade step, mount back the same configuration. Additionally, enable back the mount entry in /etc/fstab.

> mount -t nfs -o nolock,tcp,hard,intr <IP_Address_for_SAW>:/mapr/<cluster-name> /<directory_name>

Where <IP_Address_for_SAW> is the IP address of the primary Warehouse appliance in the cluster and <cluster-name> is the name provided in the template file.

Setting Recovery Password for Lockbox

After upgrading all the Log Collectors and WLC in the NW deployment to 12.5.1.0, administrator should execute the Recovery Password Utility using SATools. This tool sets the Lockbox recovery password for all the log collector services (version 12.5.1.0)  within the deployment. Administrators are advised to keep a note of the Lockbox recovery password as required during disaster recover scenarios.

To set the recovery password, the administrator should SSH into Admin Server(Node 0) and execute the utility set-lockbox-password from the path /opt/rsa/saTools/bin/set-lockbox-password. Administrators should enter the new password to be set as recovery password for the Lockbox available in all the Log Collectors.

Note: On re-executing the utility and setting an updated Lockbox, the recovery password will reset the password for all the applicable log collector services (version 12.5 and above).

Perform Validation Checks After Upgrade

You must perform the following validation checks after upgrading to NetWitness 12.5.1.0

1. Go to  (Admin) > Services view to verify that all the services are active (appearing in green) after upgrade.

2. Verify that the services are upgraded to match the host version. The service version in  (Admin) > Services view must match the host version in  (Admin) > Hosts view after upgrade.

3. In the  (Admin) > Services view, do the following:

   • Select a Log Collector service and go to  (actions) > View > System view to verify if the required logs collection is started. You should click the  drop-down option and go to the right collection protocol to check if the logs collection is started. If the required collection is not started, select  next to the required collection protocol from the list to start the collection.

   • Select a Log Decoder service and go to  (actions) > View > System view to verify if the Log Decoder is capturing the logs properly.

   • Select a Packet Decoder service and go to  (actions) > View > Config view to check if the capture interface is configured under Decoder Configuration section. If the capture interface is not configured, you must select the required capture interface from the drop-down list to configure it. If the capture interface is already configured, go to the  (actions) > View > System view of the Packet Decoder and check if the capture is started. If the capture is not started, click to start the packet capture.

4. Go to  (Admin) > Services > Select a Log Decoder or Packet Decoder service >  (actions) > View > Stats > General view to analyze the current capture rate.

5. Verify that the Concentrators, Archivers, and Brokers are aggregating the data. Make sure that you can investigate from each Concentrator, Archiver, and Broker to validate that it is operational.

6. Go to Respond > Alerts view to verify if the alerts are triggering from different sources.

7. Go to  (Admin) > Health & Wellness > Alarms view and verify if the SMS server is up and running.

8. Go to  (Admin) > Event Sources > Monitoring Policies view and verify if the policies configured before upgrade are appearing.

9. Go to  (Admin) > Health & Wellness > New Health & Wellness > Pivot to Dashboard > Elastic > Dashboard view and ensure the following.

   • The visualizations you created before upgrade still exist.

   • The metric server is up and running.

   • Alerts are generated properly for the monitors you have configured before upgrade.