Update Your ESA Rules for the Required Multi-Value and Single-Value Meta Keys

11.4 Update Your ESA Rules for the Required Multi-Value and Single-Value Meta Keys

Note: This procedure applies only to ESA Correlation Rules in NetWitness Platform 11.3.0.2 and later versions.

To support Endpoint, UEBA, and RSA Live content, a data change from single-value (string) to multi-value (string array) is required for several meta keys within the ESA Correlation service for 11.3 and later. Additional string meta keys are required within the ESA Correlation service for 11.3.0.2 and later.

If the meta keys used for your ESA rules are different from the required default multi-value meta keys, your ESA rules continue to work, but you should update your ESA rules to use the required meta keys as soon as possible to ensure that your rules continue to deploy properly.

Note: On a new installation of ESA on 11.3.0.2 and later, no ESA rule adjustments are necessary.

The ESA Correlation service has the following multi-valued (string array) and single-valued (string) parameters:

  • multi-valued - Shows the string array meta keys currently used for your ESA rules.
    • For an upgrade to NetWitness Platform 11.3.0.2 and later, it shows the existing string array meta keys before the upgrade. (This parameter is equivalent to the Event Stream Analysis service ArrayFieldNames parameter in NetWitness Platform versions 11.2 and earlier.)

    • For a new installation of NetWitness Platform 11.3.0.2 and later, it contains all the required string array meta keys for the latest version.

  • single-valued - Shows the string meta keys currently used for your ESA rules.
    • For an upgrade to NetWitness Platform 11.3.0.2 and later from versions prior to 11.3, this parameter value is empty.

    • For a new installation of NetWitness Platform 11.3.0.2 and later, it contains all the required string meta keys for the latest version.

  • default-multi-valued - Shows the required string array meta keys for the latest version.

    • For a new installation of NetWitness Platform 11.3.0.2 and later, this parameter value is empty.

  • default-single-valued - Shows the required string meta keys for the latest version.

    • For a new installation of NetWitness Platform 11.3.0.2 and later, this parameter value is empty.

Note: If you have the same value in the single-valued and multi-valued parameter fields, the single-valued meta key value takes precedence over the multi-valued meta key value.

To use the latest Endpoint, UEBA, and Live content rules, you must update the multi-valued parameter on the ESA Correlation service to include all of the meta keys in the default-multi-valued field. You must also update the single-valued parameter field to include all of the meta keys in the default-single-valued field. To do this, follow the Update the Multi-Valued and Single-Valued Parameter Meta Keys for the latest Endpoint, UEBA, and RSA Live Content Rules procedure.

Caution: Any changes that you make to the multi-valued parameter may cause an error when you deploy your existing rules. You can update the multi-valued parameter, resync your meta keys, and update the ESA rules at your convenience. You may want to add a couple meta keys at a time to reduce the number of reported errors.

Note: If you are using multiple ESA Correlation services, the multi-valued and single-valued parameters should be the same on each ESA Correlation service.

In NetWitness Platform 11.3.0.2 and later, ESA automatically adjusts the operator in the rule statement when there is a change from string to string array, but you still may need to make manual adjustments to adjust for the string array changes.

To change the string type meta keys to string array type meta keys manually, see Configure Meta Keys as Arrays in ESA Correlation Rule Values.

To use the latest Endpoint, UEBA, and Live content rules, the following default multi-valued meta keys are required on the ESA Correlation service in NetWitness Platform version 11.3 and later:

action , alert , alert.id , alias.host , alias.ip , alias.ipv6 , analysis.file , analysis.service , analysis.session , boc , browserprint , cert.thumbprint , checksum , checksum.all , checksum.dst , checksum.src , client.all , content , context , context.all , context.dst , context.src , dir.path , dir.path.dst , dir.path.src , directory , directory.all , directory.dst , directory.src , email , email.dst , email.src , eoc , feed.category , feed.desc , feed.name , file.cat , file.cat.dst , file.cat.src , filename.dst , filename.src , filter , function , host.all , host.dst , host.orig , host.src , host.state , inv.category , inv.context , ioc , ip.orig , ipv6.orig , netname , OS , param , param.dst , param.src , registry.key , registry.value , risk , risk.info , risk.suspicious , risk.warning , threat.category , threat.desc , threat.source , user.agent , username

The following default single-valued meta keys are also required on the ESA Correlation service in NetWitness Platform 11.3.0.2 and later:

accesses , context.target , file.attributes , logon.type.desc , packets

If you used any meta keys in the ESA rule notification templates from the Required String Array or String Meta Keys list, update the templates with the meta key changes. See "Configure Global Notification Templates" in the System Configuration Guide.

Note: Advanced EPL rules may get disabled and are not automatically updated so they must be fixed manually.

For additional troubleshooting information, see “Troubleshoot ESA” in the Alerting with ESA Correlation Rules User Guide.

Update the Multi-Valued and Single-Valued Parameter Meta Keys for the latest Endpoint, UEBA, and RSA Live Content Rules

To use the latest Endpoint, UEBA, and Live content rules, you must update the multi-valued parameter field on the ESA Correlation service to include all of the meta keys in the default-multi-valued field. You must also update the single-valued parameter field to include all of the meta keys in the default-single-valued field.

Caution: Any changes that you make to the multi-valued parameter may cause an error when you deploy your existing rules. You can update the multi-valued parameter, resync your meta keys, and update the ESA rules at your convenience. You may want to add a couple meta keys at a time to reduce the number of reported errors.

Note: If you see a warning message in the ESA Correlation server error logs for missing multi-valued meta keys, there is a difference between the default-multi-valued parameter and multi-valued parameter meta key values, and the new Endpoint, UEBA, and Live content rules will not work. The same is true for missing single-valued meta keys. Completing this procedure should fix the issue. For example warning messages, see Example ESA Correlation Server Warning Message for Missing Meta Keys.

  1. After an upgrade to 11.3.0.2 or later, go to netwitness_adminicon_25x22.png (Admin) > Services, and in the Services view, select an ESA Correlation service and then select netwitness_ic-actns.png > View > Explore.
  2. In the Explore view node list for the ESA Correlation service, select correlation > stream.
  3. Compare the multi-valued parameter meta keys with the required default-multi-valued meta keys. Copy and paste the missing string array meta keys from the default-multi-valued parameter to the multi-valued parameter. (You may want to copy only a couple meta keys at one time to reduce the number of reported errors).
  4. Copy and paste the string meta keys from the default-single-valued parameter to the single-valued parameter.
  5. If you have multiple ESA Correlation services, make the same meta key changes on each ESA Correlation service.
  6. Apply the changes on the ESA Correlation services:
    1. Go to netwitness_configureicon_24x21.png (Configure) > ESA Rules and click the Settings tab.
    2. In the Meta Key References, click the Meta Re-Sync (Refresh) icon (netwitness_ic-refresh.png ).
  7. If you are using any of the default-multi-valued or default-single-valued meta keys in your ESA Advanced rules, update the rule syntax. See also Adjust Custom ESA Rule Builder and ESA Advanced Rules.
  8. If you used any meta keys in the ESA rule notification templates from the default-multi-valued parameter list, update the templates with the meta key changes. See "Configure Global Notification Templates" in the System Configuration Guide.
  9. Deploy your ESA rule deployments.
  10. Check your rules for error messages in the ESA Rules section of the ESA rule Deployment or check the ESA Correlation error logs for errors.
    • (This option is available in NetWitness verson 11.3.0.2 and later.) To access the error messages in the ESA rule deployment, go to netwitness_configureicon_24x21.png (Configure) > ESA Rules > Rules tab, select a deployment in the options panel on the left, and go to the ESA Rules section. If the ESA rule status shows “Disabled” or shows the netwitness_depupdicon.png icon in the Status column, you need to determine the issue to fix the rule. If a disabled rule has an error message, it shows netwitness_depupdicon.png in the Status field. You can hover over the rule to view the error message tooltip without going to the error log.
    • To access the ESA Correlation service logs, you can use SSH to get in the system and go to: /var/log/netwitness/correlation-server/correlation-server.log.

Adjust Custom ESA Rule Builder and ESA Advanced Rules

Update your ESA Rule Builder and ESA Advanced rules to work with the string and string array meta keys listed in the default-multi-valued and default-single valued parameter fields for the ESA Correlation service. You can add additional meta keys to the multi-valued and single-valued parameters.

For example, if you use ec.outcome as a single-valued meta key in your ESA rule as shown below:

@RSAAlert

SELECT * FROM Event((ec_outcome IN ( 'Success' )))

.win:time_length_batch(2 Minutes, 2)

HAVING COUNT(*) >= 2;

If you add ec.outcome to the multi-valued parameter field, you need to update your rule as shown below:

@RSAAlert

SELECT * FROM Event(( 'Success' = ANY( ec_outcome ) ))

.win:time_length_batch(2 Minutes, 2)

HAVING COUNT(*) >= 2;

For more information, see Configure Meta Keys as Arrays in ESA Correlation Rule Values.

Example ESA Correlation Server Warning Message for Missing Meta Keys

If you see a warning message in the ESA Correlation server error logs for missing multi-valued meta keys, there is a difference between the default-multi-valued parameter and multi-valued parameter meta key values, and the new Endpoint, UEBA, and Live content rules will not work. The same is true for missing single-valued meta keys. Completing the Update the Multi-Valued and Single-Valued Parameter Meta Keys for the latest Endpoint, UEBA, and RSA Live Content Rules procedure should fix the issue.

Multi-Valued Warning Message Example

2019-08-23 08:55:07,602 [ deployment-0] WARN Stream|[alert, alert_id, browserprint, cert_thumbprint, checksum, checksum_all, checksum_dst, checksum_src, client_all, content, context, context_all, context_dst, context_src, dir_path, dir_path_dst, dir_path_src, directory, directory_all, directory_dst, directory_src, email_dst, email_src, feed_category, feed_desc, feed_name, file_cat, file_cat_dst, file_cat_src, filename_dst, filename_src, filter, function, host_all, host_dst, host_orig, host_src, host_state, ip_orig, ipv6_orig, OS, param, param_dst, param_src, registry_key, registry_value, risk, risk_info, risk_suspicious, risk_warning, threat_category, threat_desc, threat_source, user_agent] are still MISSING from multi-valued

Single Value Warning Message Example

2019-08-23 08:55:07,602 [ deployment-0] WARN Stream|[accesses, context_target, file_attributes, logon_type_desc, packets] are still MISSING from single-valued

Configure Meta Keys on New ESA Hosts to Match Upgraded ESA Hosts in the Same NetWitness Platform Network

Note: If you are upgrading from version below 11.4.x.x, make sure that meta keys on new ESA hosts are updated to match upgraded ESA Hosts in the Same NetWitness Platform Network.

If you have one or more ESA hosts in a NetWitness Platform network, which were upgraded from a version before 11.3.0.2 to 11.6, and you add a new ESA host, you must configure the meta keys on the new ESA host to match the other ESA hosts. All ESA Correlation services on the same NetWitness Platform network must have the same Meta Key configurations.

  1. For each ESA Correlation service on an upgraded ESA host and for the ESA Correlation service on the newly installed ESA host:
    1. Open a new tab, go to netwitness_adminicon_25x22.png (Admin) > Services, and in the Services view, select the ESA Correlation service and then select netwitness_ic-actns.png > View > Explore.

    2. In the Explore view node list for the ESA Correlation service, select correlation > stream.

  2. Ensure that the multi-valued and single-valued meta key values are the same on each of the upgraded ESA Correlation services.
  3. Ensure that the multi-valued and single-valued meta key values on the newly installed ESA host are the same as those on the upgraded services.
  4. To apply any changes on the ESA Correlation services, go to netwitness_configureicon_24x21.png (Configure) > ESA Rules and click the Settings tab. In the Meta Key References, click the Meta Re-Sync (Refresh) icon (netwitness_ic-refresh.png ).
  5. If you updated the ESA Correlation services, redeploy the ESA rule deployments.

For more information, see Update Your ESA Rules for the Required Multi-Value and Single-Value Meta Keys.