Add an Advanced EPL Rule

This topic provides instructions to define rule criteria by writing an EPL query. EPL is a declarative language for handling high-frequency time-based event data. It is used to express filtering, aggregation, and joins over possibly sliding windows of multiple event streams. EPL also includes pattern semantics to express complex temporal causality among events.

Write an advanced EPL rule when rule criteria is more complex than what you can specify in Rule Builder.

It is outside the scope of this guide to explain EPL syntax.

For best practices on writing advanced EPL rules, see ESA Rule Writing Best Practices.

Prerequisites

The following are prerequisites for adding an advanced rule:

  • You must know Event Processing Language (EPL).
  • You must understand ESA Annotations to mark which EPL statements are linked to generating alerts.

Add an Advanced EPL Rule

  1. Go to netwitness_configureicon_24x21.png (Configure) > ESA Rules.

  2. In the Rule Library, select netwitness_ic-addlist.png > Advanced EPL.
    netwitness_121_adveplbldr_esacorr_1122_768x434.png

  3. Type a unique, descriptive name in the Rule Name field.

    This name will appear in the Rule Library so be specific enough to distinguish the rule from others.

  4. In the Description field, explain which events the rule detects.

    The beginning of this description will appear in the Rule Library

  5. Select Trial Rule to automatically disable the rule if all trial rules collectively exceed the memory threshold.

    Use trial rule mode as a safeguard to see if a rule runs efficiently and to prevent downtime caused by running out of memory. For more information, see Working with Trial Rules.

  6. (This option applies to 11.5 and later.) Enter a Memory Threshold for a rule that uses memory, such as a rule that contains windows or pattern matching. If the configured memory threshold is exceeded, the rule gets disabled individually and an error is displayed for that rule on the netwitness_configureicon_24x21.png (Configure) > ESA Rules > Services tab. The Memory Threshold option works for trial rules and non-trial rules. New rules default to a 100 MB memory threshold. Rules that existed before version 11.5 do not have a default value and a memory threshold is not set.
  7. (This option applies to 11.3 and later.) Select Alert to send an alert to Respond. Clear the checkbox if you do not want to send an alert to Respond. To turn alerts on or off for ALL rules, see the ESA Configuration Guide.
  8. For Severity, classify the rule as Low, Medium, High or Critical.

  9. To define rule criteria, write a Query in EPL.

    Note: For all meta key names, use an underscore not a period. For example, ec_outcome is correct but ec.outcome is not.

    Supported meta entities:

    S. No Supported Meta Entities Description
    1 fullname_all
    2 eth_all
    3 ip_all Combines all the IPv4 meta keys.
    4 ipv6_all Combines all the IPv6 meta keys.
    5 port_src_all
    6 port_dst_all
    7 dir_path_all
    8 org_all
    9 geoip_all
    10 port_all
    11 domain_all
    12 email_all
    13 filename_all
    14 directory_all
    15 checksum_all
    16 param_all
    17 context_all
    18 attack_all
    19 analysis_all
    20 compromise_all
    21 inv_all
    22 outcome_all
    23 ec_all
    24 user_all
    25 host_all
    26 client_all

  10. For dynamic statement name generation in ESA, you must enclose the meta keys in curly brackets and include this annotation in the syntax:

    @Name("RIG {ip_src} {alias_host} {ec_activity}")

    where,

    • RIG is the static part of the statement name
    • {ip_src}, {alias_host}, {ec_activity} is the dynamic part of the statement name

    Note: If any of the metas in the dynamic part of the statement name has a null value, it is displayed as a static text.

    If a rule should generate an alert, include this ESA annotation in the syntax:

    @RSAAlert

    For more information on ESA Annotations, see ESA Annotations.

Validate an Advanced EPL Rule

You can confirm that an ESA rule generates the expected alerts by testing the rule logic using JSON input data. You can view the alerts in the output, but this test does not send any alert notifications.

If you want to view all of the debug information, include an @Audit(‘stream’) annotation to your rule query and view the Debug Log in the test output. To enable auditing you require to add @Audit to the rule and set the logging level for the Esper audit package to INFO. This can be done by creating correlation-server.yml file under /etc/netwitness/correlation-server with this content and restarting the correlation-server service with systemctl restart rsa-nw-correlation-server:

logging:

level:

com.espertech.esper.audit: INFO

The following basic example query contains the @Audit(‘stream’) annotation and queries for events that do not have a source IP of 1.1.1.1 or 2.2.2.2.

netwitness_advauditexample_768x172.png

Validate an ESA Rule

  1. If you are not already in the rule, go to netwitness_configureicon_24x21.png (Configure) > ESA Rules > Rules tab and in the Rule Library, open the ESA rule that you want to test.
  2. Scroll down to the Test Rule section.
    netwitness_testrulesection_576x252.png
  3. In the ESA Service field, select the ESA Correlation service to process the rule. Use the same ESA Correlation service that you plan to use in the ESA rule deployment that contains the rule.
  4. In the Input Data field, enter the input events to test the rule. Download the events from the Investigate view in JSON format, copy the events, and paste them in this field. You can do this from the Investigate > Navigate view or the Investigate > Events view.

    To download the events from the Investigate > Navigate view:
    1. In the main menu, go to Investigate > Navigate in a new tab, select a data source, and click Navigate.
    2. In the Navigate view, click Load Values and click a meta value to filter the events.
    3. Save the events as meta in the JSON file format [Save Events > Meta > (name the file) > Export Meta Format: choose JSON].
    4. In the toolbar click the netwitness_jobsicon_22x21.png (Jobs) icon and then click View Your Jobs.
    5. In the Jobs panel, download your extracted meta, for example: investigation-2020-May-19-08-30-20.json.
    6. Go to back to the netwitness_configureicon_24x21.png (Configure) > ESA Rules tab opened previously and copy the contents of the JSON file into the Input Data field in your ESA rule.

    To download the events from the Investigate > Events view:

    1. In the main menu, go to Investigate > Events in a new tab.
    2. In the Events view, enter a query for the ESA rule test.
    3. Select the events to use and in the Download or Download All menu, select Visible Meta as JSON or All Meta as JSON, depending on the size of your selection.
    4. In the main menu, go to Dashboards and in the toolbar click the netwitness_jobsicon_22x21.png (Jobs) icon and then click View Your Jobs.
    5. In the Jobs panel, download your extracted meta, for example: Concentrator_ALL_EVENTS_ALL_META.json.
    6. Go to back to the netwitness_configureicon_24x21.png (Configure) > ESA Rules tab opened previously and copy the contents of the JSON file into the Input Data field in your ESA rule.
  5. Click Test Rule. The Output field shows the output of your rule and you can determine if the results meet your requirements.

netwitness_advrb_testrulesection_576x474.png

Note: If you are testing any Rule that has meta key defined as type 'short', the Test Rule will not generate alert for the event.

The following table describes the test rule output Engine Stats.

Field Description
Engine Version Esper version running on the ESA service
Events Offered Number of events processed by the ESA service since the last service start
Offered Rate The rate that the ESA service processes current events / The maximum rate that the ESA service processed events
Runtime Errors If applicable, this field can contain a link to runtime error messages related to the ESA rule deployment.

The following table describes the test rule output Rule Stats.

Field Description
Deployed A green checkmark indicates that the rule is deployed on the selected ESA service.
Statements Fired The number of statements that fired the alerts
Alerts Fired The number of alerts generated from the test data
Events in Memory The number of events placed in memory by the rule
Memory Usage The total amount of memory used by the rule
CPU % The percentage of the deployment CPU used by the rule. For example, a deployment with 1 rule shows 100% CPU usage for that rule and a deployment with two equally CPU heavy rules show 50% each.
Events Matched The number of events that matched the rule
Alerted Events If applicable, this field can contain a link to events that caused an alert.
Runtime Errors If applicable, this field can contain a link to runtime error messages related to the rule.
Debug Logs This field contains a link to Esper debug (audit) logs.