Configure an In-Memory Table Using an EPL Query

Note: It is preferable to use Context Hub List enrichment sources instead of In-Memory Table enrichment sources for rules. Recurring In-Memory Tables are no longer supported; use Content Hub Lists as enrichment sources. For more information, see Configure a Context Hub List as an Enrichment Source.

When you use an In-Memory Table configuration in expert mode, you can create an enrichment source or named window based on an Esper query. This allows you to have more control over the content and create more dynamic content. When you do this, an EPL query constructs the named window to capture interesting states from the event stream.

Workflow

The following shows the workflow for creating a query using a named window:

  1. The event is sent to the Esper Engine.
  2. An EPL query is generated.
  3. An alert is triggered.
  4. The query checks to see if there is a connection between the event and the Named Window.
  5. If there is a connection, the query that populates the Named Window is run and populated.
  6. The content from the Named Window is added to the alert content and sent or displayed (depending on your settings).

netwitness_esper_query_576x276.png

Prerequisites

  • The meta used in the EPL statement must exist in the data.
  • You must create well-formed EPL statements.

Procedure

Note: It is preferable to use Context Hub List enrichment sources instead of In-Memory Table enrichment sources for rules.

  1. Go to netwitness_configureicon_24x21.png (Configure) > ESA Rules.
    The Configure view is displayed with the Rules tab open.
  2. Click the Settings tab.
  3. In the options panel, select Enrichment Sources.
  4. In the Enrichment Sources section, click netwitness_ic-addlist.png > In-Memory Table.
    netwitness_imtabledg2_480x404.png
  5. Select Adhoc.
    By default, Enable is selected. When you add the in-memory table to a rule, alerts will be enriched with data from it.
  6. In the User-Defined Table Name field, type a descriptive name to describe the in-memory table.
  7. If you want to explain what the enrichment adds to an alert, enter information in the Description field.
    This description displays when you view the list of enrichments from the Enrichment Sources view, so it's a good idea to enter a thorough description as a best practice. Doing this allows other users to understand the content of the enrichment without opening it to examine its contents.
  8. Select Expert Mode to define an advanced in-memory table configuration by writing an EPL query.
    The Table Columns are replaced by a Query field.
  9. Select Persist to preserve the in-memory table on disk when the ESA service stops and to re-populate the table when the service restarts.
  10. Enter the EPL query in the Query field. The query should be well-formed, and it's a good idea to test it before entering it in the field.
  11. Click Save.

Example

For example, you want to know when an IPS or IDS is giving five or more inbound events with an event identified with malicious code. Additionally, you would like to know when the source IP of those events has been identified as suspicious by other sources. This information helps to more quickly triage the event and determine whether the alert is a true positive.

Step 1: Create the Enrichment

In this example, this enrichment is a watchlist of IPs that have been identified as suspicious by third party sources or by internal staff. The meta of threat_desc equal to ‘suspicious ip’ is generated when a match to a feed occurs. This meta can be matched and output based on a log, packet, or endpoint event.

The enrichment should look like the following:

netwitness_imtabledg.png

Parameters Description
Upload Type Adhoc
IP_Watchlist IP_Watchlist
Description Dynamically populated whitelist based on a feed of IPs that are considered suspicious.
Expert Mode Selected

Query

create window IpWatchlist .std:unique(ip_src) as (ip_src string, threat_source string, threat_category string);

insert into IpWatchlist

select ip_src, threat_source, threat_category from Event

where threat_desc = 'suspicious ip';

Step 2: Create Your Rule

First, you need to create your ESA Correlation rule. This example rule looks for inbound IPS or IDS log events with the event_cat_name beginning with Attacks.Malicious Code. If five or more events for the same ip_src occur within 60 minutes, then an alert will be triggered. If an ip_src from the Enrichment equals the ip_src from the alert, then that alert will be enriched with additional meta. In this case, the analyst would see the values for threat_source and threat_category in the raw alert. Threat_category would indicate the type of malware and threat_source would indicate the entity that has reported the ip as suspicious. The analyst could use this information to do additional research or escalate to the next tier for creation of a possible incident.

Rule Statement

netwitness_imtablestmt_768x295.png

Rule Logic with Enrichment Added

netwitness_imtableruleenr_768x605.png