Configure an In-Memory Table as an Enrichment Source

This topic provides instructions on how to configure an in-memory table. When you configure an in-memory table, you upload a .CSV file as an input to the table. You can associate this table with a rule as an enrichment source. When the associated rule generates an alert, ESA will enrich the alert with relevant information from the in-memory table.

For example, a rule could be configured to detect when a user tries to download freeware and to identify the person by user ID in the alert. The alert could be enriched with additional information from an in-memory table that contains details such as full name, title, office location and employee number.

Note: It is preferable to use Context Hub List enrichment sources instead of In-Memory Table enrichment sources for rules. Recurring In-Memory Tables are no longer supported; use Content Hub Lists as enrichment sources. For more information, see Configure a Context Hub List as an Enrichment Source.

Prerequisites

  • The column name in the .CSV file cannot have whitespace characters.
    For example Last_Name is correct, and Last Name is incorrect.
  • The .CSV file must begin with a header line that defines fields and types.
    For example, address string would define the header field as address, and the type as string.

The following shows a valid .CSV file represented as a .CSV and as a table.

netwitness_valid_csv_esa_541x233.png

Configure an Ad hoc In-Memory Table

Note: It is preferable to use Context Hub List enrichment sources instead of In-Memory Table enrichment sources for rules. Recurring In-Memory Tables are no longer supported; use Content Hub Lists as enrichment sources. For more information, see Configure a Context Hub List as an Enrichment Source.

  1. Go to netwitness_configureicon_24x21.png (Configure) > ESA Rules.
    The Configure view is displayed with the ESA Rules tab open.
  2. Click the Settings tab.
  3. In the options panel, select Enrichment Sources.
    netwitness_121_enrsources_1122_768x341.png
  4. In the Enrichment Sources section, click netwitness_ic-addlist.png > In-Memory Table.
    netwitness_imtbladhoc_480x468.png
  5. Describe the in-memory table:
    1. Select Ad hoc.
    2. By default, Enable is selected. When you add the in-memory table to a rule, alerts will be enriched with data from it.
      If you add an in-memory table to a rule but do not want alerts to be enriched, deselect the checkbox.
    3. In the User-Defined Table Name field, type a name, such as Student Information, for the in-memory table configuration.

    Note: Do not use any Esper keyword as User-Defined Table Name since this causes an error while using this enrichment in the ESA Rule. For Esper keywords, see Reserved keywords.

    1. If you want to explain what the enrichment adds to an alert, type a Description such as:
      When an alert is grouped by Rollno, this enrichment adds student information, such as name and marks.
  6. In the Import Data field, select the .CSV file that will feed data to the in-memory table.
  7. If you want to write an EPL query to define an advanced in-memory table configuration, select Expert Mode.
    The Table Columns are replaced by a Query field.
  8. In the Table Columns section, click netwitness_ic-add.png to add columns to the in-memory table.
  9. If a valid file is selected in the Import Data field, the columns populate automatically.

Note: If you selected Expert mode, a Query field is displayed instead of Table Columns.

  1. In the Key drop-down menu, select the field to use as the default key to join incoming events with the in-memory table when using a CSV-based in-memory table as an enrichment. By default, the first column is selected. You can also later modify the key when you open the in-memory table in enrichment sources.
  2. In Max Rows drop-down menu, select the number of maximum number of rows that can reside in the in-memory table at a particular instance.
  3. Select Persist to preserve the in-memory table on disk when the ESA service stops and to re-populate the table when the service restarts.
  4. In Stored File Format field, do one of the following:
    • Select Object, if you want to store the file in a binary format.
    • Select JSON, if you want to store the file in a text format.
      By default, Object is selected.
  5. Click Save.
    The adhoc in-memory table is configured. You can add it to a rule as an enrichment or part of the rule condition. See Add an Enrichment to a Rule.

When you add an in-memory table, you can add it to a rule as an enrichment or as a part of the rule condition. For example, the following rule uses an in-memory table as a part of the rule condition to create a whitelist, and it also uses an in-memory table of details in the user_dst file to enrich the alert that is displayed.

The rule shows the in-memory table as a whitelist rule condition:

netwitness_in-rule-enrichment_576x277.png

Next, the alert is enriched with the User_list in-memory table:

netwitness_post_alert_enrichment_576x61.png

Therefore, the user_dst in-memory table is used to create a whitelist, and it is also used to enrich the data in the alert if the alert is triggered.

​​Add a Recurring In-Memory Table

Recurring In-Memory Tables are no longer supported; use Content Hub Lists as enrichment sources. For more information, see Configure a Context Hub List as an Enrichment Source.

It is preferable to use Context Hub List enrichment sources for ESA rules instead of In-Memory Table enrichment sources. You can share Context Hub List enrichment sources across the NetWitness Platform. You can only use the In-Memory Table with ESA.

Note: Database, Database Connection, Warehouse Analytics, and Recurring In-Memory Tables as enrichment sources are not supported for the ESA Correlation service in NetWitness 11.3 and later.