Create a Custom Feed

You can create a custom feed using the Custom Feed wizard. To complete this procedure, you need a feed data file in .csv or .xml format. If you also have an associated feed definition file in .xml format, which describes the structure of the feed data file, you can use the feed definition file to create a feed. The Custom Feed wizard can create the feed based on a feed data file, or based on a feed data file and corresponding feed definition file.

Before you create or modify custom feeds, keys, or data types, ensure that you have a good understanding of the relationship between keys and datatypes in databases. It is important that any custom keys that you create match the corresponding datatypes. If your feed utilizes a custom meta key, you must define that key’s data type in index-decoder-custom.xml. This is to ensure that the key data is generated with correct data format.

Also, to enable investigation on the custom meta key, you must define its index on the concentrator, using the index-concentrator-custom.xml file. For more information on how to write the index-decoder-custom.xml or index-concentrator-custom.xml file, see the topic "Index Customization" in the Core Database Tuning Guide for RSA NetWitness Platform.

Note: For information about STIX and creating a STIX custom feed, see "Create a STIX Custom Feed" in the Decoder and Log Decoder Configuration Guide.

The feed data file and optionally the feed definition file (.xml) must be available on the local file system for an on-demand custom feed. For a recurring custom feed, the files must be available at a URL that is accessible to the NetWitness server.

Note: When you create a source and destination-based feed on a Log Decoder, it only populates the source meta key. You cannot use a range-based or CIDR feed. You must list every single IP address. To resolve this issue, create two different feeds using IP addresses and you can use CIDR in these feeds.

To create a custom feed:

  1. Go to Configure > Custom Feeds.

    2. The Custom Feeds view is displayed.

    netwitness_121_feedsconfigcustom_1122.png

  2. In the Feeds panel, click netwitness_add.png > Custom Feed > Next.

    The Configure a Custom Feed wizard is displayed, with the Define Feed form open.

    netwitness_cfgcusfddg.png

  3. Select the Feed Type: CSV or STIX.
  4. To define a feed based on a .csv formatted feed data file, select CSV (which is the default) in the Feed Type field.

  5. To define an on-demand feed task that executes once, select Adhoc in the Feed Task Type field and do one of the following:

    1. (Conditional) To define a feed based on a CsvFileFeed file, select the Upload as Csv File Feed checkbox, type the feed Name, select a .csv content file from the local file system, and click Next. If you do not select the checkbox, the .csv file will be a FlatFileFeed file.

      Note: When you select the Upload as Csv File Feed checkbox, the XML feed options under Advanced are unavailable.

    2. (Conditional) To define a feed based on an XML feed file, select Advanced Options.

      Note: Ensure that the Upload as Csv File Feed checkbox is deselected.

      3. The Advanced Options are displayed:

      netwitness_configurecustfeed_600x492.png

    3. Select an XML feed file from the local file system, choose the separator (default is comma), specify the comment characters used in the feed data file (default is #), and click Next.
      The Select Services form is displayed. This is an example of the form for a feed based on a feed data file with no feed definition file. If you are defining a feed based on a feed definition file, the Define Columns tab is not needed.

      netwitness_selectservices_600x496.png

  6. To define a recurring feed task that executes repeatedly at specified intervals, during a specified date range:

    1. In the Define Feed form, select Recurring in the Feed Task Type field.

      The Define Feed form includes the fields for a recurring feed.

      netwitness_recurringfeed_600x493.png

    2. In the URL field, enter the URL where the feed data file is located, for example, http://<hostname>/<feeddatafile>.csv, and click Verify. NetWitness verifies the location where the file is stored in order to enable checking for the latest file automatically before each recurrence.

    3. (Optional) If the URL has restricted access and requires authentication using your username and password, select Authenticated.
      NetWitness provides your user name and password for authentication to the URL.

    4. If you want the NetWitness server to access the Feed URL through a proxy, select Use Proxy. For more information on configuring a proxy, see "Configure Proxy for NetWitness" in the System Configuration Guide. By default, the Use Proxy checkbox is not set.

    5. To define the interval for recurrence, do one of the following:

      - Specify the number of minutes, hours, or days between recurrences of the feed.

      - Specify recurrence every week, and select the days of the week.

    6. To define the date range for the execution of the feed to recur, specify the Start Date and time and the End Date and time.

      netwitness_testrecurringfeed_600x491.png

  7. (Conditional) If you want to define a feed based on an XML feed file:

    1. Type the feed Name, select Advanced Options. The Advanced Options fields are displayed.
    2. Select an XML feed file from the local file system, choose the Separator (default is comma), specify the Comment characters used in the feed data file (default is #) and click Next. The Select Services form is displayed.

    netwitness_selectservices_600x496.png

  8. To identify services on which to deploy the feed, do one of the following:

    1. Select one or more Decoders and Log Decoders, and click Next

    2. Click the Groups tab and select a group. Click Next.

      The Define Columns form is displayed.

  9. To map columns in the Define Columns form:

    1. Define the Index type: IP, IP Range, or Non IP, and select the index column.
    2. (Conditional) If the index type is IP or IP Range and the IP address is in CIDR notation, select CIDR.

    3. Conditional) If the index type is Non IP, additional settings are displayed. Select the service type and Callback Keys, and optionally select the Truncate Domain option.

      netwitness_104ccfdefinecolumns_600x493.png

    4. Select the language key to apply to the data in each column from the drop-down list. The meta keys displayed in the drop-down list is based on the meta keys available for the service define values. You can also add other meta keys based on advanced expertise.

      netwitness_104ccfdefinecolfilled_600x493.png

    5. Click Next.

      The Review form is displayed.

      netwitness_104ccfreview_600x493.png

  10. Anytime before you click Finish, you can:

    • Click Cancel to close the wizard without saving your feed definition.
    • Click Reset to clear the data in the wizard.
    • Click Next to display the next form (if not viewing the last form).
    • Click Prev to display the previous form (if not viewing the first form)
  11. Review the feed information, and if correct, click Finish.

  12. Upon successful creation of the feed definition file, the Create Feed wizard closes, and the feed and corresponding token file are listed in the Feed grid and progress bar tracks completion. You can expand or collapse the entry to see how many services are included, and which services were successful.

    netwitness_121_custfdadded_1122.png