Custom Feed Definition File Structure

The NetWitness Custom Feed wizard allows creation and deployment of custom Decoder feeds based on deterministic logic that offers the meta keys specific to the selected Decoders and Log Decoders. Although the wizard guides users through the process to create both on-demand and recurring feeds, it is helpful to understand the form and content of a feed file when you create a feed.

Before you create or modify custom feeds, keys, or data types, ensure that you have a good understanding of the relationship between keys and datatypes in databases. It is important that any custom keys that you create match the corresponding datatypes. If your feed utilizes a custom meta key, you must define that key’s data type in index-decoder-custom.xml. This is to ensure that the key data is generated with correct data format.

Also, to enable investigation on the custom meta key, you must define its index on the concentrator, using the index-concentrator-custom.xml file. For more information on how to write the index-decoder-custom.xml or index-concentrator-custom.xml file, see the topic "Index Customization" in the Core Database Tuning Guide for RSA NetWitness Platform.

Feed filenames in RSA NetWitness are in the form <filename>.feed. To create a feed, NetWitness requires a feed data file in .csv or .xml format and a feed definition file in .xml format, which describes the structure of a feed data file. The Custom Feed wizard can create the feed definition file based on a feed data file, or based on a feed data file and the corresponding feed definition file.

The files that you use to create an on-demand feed must be stored on your local file system. The files used to create a recurring feed must be stored at an accessible URL, whence NetWitness can fetch the most current version of the file for each recurrence. After a NetWitness feed is created, you can download the feed to your local file system, edit the feed files, and then edit the NetWitness feed to use the updated feed files.

Sample Feed Definition File

This is an example of a feed definition file named dynamic_dns.xml, which NetWitness creates based on your entries in the Custom Feed wizard. It defines the structure of the feed data file named dynamic_dns.csv.

Note: The feed file path should be .csv regardless of the Feed Type (Default or STIX).

<?xml version="1.0" encoding="utf-8"?>
<FDF xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="feed-definitions.xsd">

<FlatFileFeed name="Dynamic DNS Domain Feed"
path="dynamic_dns.csv"
separator=","
comment="#"
version="1">

<MetaCallback
name="alias.host"
valuetype="Text"
apptype="0"
truncdomain="true"/>

<LanguageKeys>
<LanguageKey name="threat.source" valuetype="Text" />
<LanguageKey name="threat.category" valuetype="Text" />
<LanguageKey name="threat.desc" valuetype="Text" />
</LanguageKeys>

<Fields>
<Field index="1" type="index" key="alias.host" />
<Field index="4" type="value" key="threat.desc" />
<Field index="2" type="value" key="threat.source" />
<Field index="3" type="value" key="threat.category" />
</Fields>
</FlatFileFeed>

</FDF>

Define Multiple Values in a Single Field

The information in this section applies to NetWitness version 11.3.1 and later.

Custom feeds support multiple values in a single field. This allows feeds to generate multiple values of the same meta.

For example:

<Field index="1" type="index" key="lc.cid" />

<Field index="2" type="value" key="user" separator=";" openchar="(" closechar=")"/>

</Fields>

The second field is defined to accept multiple values in the corresponding CSV file. The following is an example of the defined values:

clc1,(bob;tom;sam) clc12,susan clc123,doris

This definition means that when we see the lc.id with a value of cic1, three username meta values are generated with the values of bob, tom and sam.

The separator in the field definition for a multi-value field must be different than the separator for the feed file itself. Escaping the separator character in field values is not supported.

Feed Definition Equivalents for Custom Feed Wizard Parameters

The NetWitness Custom Feed wizard provides options to define the structure of the data feed file. These correspond directly to attributes in the feed definition (.xml) file.

NetWitness Parameter	Feed Definition File Equivalent
(Define Feed Tab) Feed Type	Select: Default - to define a feed based on a .csv formatted feed data file. STIX - to define a feed based on STIX formatted .xml file.
(Define Feed Tab) Feed Task Type	Select: Adhoc - to create an on-demand feed. Recurring - to update the .csv or .xml file persistently and store it in a location accessible by NetWitness, so NetWitness downloads a file at regular intervals and pushes it to the downstream devices.
(Define Feed tab) Name	The custom feed name in the feed data file. It corresponds to the flatfeedfile name attribute in the feed definition file. For example, Dynamic DNS Test Feed. Note: You can use special characters to define the name of the custom feed.
(Define Feed tab) File/ Browse	This is the name of the feed data file. It corresponds to the flatfeedfile path attribute in the feed definition file. For example, dynamic_dns.csv.
(Advanced Options tab) XML Feed File	The name of the feed definition file. For example, dynamic_dns.xml.
(Advanced Options tab) Separator	The separator character used to separate attributes in the feed data file. It corresponds to the flatfeedfile separator in the feed definition file. For example, a comma.
(Advanced Options tab) Comment	The character used to identify a comment in the feed data file. It corresponds to the flatfeedfile comment attribute in the feed definition file. For example, #.
(Define Columns tab, Define Index) Type	The type of lookup value in the index position of the feed data file. IP means that each row in the feed data file contains an IP address in the lookup value position. The IP value is in dotted-decimal format (for example, 10.5.187.42). IP Range means that each row in the feed data file contains a range of IP addresses in the lookup value position. The IP range is in CIDR format (for example, 192.168.2.0/24). Non IP means that the each row in the feed data file contains a metadata value other than IP address in the lookup value position. The Service Type and Truncate Domain, and Callback Keys fields become active for a Non IP index.
(Define Columns tab, Define Index) CIDR	Specifies that the IP value in the lookup position is in CIDR format. The CIDR attribute sets the IP address format in the field to Classless Inter-Domain Routing (CIDR) notation.
(Define Columns tab, Define Index) Service Type	For a Non IP index, the integer service type to filter meta lookups. It corresponds to the MetaCallback apptype attribute in the feed definition file. A value of 0 indicates no filtering by service type.
(Define Columns tab, Define Index) Truncate Domain	For a Non IP index, for meta values that contain domain names (for example, hostnames), the system can strip off the host specific element in the data. Truncate Domain corresponds to the MetaCallback truncdomain attribute. If the value is www.example.com, it is truncated to example.com. A value of False selects no truncation, and True selects truncation.
(Define Columns tab, Define Index) Callback Keys	For a Non IP index, the available meta keys to match on instead of ip.src/ip.dst (the defaults for IP index type) are selectable from the drop-down list. The Callback Key corresponds to the MetaCallback name attribute, and the index column of the csv file must contain data that can match the chosen meta key. For example, if the user meta key is chosen, the index column of the .csv file needs to be populated with users to be matched.
(Define Columns tab, Define Index) Index Column	Identifies the column in the feed data file that provides the lookup value for the row. Each position in each row of the feed data file is identified by a Field index attribute in the feed definition file. A field with an index of 1 is the first entry in a row, the second field has an index of 2, the third field has an index of 3, and so on.
(DEFINE VALUES) Key	The name of the LanguageKey, as defined in the feed definition file, for which meta is created from this row of the feed data file. It corresponds to the Field key attribute in the feed definition file. A key applies only to a field whose type is set to value. In the feed definition file, there is a list of LanguageKeys from index.xml, or a summary name if Source Name and Destination Name are used. For example, reputation is a summary name for reputation.src and reputation.dst. This value is referenced by the Field key attribute.

Sample Files for a MetaCallback Feed Using CIDR Index Range for IPv4 and IPv6

These sample files demonstrate how to use CIDR index ranges for IPv4 and IPv6 in custom MetaCallback feeds. As with other custom feeds, you must create feed data file in .csv format, and a feed definition file in .xml format.

Note: Using MetaCallback feeds with CIDR index ranges is supported only through the Advanced Configuration wizard or the REST interface.

The following example shows the contents of both a .csv file and an .xml file for a MetaCallback feed using CIDR index ranges for IPv4.

.csv file:

192.168.0.0/24, Sydney
192.168.1.0/24, Melbourne

.xml file:

<?xml version="1.0" encoding="UTF-8"?>

</MetaCallback>

</LanguageKeys>

</Fields>

</FlatFileFeed>

</FDF>

The following example shows the contents of both a .csv file and an .xml file for a MetaCallback feed using CIDR index ranges for IPv6.

.csv file:

192.168.0.0/24, Sydney
192.168.1.0/24, Melbourne

.xml file:

<?xml version="1.0" encoding="UTF-8"?>

</MetaCallback>

</LanguageKeys>

</Fields>

</FlatFileFeed>

</FDF>

Note: To configure a CIDR index range for feeds with single or multiple MetaCallbacks of value type IPv4 or IPv6, the field of type index MUST contain a range attribute with range="cidr". Also, configuring "cidr" index ranges for feeds with MetaCallbacks of multiple different value types is not supported.

NetWitness Community

Table of Contents

Product Resources

Custom Feed Definition File Structure

Sample Feed Definition File

Define Multiple Values in a Single Field

Feed Definition Equivalents for Custom Feed Wizard Parameters

Sample Files for a MetaCallback Feed Using CIDR Index Range for IPv4 and IPv6

The following example shows the contents of both a .csv file and an .xml file for a MetaCallback feed using CIDR index ranges for IPv4.

The following example shows the contents of both a .csv file and an .xml file for a MetaCallback feed using CIDR index ranges for IPv6.

On this page