Creating Domain Feeds

[RSAAdmin]

Does anyone have a good example for creating domain based netwitness feeds? We have Live Manager and many of the domains feeds enabled, but would like to deploy custom domain feeds from other internal threats or intel. Any assistance would be greatly appreciated, thanks!

[RSAAdmin]

Here's some sample XML for the feed def.

<?xml version="1.0" encoding="utf-8"?>
<FDF xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="feed-definitions.xsd" version="1">
<FlatFileFeed name="##Feed Name##" path="##CSV FILE##.csv" separator="," comment="#">
         <!-- This is the meta-key you want to match against, in this example it's alias.host and it'll be referenced in this .xml file by the var name "hostname" -->
        <MetaCallback name="hostname" valuetype="Text" apptype="0">
                <Meta name="alias.host"/>
        </MetaCallback>
         <!-- Where you want more blue text to show up, aka meta-values -->
        <LanguageKeys>
                <LanguageKey name="alert" valuetype="Text"/>
        </LanguageKeys>
         <!-- 
                    when I match something in alias.host; I want to print a value of "alert" 
                    Example CSV:
                         google.com,lol google
                         bing.com,who uses this search engine anyway
                    * Note this will only work for EXACT matches. If you want to just match domain name and ignore hostnames you have
                    change 
                    "<MetaCallback name="hostname" valuetype="Text" apptype="0">"
                    for
                    "<MetaCallback name="hostname" valuetype="Text" truncdomain="true">"
          -->
        <Fields>
                <Field index="1" type="index" key="hostname"/>
                <Field index="2" type="value" key="alert" />
        </Fields>
</FlatFileFeed>
</FDF>

[Anonymous]

What other steps are required to get such a feed to work? I created a similar feed and the associated .csv file with a couple of test domains. I then created the feed using the NWConsole "Feed create"command and then uploaded the feed to my decoders. When I visit the sites in the .csv file, I get no alerts that I can see in Investigator.

What step am I missing?

[RSAAdmin]

You need the feed definition and the csv in the same directory when you create the feed.  From the definition file above, no special index keys are required.  feel free to post your def file and csv here and I can take a look.

[Anonymous]

We have a feeds solution to help this issue.  I will put some example feeds below. 

Technical Note:  Below are the supported range types for the index field of the feed definition xml file:

Type 1:  None - Do not specify a range parameter if your csv file contains only individual IP addresses in its index field.
Example XML:
                    <Field index="1" type="index" />
Example CSV:
                    10.0.2.5,MainOffice,Servers

Type 2:  range="cidr" - Use this syntax to denote CIDR notation in your csv file.  If using this, individual IP addresses in your csv file must specify /32.
Example XML:
                    <Field index="1" type="index" range="cidr" />
Example CSV:
                    10.0.0.0/24,MainOffice,Servers
                    10.0.1.0/25,MainOffice,PCs
                    10.0.2.5/32,MainOffice,Servers

Type 3.  range="low" and range="high - Use this syntax when using two index fields to denote a range of IP addresses.
Example XML:
                    <Field index="1" type="index" range="low"/>
                    <Field index="2" type="index" range="high"/>
Example CSV:
                    10.0.0.0,10.0.0.255,MainOffice,Servers
                    10.0.1.0,10.0.1.255,MainOffice,PCs

Note: when IP addresses are expected in CSV file, only individual IP addresses or CIDR notations are allowed, no other values are allowed.

Kind Regards,

Sanjib Sarkar

Netwitness Support

[Anonymous · Accepted solution]

Sorry about the delay in replying been enjoying a holiday.

my xml:

<?xml version="1.0" encoding="utf-8"?>

<FDF xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="feed-definitions.xsd" version="1">

          <FlatFileFeed name="WatchlistHostNames" path="watchlisthostnames.csv" seperator="," comment="#">

          <MetaCallback name="watchlisthostname" valuetype="Text" truncdomain="true">

                <Meta name="alias.host"/>

          </MetaCallback>

          <LanguageKeys>

                <LanguageKey name="alert" valuetype="Text"/>

        </LanguageKeys>

          <Fields>

                     <Field index="1" type="index" key="watchlisthostname"/>

                      <Field index="2" type="value" key="alert"/>

          </Fields>

          </FlatFileFeed>

</FDF>

my csv:

google.com, watchlisted domain name

bing.com, watchlisted domain name

--EDITED TO ADD--

Dang, think I found a typo in the xml.......

[RSAAdmin]

Yep, typos kill.  Did correcting your typo fix it for you?  The local Investigator logs will sometimes give clues if the FDF is problematic.

[Anonymous]

Fed the corrected xml and csv into liveManager .all folder. It consumed the files but I see no record of a push to the decoders in the lM log files.

[RSAAdmin]

You might need a manual push, or alternatively, you can upload the .feed file to the feeds folder on the decoders.  Do it via Administrator or SSH it on them.

[Anonymous]

Found the .xml .feed and .csv files in the __cftemp folders for the decoders.

Not seeing any alerts.

-----And I found another typo---- facepalm

Still not seeing any alerts, though the feed has been consumed in
LiveManager.