Create a custom feed for Netwitness 10.2

[RSAAdmin]

Hi, I am trying to generate custom feeds for netwitness

Issue: I want to add a meta fields (device.domain) depending on the ip address from the log.

Each log has ip address in it. But i also want to add device name meta field

index-concentrator.xml is

<key description="Device IP" level="IndexValues" name="device.ip" format="IPv4" valueMax="100000" defaultAction="Open"/>

For new mata. I have added a filed in index-concentrator-custom.xml

<key description="domain" name="device.domain"  level="IndexValues" format="Text" valueMax="0" defaultAction="Open" />

My feed definition looks like

<?xml version="1.0" encoding="utf-8"?>
<FDF xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"  xsi:noNamespaceSchemaLocation="feed-definitions.xsd">
  <FlatFileFeed name="Domain name" path="domainmap.csv" separator="," comment="#">
           <LanguageKeys>
                   <LanguageKey name="device.domain" valuetype="Text"/>
           </LanguageKeys>
           <Fields>
                   <Field index="1" type="index" range="cidr"/>
                  <Field index="2" type="value" key="device.domain"/>
            </Fields>
   </FlatFileFeed>
</FDF>

Then create the csv file

#Example
#IP address   Domain
192.168.1.1	Coke
192.168.1.12	Pepsi

Compiled the feed from netwitness console

> feed create feed-definitions.xml
Creating feed Domain name...
done.  0 entries, 24 invalid records
All feeds complete.

I got some invalid record message. Not sure about this.

Then copied the new .feed file to desktop and added the feed from Security Analytics. Uploaded the feed

After this. Restarted both decoder and concentrator services. And many logs with device.ip meta were sent to decoder.But devoce.domain meta is empty under investigator. Since investigator in Security analytics is slow. I also tried to get the data over rest API. But no data for device.domain meta field.

Did i miss some steps to push the custom feed? Or anything wrong in feed definitions?

Regards

DJ

Message was edited by: dheerajjoshim

[RSAAdmin]

1. Your CSV uses tab as the separator character, but your feed declaration requires comma.
2. CIDR ranges are missing. Write 192.168.1.1/32 if you want a single address.
3. Bonus tipp: make sure your ranges don't overlap. Overlapping ranges are not supported by Netwitness, but nwconsole doesn't throw errors or warnings if ranges overlap.

[RSAAdmin]

CSV is not tab separated. It is comma separated as expected from NW.

It was pasted as a table in this edit window.

The log doesn't contain 192.168.1.1/32 instead it just contain IP address 192.168.1.1. So what type is appropriate for this text? Can i just mention

range=low

instead of

range= cidr

Does this work?

Regards

DJ

[RSAAdmin]

Either CIDR + one column with IP ranges. You need to add /32.

Or you use two colums, one for lower one for upper end. For example

192.168.1.1,192.168.1.1,Description

[Anonymous]

You don't need to use "range" or CIDR for a single IP address.

Here is an example XML file that uses a single IP address for an index and creates a HOSTNAME meta value:

<FDF>

<FlatFileFeed name="HostName" path="hosts.csv" separator="," comment="#">

  <LanguageKeys>

        <LanguageKey name="hostname" valuetype="Text" srcname="hostname.src" destname="hostname.dst" />

    </LanguageKeys>

    <Fields>

        <Field index="1" type="index" />

        <Field index="2" type="value" key="hostname" />

    </Fields>

</FlatFileFeed>

</FDF>

I also noticed in your index-concentrator-custom.xml you have valueMax="0"

This sets the maximum number of entries that you want to be able to index, so 0 is not a good number to use!

Service Pack 2 for SA 10.2 has just been released. There is a new feature in this service pack for creating custom feeds using a wizard based approach that makes it much easier!

[RSAAdmin]

Hi, I have made changes to CSV to have 2 IP's and made changes to field section. But still no data in meta device.domain.

New CSV is

#device.ip ,domain
192.168.1.1,192.168.1.1,Pepsi
192.168.1.2,192.168.1.2,Coke

And changed the feed-definitions.xml as following

<Fields>
      <Field index="1" type="index" range="low"/>
      <Field index="2" type="index" range="high"/>
      <Field index="3" type="value" key="device.domain"/>
</Fields>

Compiled the feeds

> feed create feed-definitions.xml
Creating feed Domain name...
done.  2 entries, 1 invalid records
All feeds complete.

Then uploaded the newly generated feed from Security analytics web console and it is uploaded successfully. But still i can't see the data for meta device.domain.

Regards

DJ

[RSAAdmin]

Hi, I have added ValueMax=10000 as you mentioned. But looks like it has nothing to do with the problem i have. Since it didn't resolve my original problem.

My Definition.xml is

<?xml version="1.0" encoding="utf-8"?>
<FDF xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="feed-definitions.xsd">
        <FlatFileFeed name="Domain name" path="domainmap.csv" separator="," comment="#">
                <MetaCallback name="Device IP" valuetype="IPv4" truncdomain="false">
                         <Meta name="device.ip"/>
                </MetaCallback>
                <LanguageKeys>
                        <LanguageKey name="dev.dom" valuetype="Text" srcname="dev.dom.src" destname="dev.dom.dst"/>
                </LanguageKeys>
                <Fields>
                        <Field index="1" type="index" />
                        <Field index="2" type="value" key="dev.dom"/>
                </Fields>
       </FlatFileFeed>
</FDF>

index-concentrator-custom.xml has these entries

<key description="domain" name="dev.dom"  level="IndexValues" format="Text" valueMax="10000" defaultAction="Open" />
<key description="domainSource" name="dev.dom.src"  level="IndexValues" format="Text" valueMax="10000" defaultAction="Open" />
<key description="domainDestination" name="dev.dom.dst"  level="IndexValues" format="Text" valueMax="10000" defaultAction="Open" />

Changed the meta names to support 16 character max length restriction.

After this i have created the feed from NwConsole

> feed create feed-definitions.xml
Creating feed Domain name...
done.  2 entries, 1 invalid records
All feeds complete.

I still get 1 invalid record. Not sure what is the issue.

Uploaded the new feed from analytic web cosole. It is uploaded successfully.

Pumped test logs but no data for mata fields dev.dom or dev.dom.src or dev.dom.dst

Regards

DJ

[RSAAdmin · Accepted solution]

Why are you using the callback "device.ip"? If I remove that part, i.e. implicitly set the callback to ip.src and ip.dst, then the feed works on my freeware version of Netwitness (9.7.5.9). The invalid record is probably due to an empty line at the end of the CSV.

<?xml version="1.0" encoding="utf-8"?>  
<FDF xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="feed-definitions.xsd">  
        <FlatFileFeed name="Domain name" path="domainmap.csv" separator="," comment="#">                  
                <LanguageKeys>  
                        <LanguageKey name="dev.dom" valuetype="Text" srcname="dev.dom.src" destname="dev.dom.dst"/>  
                </LanguageKeys>  
                <Fields>  
                        <Field index="1" type="index" />  
                        <Field index="2" type="value" key="dev.dom"/>  
                </Fields>  
       </FlatFileFeed>  
</FDF> 

At least on NW 9.8, feeds cause a lot of troubles. There are many issue that can haunt you.

[RSAAdmin]

Hi Johannes_Bader, I was able to see the logs for dev.dom.src meta.

I am using meta call back since i am not using ip.src to store the sender ip address instead i am using device.ip as the meta.

So another question on feeds

If a new entry is added to csv file then we need to create the feed again and then upload it?

Regards

DJ

[RSAAdmin]

Following steps are used to generate custom feeds through REST

Create your csv file and definition file then create a zip file. Upload it to NW using URL

http://IP:PORT/decoder/parsers/upload?name=zipfilename&dateInstalled=timeinmillis 

I made customfeed.csv and customfeed.xml (csv and definition file) and created a customfeed.zip file and put the zip file in post message body and executed above URL. It creates new customfeed.feed file in decoder

Then reload the parser it will starts capturing the data for new Meta.

Regards

DJ