Applying custom feeds to already parsed decoder data

[Anonymous]

I am wondering if anyone has any real world examples where they have the ability to be able to apply a single feed to already parsed data on a decoder.

The use case is that we have custom feeds that are produced and want to have them applied to all data on the decoder, not just data that has not been parsed. I understand that the only way this can be accomplished is to extract the already parsed data from the decoder to a pcap and then have it replayed back through a decoder.  The main issue with this approach is that all the data being replayed would be parsed as opposed to just the single feed.

[RSAAdmin]

I assume what you are talking about is retroactively applying a newly added/updated feed to historical pcaps/metadata.

There is no existing functionality in NW that I know of to do that. Basically yes, you would need to do full packet replay against a whole new collection to achieve that.

However, depending on your goals/needs for doing a retroactive check there are several other possible solutions you could try that I'll suggest:

1. For updates to a custom feed, programatically generate a list of new elements added to the feed and then query all data for those new elements as a one-time "catch-up" operation. The query could be done automatically using NW API calls.
2. If there is a specific area of concern, such as a specific subnet, service or user, you can extract pcaps for just that period and then create a local collection in Investigator and import those packets. This would require adding the custom feed to your local Investigator.

[Anonymous]

Option 1 was something I was hoping may be feasible as it is a single custom feed that only needs to be applied and only the new elements from that.

I will look into developing some proof of concept code and post back what I find.

If anyone has anything that they have put together to do something similar that could point this in the right direction it would be much appreciated.

[RSAAdmin]

Jedi Mercer's solution is excellent. I'm doing exactly that. First off, I highly recommend that you put all your feeds under version control if you haven't already. Among many other benefits this allows you to do diffs very easily,

def _diff(feed_folder, csv_file):
    csv_path = os.path.join(feed_folder,csv_file)
    cmd = 'git.exe diff --ignore-space-at-eol master ' + csv_path
    p = subprocess.Popen(cmd , shell=True, stdout=subprocess.PIPE,
        cwd=feed_folder, stderr=subprocess.PIPE)
    (std, err) = p.communicate()  
    changes = {}
    for line in std.split('\n'):
        for mode in ('+', '-'):
            if line.startswith(mode) and not line.startswith(3*mode):               
                changes.setdefault(mode,[]).append( line[1:] )
    return changes

Then we build API calls to see if there any matches in the past week and how many.

[RSAAdmin]

Yes! Put custom feeds (and other customizations you do) for Netwitness under a version control system like SVN or git or whatever you prefer. Especially good if you have more than one person making updates or an automated tool adding to feeds.

[RSAAdmin]

I uploaded my URL checking code - that may help you with your implementation:

[DEAD LINK /docs/DOC-25152]https://community.emc.com/docs/DOC-25152

The following GIT command is quite to get the list of recent additions to the feeds:

git.exe whatchanged --since="1 day ago" -p -- urlquery.csv