How to use external lists in making rules/queries/alerts when they are auto-updated by HTTP/S

[ChrisIchelson]

Greetings! 

NW Platform XDR

Current Version

12.3.0.0-230630083209.5.b2aa0d7

Current Build

230630083209

I'm attempting to pull in blacklists from external sources to keep our alerts fresh and up to date. I'm having issues with utilizing the lists once they are brought into NW. For example I can go the route of ADMIN>CONTEXT HUB>VIEW>CONFIG>DATA SOURCES>LIST>HTTPS>ADD URL , however I can't find a way to use the data this pulls in when writing alerts, etc...

I also see a way to do this through CUSTOM FEEDS but it fails every time.

In contrast I can go into ADMIN>SERVICES>CONTEXT HUB>LISTS and create a list manually, but it also has to be updated manually. Those lists I can use in creating reports via the UI , and it appears as "ip.src=$[strong] " in the WHERE section of the report ( STRONG is the name of the list I created ) 

My overall goal is to make a list that is automatically updated by HTTP/S , that can be used in reports, rules, and queries. I've spent hours with the docs but there is no clear answer. 

Thanks in advance

[JohnKisner]

@ChrisIchelson 

The old school way to do what you are looking for is the Custom Feeds. Is it possible to provide a very small csv sample of the data you would like to use for your blacklist? The format of the csv is as important as the data in it. This would help to provide a basis for discussion to make the steps more meaningful for your situation.

[ChrisIchelson]

This is not the problem I am facing. I am able to correctly format and get the data in - it's writing a query or an alert that uses the data. Reports are one thing, but I want to use the blacklist data in an alert. 

[JohnKisner]

@ChrisIchelson 

Ok since you are getting the data in to NetWitness and it is generating meta data here is an example from a made up feed that may be able to help illustrate what you would need to do when creating an alert, say in the Reporting engine or running a manual query for Investigation.

The CSV below will consist of an index column 1 that contains IP addresses. The second column 2 will contain a generic blacklist name that can be referenced in queries. The meta key that the column 2 data will be placed in based on how the feed was constructed will be a custom meta key called blacklistAlerts.

Blacklist CSV

Column 1                     Column 2 

 20.14.54.189/32         HackerHaven

125.24.0.0/16              BotNetAlpha

15.234.12.0/18            TerribleTiger

20.14.55.122/32          HackerHaven

When this feed is used and ip.src/ip.dst sees any ip addresses from the above list it will place the column 2 information on that session into the custom meta key blacklistAlerts that was created specifically for the output from this feed. You can use any meta key but I wanted to use a custom one to make it easier for the example.

Now when you want to alert on anything from that blacklist you would create a query like the following in the Reporting Engine and attach it to an alert.

blacklistAlerts = 'HackerHaven'

This query will catch any session that has this tag attached and produce the alert. You can even make it more simple and catch anything that is on the blacklist:

blacklistAlerts exists

If you are going to use the exists query then best practice dictates that a new meta key used only for the feed is created so you do not get false positives on the alert. If you are pulling a raw csv directly from a third party and the CSV does not contain a column that you can use in a meta key then you will have to take the raw CSV file and add a column that you can reference as meta data. Otherwise the process of accessing the data coming from the feed won't work as expected.

I hope this helps to answer your question.

[ChrisIchelson]

First off, thanks for this great info! It's heading me on the right direction. 

the problem is this is a fully manual process that someone has to do - make the csv, upload the csv, etc..etc.. what I'm trying to do is to have the data brought in automatically using the methods available , and then use that data in rules / alerts / reports which seems impossible to do. 

For example if I want to use the abuse.ch lists, or the blocklist.de lists and have them reload based on the TTL settings. 

when I configure these , they only show up as lists in the report builder - and sometimes not even there. I have tried with varying success - sometimes it appears , sometimes only in report making, and not in queries / alerts. Sometimes not at all. 

Is there any hope for a reasonable and accurate guide to do this? The end goal is to use a URL to on a scheduled basis to retrieve a blocklist and use it in rules, alerts, and reports. 
I've had limited success with the blocklist.de but only in rules. There seems to be multiple ways to do this via configure or admin menus but none that in the end work as intended.  
I've tried both lists and feeds , with little luck. 

Thanks! 

[JohnKisner]

Chris,

What is the source where you are looking to pull your blacklist from? Do you already get it in a CSV format? Or is it more like a screen scrap or in some other format? 

When creating a recurring custom feed the source must already be in a csv format and have consistent columns. Then you can create the feed, setup where the language key meta will appear, and it will continue to run regularly updating the CSV without further intervention. 

Check out the documentation here: https://community.netwitness.com/t5/netwitness-platform-online/create-a-custom-feed/ta-p/669251

Step 6 starts to talk about the setup of the recurring feed. 

I'd be more than happy to walk through the steps but I'll need to know exactly the URL that the CSV file would be coming from. Then I can fill in the blanks that you may be running into. 

[ChrisIchelson]

Thank you! Let's try this one : https://sslbl.abuse.ch/blacklist/#botnet-c2-ips-csv