2024-08-21 09:47 PM
Greetings!
NW Platform XDR
12.3.0.0-230630083209.5.b2aa0d7
|
230630083209
|
I'm attempting to pull in blacklists from external sources to keep our alerts fresh and up to date. I'm having issues with utilizing the lists once they are brought into NW. For example I can go the route of ADMIN>CONTEXT HUB>VIEW>CONFIG>DATA SOURCES>LIST>HTTPS>ADD URL , however I can't find a way to use the data this pulls in when writing alerts, etc...
I also see a way to do this through CUSTOM FEEDS but it fails every time.
In contrast I can go into ADMIN>SERVICES>CONTEXT HUB>LISTS and create a list manually, but it also has to be updated manually. Those lists I can use in creating reports via the UI , and it appears as "ip.src=$[strong] " in the WHERE section of the report ( STRONG is the name of the list I created )
My overall goal is to make a list that is automatically updated by HTTP/S , that can be used in reports, rules, and queries. I've spent hours with the docs but there is no clear answer.
Thanks in advance
2024-08-22 05:29 PM
The old school way to do what you are looking for is the Custom Feeds. Is it possible to provide a very small csv sample of the data you would like to use for your blacklist? The format of the csv is as important as the data in it. This would help to provide a basis for discussion to make the steps more meaningful for your situation.
2024-08-22 09:53 PM
This is not the problem I am facing. I am able to correctly format and get the data in - it's writing a query or an alert that uses the data. Reports are one thing, but I want to use the blacklist data in an alert.
2024-08-23 10:45 AM
Ok since you are getting the data in to NetWitness and it is generating meta data here is an example from a made up feed that may be able to help illustrate what you would need to do when creating an alert, say in the Reporting engine or running a manual query for Investigation.
The CSV below will consist of an index column 1 that contains IP addresses. The second column 2 will contain a generic blacklist name that can be referenced in queries. The meta key that the column 2 data will be placed in based on how the feed was constructed will be a custom meta key called blacklistAlerts.
Blacklist CSV
Column 1 Column 2
20.14.54.189/32 HackerHaven
125.24.0.0/16 BotNetAlpha
15.234.12.0/18 TerribleTiger
20.14.55.122/32 HackerHaven
When this feed is used and ip.src/ip.dst sees any ip addresses from the above list it will place the column 2 information on that session into the custom meta key blacklistAlerts that was created specifically for the output from this feed. You can use any meta key but I wanted to use a custom one to make it easier for the example.
Now when you want to alert on anything from that blacklist you would create a query like the following in the Reporting Engine and attach it to an alert.
blacklistAlerts = 'HackerHaven'
This query will catch any session that has this tag attached and produce the alert. You can even make it more simple and catch anything that is on the blacklist:
blacklistAlerts exists
If you are going to use the exists query then best practice dictates that a new meta key used only for the feed is created so you do not get false positives on the alert. If you are pulling a raw csv directly from a third party and the CSV does not contain a column that you can use in a meta key then you will have to take the raw CSV file and add a column that you can reference as meta data. Otherwise the process of accessing the data coming from the feed won't work as expected.
I hope this helps to answer your question.
2024-08-27 02:01 AM - edited 2024-08-27 03:04 AM
First off, thanks for this great info! It's heading me on the right direction.
the problem is this is a fully manual process that someone has to do - make the csv, upload the csv, etc..etc.. what I'm trying to do is to have the data brought in automatically using the methods available , and then use that data in rules / alerts / reports which seems impossible to do.
For example if I want to use the abuse.ch lists, or the blocklist.de lists and have them reload based on the TTL settings.
when I configure these , they only show up as lists in the report builder - and sometimes not even there. I have tried with varying success - sometimes it appears , sometimes only in report making, and not in queries / alerts. Sometimes not at all.
Is there any hope for a reasonable and accurate guide to do this? The end goal is to use a URL to on a scheduled basis to retrieve a blocklist and use it in rules, alerts, and reports.
I've had limited success with the blocklist.de but only in rules. There seems to be multiple ways to do this via configure or admin menus but none that in the end work as intended.
I've tried both lists and feeds , with little luck.
Thanks!
2024-08-27 05:41 PM
Chris,
What is the source where you are looking to pull your blacklist from? Do you already get it in a CSV format? Or is it more like a screen scrap or in some other format?
When creating a recurring custom feed the source must already be in a csv format and have consistent columns. Then you can create the feed, setup where the language key meta will appear, and it will continue to run regularly updating the CSV without further intervention.
Check out the documentation here: https://community.netwitness.com/t5/netwitness-platform-online/create-a-custom-feed/ta-p/669251
Step 6 starts to talk about the setup of the recurring feed.
I'd be more than happy to walk through the steps but I'll need to know exactly the URL that the CSV file would be coming from. Then I can fill in the blanks that you may be running into.
2024-09-05 01:12 AM
Thank you! Let's try this one : https://sslbl.abuse.ch/blacklist/#botnet-c2-ips-csv
2024-09-18 05:08 AM
Bump?
2024-09-18 03:18 PM
Hello @ChrisIchelson sorry about the delay. I was out of the office all last week. I'll take a look at the URL provided and see what I can provide for an answer by Friday.
2024-09-18 07:01 PM
Ok I've looked over the SSLBL list. Since you are looking to pull this feed directly from SSLBL there is going to be a slight issue. The issue is that you need a column that contains what will be placed into the meta language key to reference when the feed gets a match. What this means is that you will probably need to write a small script that reaches out to the SSLBL csv, pulls it down, and then adds a new entry to the end of all the lines that are not comments. Here is an example of what I mean.
# Firstseen,DstIP,DstPort
2024-09-16 12:04:19,185.146.88.217,1024, SSLBL Blacklist
2024-09-13 19:50:40,147.45.44.107,443, SSLBL Blacklist
2024-09-13 11:37:46,63.141.252.2,3736, SSLBL Blacklist
2024-09-06 14:47:02,104.21.59.197,443, SSLBL Blacklist
2024-09-06 14:47:02,104.21.57.118,443, SSLBL Blacklist
As you can see I have added SSLBL Blacklist to the end of every line. This is what you will key off of when you do an Investigation, report, or run an alert rule. Without this in the CSV there would be no easy way to find the results of this feed. I guess you could use the port column but that could be extremely cryptic and you wouldn't know what actually caused the session to be tagged. Using a name like this allows for easier identification and filtering. If you want to pursue this method there are a few things you need to keep in mind:
1. There cannot be any blank spaces at the end of the line as this will cause the feed not to compile correctly.
2. There cannot be any blank lines in the CSV file, either at the beginning, between lines, or at the end. Otherwise the feed will not compile correctly.
3. Any line that starts with a # will be considered a comment and skipped. Comments can occur anywhere in the file.
So as long as you are able to pull the raw CSV file down and do the above manipulations and then have that CSV file somewhere that is accessible to the Admin server you can use the Custom Feed documentation provided earlier for creating the recurring feed. Here is a recap of those steps.
1. Feed type is CSV
2. Feed Task Type is Recurring
3. Name the feed something like SSLBL_Blacklist. Anything will do as this is just the label for the feed in the Feeds area and if you need to locate it on a decoder.
4. Set the URL to a web location that contains the modified CSV that the admin server can get to. Ex. https://internal.server/blacklist/modified.sslipblacklist.csv
5. Click Verify to make sure that the Admin server can reach the CSV file. If it can't then you'll need to determine why. If there is a HTTP proxy server in the way you'll need to check the Use Proxy checkbox and fill in the appropriate proxy information that the Admin server will need to use to reach the CSV.
6. Set how often you want the Admin server to refresh the feed under Recur Every
7. Click Next
8. Select all the decoders you want to have the feed or select the group that contains the decoders that will receive the feed.
9. Click Next
10. Now you have to define the columns that will make up the configuration of the feed. When this is combined with the CSV it tells the decoders how to use and read the contents of the CSV.
11. Type should be IP since the IP addresses are single IPs. If the addresses were in CIDR notation then you would set it to IP Range. Remember you cannot combine CIDR notation and non CIDR notation ip addresses in a single feed. So if you were using a primarily CIDR notated CSV you would have to set all single IP addresses to /32 to match the notation and allow it to compile correctly.
12. As you are using IP as the type you would not need to worry about the callback key. This is used only for Non IP feeds and represents the meta type of the index. By default when IP and IP range is selected it uses ip.src and ip.dst as the internal callback keys.
13. Index column should be set to the column that contains the IP address. In this case column 2. This will gray out that column header in the Define Values list below and there will be no drop down in the Key row.
14. Under the Define Values you should see a sample list from your CSV file. The column that has the added SSLBL Blacklist label that was added will need to have its Key set to the Meta Language Key you want the SSLBL Blacklist label to appear in. As an example you could set this to the alert Meta Language Key and when you investigated on Alert you would see SSLBL Blacklist tagged on any sessions where the ip.src or ip.dst matched the IP address in your CSV file. You can also use custom meta language keys here. Just make sure that any custom key you want to use is already fully setup in the system so it will show up in the drop down key list for the column you are using.
15. Click Next. This will take you to the Review area. Here you can confirm that everything looks the way you want and it is ready to compile and push to the decoders. If there is anything that doesn't look right you can use the back button to make changes. Once you hit finish it will compile and push the first run of the feed and then the Recur Every will take over from there.
You can confirm if the feed worked correctly by going to one of your decoder's explore page. Go under the decoder -> parser -> feeds folder if I remember correctly. You should see the feed name in there. There should be a stats node or something similar where you can see information about the feed. There should be a Values entry. The number in that entry should match up with the number of uncommented lines in your CSV file. This allows you to make sure that it compiled correctly. If there were any issues these numbers should not match up. If they don't, you'll need to review the CSV file to see if one of the situation I mentioned at the start happened concerning spaces.
You should be able to use this method for any custom feed you want, recurring or otherwise. The key is making sure you have a column that you can use as the entry for the meta language key that makes sense when it comes time to do your investigations or alerting.
I know there is a lot here but I hope this helps to clear up any uncertainty you may have had with the process.