Every SOC analyst should spend at least part of his/her day reading various blog posts and white papers on attacker profiles and their tools and techniques. Attackers often repeat at least certain aspects of their activity on various targets, and thus provide the analysts with an opportunity to incorporate such indicators into their toolset (hopefully) prior to being targeted by such attackers.
In addition, other sites provide continuous indicators of both advanced and opportunistic attackers, which can also be incorporated into the toolset for automatic detection.
Here I will provide a guide on how to format such publicly available indicators into the NetWitness Network and NetWitness Endpoint.
Let us briefly describe what is an Indicator of Compromise (IOC). An IOC is an indicator of something that has already been observed on a compromised system or a behavior that was part of an attack. There are multiple types of IOCs, because you can track something in many different ways, for example IP addresses, filenames, file size, URLs, a particular endpoint behavior, etc.
Sometimes lists of hashes such as MD5/SHA1/SHA256 are enough to quickly identify compromised machines. For this purpose, there are multiple sites where you can find a good list of MD5 / SHA1 / SHA256 based IOCs, here are some examples:
- https://github.com/RedDrip7/APT_Digital_Weapon
- http://atm.cybercrime-tracker.net/index.php
- https://www.virustotal.com/gui/home/search
- https://github.com/eset/malware-ioc
- https://www.team-cymru.com/mhr.html
- https://www.vicheck.ca/hashquery.php
- https://virusshare.com/hashes.4n6
- https://research.metaflows.com/stats/content_md5_hash/
- https://urlhaus.abuse.ch/browse/
At this point, if you don't have your own list of IOCs based on MD5 / SHA1 / SHA256, you can use some of these lists, created by other analysts. However, such information is not necessarily in a suitable format for incorporating into the NetWitness toolset. One way to normalize the data is by following this process:
- Install Cmder (https://cmder.net/), which is a good console emulator for Windows that has the functionality needed for the rest of the steps.
- Let’s say you want to generate the MD5 list of IOC FIN7 found at:
- https://github.com/RedDrip7/APT_Digital_Weapon/tree/master/FIN7
After you download file FIN7_hash.md, you are ready to start.
- Open Cmder and go to the folder where the downloaded file is located.
- Now run following commands, as shown in the following figure.
grep -e "[0-9a-f]\{32\}" FIN7_hash.md | cut -c 3- | cut -c -32 | uniq -u > FIN7_tmp.txt | sed -e 's/$/,FIN7,blacklisted\ file/' FIN7_tmp.txt > FIN7_md5.txt
Let me explain the commands in more detail for those not familiar with these tools/commands:
| Command | Details |
|---|---|
| grep -e "[0-9a-f]\{32\}" FIN7_hash.md | Extract MD5 from file FIN7_hash.md |
| cut -c 3- | cut -c -32 | Remove all the unneeded characters |
| uniq -u > FIN7_tmp.txt | Make it unique and save the output to FIN7_tmp.txt |
| sed -e 's/$/,FIN7,blacklisted\ file/' FIN7_tmp.txt > FIN7_md5.txt | Create the final file |
The above steps are specific to this particular file, each set of IOCs will need its unique set of conversion steps, add “,FIN7,blacklisted file” to each line and write the output to FIN7_md5.txt. Where “FIN7” is the description of your APT, which we will map to the ioc key in NetWitness, and the value “blacklisted file” which we will map to the analysis.file key, this step is critical if you want the module and machine scores automatically set to 100 for these matches.
If you want to use your own toolset to format the data, then please ensure you follow these steps in order to generate a good list of IOC:
- Retrieve the file (can be a plain text, a pdf, a word, one HTML, the filetype is not important)
- Extract the IOCs from file
- Remove unneeded chars (in order to have only the useful strings)
- Make any IOC unique (ensure you remove any duplicate entries as this in an important step)
- You must have a value of “blacklisted file” in your resulting file if you want machine and module scores to be affected by your feed.
At this point we have the source CSV file with the data necessary to create a Feed for NetWitness Endpoint.
To create your feed, follow these steps:
- Go to Configure->Custom Feed and create a custom feed.
- Click on + icon, select Custom Feed and configure the custom feed by giving it a name and selecting the CSV file you created above as shown in the following figure
In this case CustomAPTFeed.csv is your FIN7_md5.txt created above, which we renamed to CustomAPTFeed.csv.
Apply the feed to the LogDecoder (second Tab), and define the Columns as shown in the following figure. Here define the Callback Key as “checksum.src”, select Index Column to be the first one, which will grey it out in the grid below, select the key for Column 2 in this case “ioc” and finally select Column 3 as the “analisys.file” meta key, again this step is critical if you want the risk scores to automatically update, it will only work for this combination of key and value.
Finish the import make sure there are no errors and the task completed successfully. Now you can go to Investigate in the UI and validate your data.
Every time the meta key “checksum.src” contains a value defined in your custom feed, meta key “ioc” will be populated with the value provided in the Column 2 of the CSV file and the “analysis.file” meta key will have the “blacklisted file” value, as shown in the following figure.
In this case, the endpoint risk score for that system will automatically be increased to 100 (the highest possible risk score), and under Critical Alert you will see the relevant indicator in our case that is “Blacklisted File”.
The same will happen to the specific module that was Blacklisted, as shown in the following figure:
Multiple types of IOCs can be loaded into NetWitness Endpoint, following the steps presented in this blog post. Always remember that IOCs are static, so the resource has to match exactly to trigger an alert. In the case of MD5 hashes of files, also remember that if the file is changed even by just one byte or for example recompiled, the MD5 hash will be different and your IOC will no longer match. This is the reason why we recommend that analysts focus instead on other possible characteristics of a file (such as the file description if it is unique) or its behavior (such as any parameters that need to be passed for it to work).
I hope this blog post can help in importing simple and fast IOCs into the NetWitness endpoint for automatic detection of known malicious files.
A special thank you goes out to Lee Kirkpatrick for his assistance and support.
