RSA NetWitness has a number of integrations with threat intel data providers but two that I have come across recently were not listed (MISP and Minemeld) so I figured that it would be a good challenge to see if they could be made to provide data in a way that NetWitness understood.
Current RSA Ready Integrations
MISP
Install the MISP server in a few different ways
VMWare image, Docker image or on an OS are all available (VMware image worked the best for me)
https://www.circl.lu/misp-images/latest/
Authenticate and setup the initial data feeds into the platform
Set the schedule to get them polling for new data
Once created and feeds are being pulled in you can look at the attributes to make sure you have the data you expect
Test the API calls using PyMISP via Jupyter Notebook
https://github.com/epartington/rsa_nw_misp/blob/master/get-misp.ipynb
- you can edit the notebook code to change the interval of data to pull back (last 30 days, all data or such to limit impact on the MISP server)
- You can change the indicator type (ip-dst, domain etc.) to pull back the relevant columns of data
- You can change the column data to make sure you have what you need as other feed data
Once that checks out and you have the output data you want via the notebook you can add the python script to the head server of NetWitness
Install PyMISP on the head server of the NetWitness system so that you can crontab the query.
- Install PyMISP using PIP
(keep in mind that updating the code on the head server could break things so be careful and test early and often before committing this change in production)
yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
yum install python-pip
OWB_FORCE_FIPS_MODE_OFF=1 python
OWB_FORCE_FIPS_MODE_OFF=1 pip install pymisp
OWB_FORCE_FIPS_MODE_OFF=1 pip install --upgrade pip
OWB_FORCE_FIPS_MODE_OFF=1 ./get-misp.py
yum repolist
vi /etc/yum.repos.d/epel.repo
change enabled from 1 to 0Make sure you disable the epel repo after installing so that you don't create package update issues later
Now setup the query that is needed in a script (export the Jupyter notebook as python script)
https://github.com/epartington/rsa_nw_misp/blob/master/get-misp.py
Crontab the query to schedule it (the OWB is required to work around FIPS restrictions that seem to break a number of script related items in python)
23 3 * * * OWB_FORCE_FIPS_MODE_OFF=1 /root/rsa-misp/get-misp.py > /var/lib/netwitness/common/repo/misp-ip-dst.csv
Now setup the NetWitness recurring feed to pull from the local feed location
map the ip-dst values (for this script) to the 3rd column and the other columns as required
Minemeld
Minemeld is another free intel aggregation tool from Palo Alto Networks and can be installed many ways (i tried a number of installs on different Ubuntu OSes and had difficulties), the one that worked the best for me was via a docker image.
https://www.paloaltonetworks.com/products/secure-the-network/subscriptions/minemeld
https://github.com/PaloAltoNetworks/minemeld/wiki
Docker image that worked well for my testing
https://github.com/jtschichold/minemeld-docker
docker run -it --tmpfs /run -v /somewhere/minemeld/local:/opt/minemeld/local -p 9443:443 jtschichold/minemeldto make it run as daemon after testing add the -d command to have it continue running after you exit the terminal
After installing (if you do this right you can get a certificate included in the initial build of the container that will help with the Certificate trust to NW) you will log in and set up a new output action to take your feeds and map them to a format and output that can be used with RSA NetWitness.
This is the pipeline that we will create which will map a sample threat intel list to an output action so that NetWitness can consume that information
And it gets defined by editing the yml configuration file (specifically this section creates the outboundhcvalues section that NetWitness reads)
https://github.com/epartington/rsa_nw_minemeld/blob/master/minemeld-netwitness-hcvalues.yml
outboundfeedhcvalues:
inputs:
- aggregatorIPv4Outbound-1543370742868
output: false
prototype: stdlib.feedHCGreenWithValueThis is a good start for how to create custom miners
Once created and working you will have a second miner listed and the dashboard will update
You can test the feed output using a direct API call like this via the browser
https://192.168.x.y:9443/feeds/"$feed_name"?tr=1&v=csv&f=indicator&f=confidence&f=share_level&f=sourcesthe query parameters are explained here:
https://live.paloaltonetworks.com/t5/MineMeld-Articles/Parameters-for-the-output-feeds/ta-p/146170
in this case:
| tr=1 | translate IP ranges into CIDRs. This can be used also with v=json and v=csv. |
| v=csv | returns the indicator list in CSV format.
The list of the attributes is specified by using the parameter f one or more times. The default name of the column is the name of the attribute, to specify a column name add |column_name in the f parameter value.
The h parameter can be used to control the generation of the CSV header. When unset (h=0) the header is not generated. Default: set.
Encoding is utf-8. By default no UTF-8 BOM is generated. If ubom=1 is added to the parameter list, a UTF-8 BOM is generated for compatibility. |
F are the column names from the feed
This command testing drops a file in your browser to look at and make sure you have the data and columns that you want
Now once you are confident in the process and the output format you can script and crontab the output to drop into the local feed location on the head server (I did this as i couldn't figure out how to accept the self signed certificate from the docker image).
https://github.com/epartington/rsa_nw_minemeld/blob/master/script-rsa-minemeld.sh
# 22 3 * * * /root/rsa-minemeld/script-rsa-minemeld.shNow create the same local recurring feed file to pull in the information as feed data on your decoders.
Define the column to match column 1 for the IP in CIDR notation and map the other columns as required
Done
Now we have a pipeline for two additional threat data aggregators that you may have a need for in your environment.
10 Comments
