As Netwitness supports the ability to ingest and use Snort rules as parsers, I thought I'd share how our company is currently deploying them to our packet decoders via Puppet.
We first had to create a 'files' folder on the puppetmaster (in this case our broker) under the following directory:
/etc/puppet/modules/decoder/
The Snort rules files and the snort.conf file will be stored in this directory. At the time of this writing, there is no version control enabled.
On each of our packet decoders, we created a Snort folder in the parsers directory which will store our rules files and snort.conf file.
/etc/netwitness/ng/parsers/snort/
On our broker/SA server, we have a short bash script that downloads the ET Pro Snort rules tarball and unpacks it. We then grab whatever rules we want and write them to the 'local.rules' file, which resides in decoder module file directory:
export http_proxy="http://<proxy_name>:<proxy_port>"
curl http://rules.emergingthreatspro.com/<oink_code_for_etpro>/snort-2.9.2/etpro.rules.tar.gz > /root/snort.rules.tar.gz
tar -zxf /root/snort.rules.tar.gz
cat /root/rules/*.rules | egrep 'search_term_1|search_term_2' > /etc/puppet/modules/decoder/files/local.rules
Then we just modified the Puppet decoder init.pp script to place the files in the parsers/snort/ directory we created on each decoder (/etc/puppet/modules/decoder/manifests/init.pp). See the relevant snippet of our init.pp file below. Because this is specific to the decoder module in Puppet, it will only be pushed out to the decoders. For each .rules file you wish to distribute, you'll have to add a similar block of code as Puppet doesn't play well with wildcards.
file { '/etc/netwitness/ng/parsers/snort/local.rules':
ensure => present,
replace => true,
owner => 'root',
group => 'root',
mode => 644,
source => 'puppet:///modules/decoder/local.rules'
}
file { '/etc/netwitness/ng/parsers/snort/snort.conf':
ensure => present,
replace => true,
owner => 'root',
group => 'root',
mode => 644,
source => 'puppet:///modules/decoder/snort.conf'
}
Since Puppet runs every half hour or so, we wait an hour before running a script to reload the parsers. To verify that the updated files have been pushed, look in the logs on the decoders. If any changes were made to an existing rules file (in this case just our local.rules file which gets updated every night), you'll see something similar to the following line in the messages file on the decoders.
puppet-agent[97145]: (/Stage[main]/Decoder/File[/etc/netwitness/ng/parsers/snort/local.rules]/content) content changed '{md5}72318e38791105c71819567cbedc0e20' to '{md5}077220edb5c5978fb3f8d2d51ae98a3b'
Once this is done, we run a script that cycles through a list of our decoder IPs and reloads the parsers via an API command thus, making the rules live.
for i in `cat decoder_list.txt` ; do curl http://admin:<admin_password>@$i:50104/decoder/parsers?msg=reload; done
Since this runs an hour after our Snort script ran, we can safely assume that the updated files have been pushed to the
decoders by now. To verify that the rules have been enabled, check the messages file on the decoders again and grep for Snort. You should see something similar to the following lines:
Nov 28 20:06:32 REDACTED NwDecoder[4711]: [Snort] [info] Loaded 786 snort rules, 197 small tokens, 294 with pcres, 739 partial
Nov 28 20:06:32 REDACTED NwDecoder[4711]: [Snort] [warning] Dropped 400 rules that did not have content options
Nov 28 20:06:32 REDACTED NwDecoder[4711]: [Snort] [info] Loaded local.rules, full 6, parital 828, failures 1
More on the warning log in the lessons and tips section.
The workflow then goes like this for daily Snort updates:
10 PM - Snort script grabs new tarball and throws the rules we want into the puppet files directory
~10:30 PM - Puppet runs automatically and deploys the new files, which smash the old ones on the decoders
11 PM - Parser script runs to reload the parsers, making all rules in the files live
Lessons Learned and Tips:
1. Be granular about which rules you want to deploy. In our environment, we began noticing performance issues after we increased the rule count to about 3000. For additional context, we are also running 124 lua parsers, 371 app rules, and 2 network rules per decoder, with our busiest decoders maintainingg a capture rate of ~8000 MbPS. Also keep in mind that complexity of the Snort rules and inclusion of pcres will affect performance as well.
2. If you see the warning error above mentioning it dropped X amount of rules because they lacked content options, this is because you most likely included a rule like the one below.
alert udp $HOME_NET any -> [100.2.4.245,100.6.61.161,103.208.86.114,103.208.86.43,103.208.86.44,103.209.192.57,103.27.202.166,104.131.182.103,104.162.93.136,104.168.102.127,104.168.169.140,
104.168.171.125,104.168.62.141,104.168.62.226,104.168.62.236,104.206.199.155,104.223.119.155,104.223.12.116,104.223.125.172,104.223.47.104,104.232.34.144,104.232.34.164,104.232.35.123,104.2
32.35.37,104.238.173.18,104.238.213.7,104.238.215.103,104.238.215.108,104.238.215.110,104.255.96.123,104.255.97.199,104.255.97.203,104.255.97.239,104.36.80.16,107.155.118.114,107.155.120.13
7,107.15.99.91,107.161.159.30,107.170.20.33,107.172.253.71,107.181.152.107,108.183.201.134,108.183.203.14,108.30.105.211,108.7.231.42,109.104.160.196,109.104.161.167,109.104.162.138,109.104
.162.56,109.104.162.61,109.104.163.54,109.104.164.178,109.104.165.241,109.104.165.79,109.104.168.111,109.104.168.159,109.104.168.193] any (msg:"ET CNC Ransomware Tracker Reported CnC Server
UDP group 1"; reference:url,doc.emergingthreats.net/bin/view/Main/BotCC; reference:url,ransomwaretracker.abuse.ch; threshold: type limit, track by_src, seconds 3600, count 1; classtype:tro
jan-activity; flowbits:set,ET.Evil; flowbits:set,ET.BotccIP; sid:2404401; rev:4416;)
When using Snort, Netwitness ingores the rule header and only focuses on the rule options, specifically the content keyword. If you include a rule like the one above that just looks for traffic to IPs without any content options specified, Netwitness will not use it. Which brings us to our next point.
3. For the snort.conf file (which should be placed in the same directory as the rules files on the decoders), there isn't much customization needed. Since rule headers are ignored, there is no reason to set any of the NET variables. Currently, we are only using the local.rules file (for ET Pro rules) and the info.rules file (for custom Snort rules). These files are already listed in the snort.conf file. If you wish to use a rules file with a custom name, you must add it to the bottom section of the snort.conf file.
4. Use a consistent naming convention when creating messages (msg:"some message") for your custom rules. Netwitness takes the msg from each rule that fired and populates the risk.info meta key with that value. Since all ET Pro Snort rules have a message that begins with ET, it was easy for us to create an app rule that looks for this (et pro snort rule = risk.info begins 'et'). Supposedly, the SID from each rule will populate the alert.id meta key, but we haven't gotten it to work quite yet.
5. Think of other ways to leverage puppet. We are using a similar method to deploy custom lua parsers to our decoders.
I can confirm this as well, and do not need screenshots. It appears this is a bug and was reported by Eric (referenced above). I am also seeing this in alert.id. It appears related to some recent 10.6 versions.
I believe this is something that would require a code fix as I don't think a parser can pick up the content in another way. The case has been assigned and hopefully a fix can be put together quickly.
