Snort to NextGen Field Mappings overrides

[RQuinn]

We have been using the snort parsing in our environment with great success but one issue we are running into is the volume of rules we have running is clogging up our risk.info category. While we're working through some tuning, one of my analysts wanted to know if it was possible to put the rules into a dedicated category rather than in the risk categories. In looking through other discussions on these forums i saw the below mappings were in place. While I'm quite comfortable looking at the snort.config file and making adjustments, these options did not seem to be available.

Is it possible to over ride these mappings?

Snort to NextGen Field Mappings

Snort Field            NextGen Meta

"snort rule"      threat.source

sid           alert.id

classtype      threat.category

message risk.*

1. rule.priority/classtype these are used to decide which risk meta category is used for message
   1 - risk.warning

            2 - risk.suspicious

            3 - risk.info

[Anonymous]

I don't believe it is possible to override them, but I am checking as to what options we have available.  What version of SA / Netwitness are you running?

Chris

[RQuinn]

10.5.0.1

[Anonymous]

Still researching this but wanted to share what I have...

I don't think there is an easy way to change where the "snort parser" writes it's meta.  It is not actually a 'parser' and would likely require a code change.

I tried writing a parser that would actually just copy meta from 'risk.xxxxx' to a new meta key called 'rule' whenever the threat.source equaled 'snort rule'.  This worked, but had some collateral damage as it would copy ANY other 'risk.xxxxx' meta that was generated...not just that generated from snort rules. 

The only possible means at this point, would be creating a feed that contained every snort rule in your decoder and writing that as meta into another key.  Beginning in 10.5, the packet decoders have the ability to make transient meta per parser.  The idea here would be to go into the Decoder/Config/General tab, drill into the Snort parser in the Parser configuration panel, and change 'risk.xxxxx' keys from 'Enabled' to 'Transient'.  The feed can still match against transient meta and write to the alternate key.

I don't think maintaining a feed of Snort rules is an ideal solution, but the only one I could come up with that would work.

Chris

[Anonymous]

As I suspected, there is no configuration to write Snort generated meta into an alternate meta key.

As I mentioned, I wrote a parser that would just copy the meta from 'risk.xxxxx' into 'rule' where threat.source equaled 'snort rule'.  The snort meta did copy, as indicated in the graphic below, but it also grabbed the other 'risk.xxxxx' meta associated with that session.

A feed may still be the best way to go.  Another would be if the Snort rules were prepended with some kind of descriptor, such as 'ET ' in the case of Emerging Threats rules, I could update the custom parser to only copy the value if it began with that prepend value.  Even that could be removed once the meta gets written.

[Anonymous]

As I mentioned, one possible way to handle this would be with a feed, but that would need to include every rule.  I didn't particularly like that option.  I had initially created a parser to copy the meta from risk.xxxxx to 'rule' but there was collateral damage in copying non-snort meta as I referenced above.  Therefore I scrapped that idea.

One alternative I came up with was using a parser that would do a string.find against values in a table.  I compiled the table by examining the rule groups of the Community and Emerging Threats Snort signatures.  I also saw a couple rule groups in Sourcefire, but did not have a complete listing.

Basically, this parser does a check for threat.source == 'snort rule'.  If that is a match, it then takes the meta from the risk.xxxxx key and does a string.find for the rule group values in the table.  If that matches, it will write the rule as meta.  By doing this comparison against a predefined value in the table, we can extract just the Snort meta and not bring in any other risk.xxxxx meta that was also associated with that session.

If you have custom Snort rules that follow a similar naming convention, you could add those to the table.  The way the function is designed, it is looking for the value in any part of the risk.xxxxx meta.  It could be set up such that it looks only at the BEGINNING of that meta for the comparison.

Parser below:

---------------------------------------

local lua_snort = nw.createParser("lua_snort", "Move Snort Meta into an alternate meta key")

--[[

    DESCRIPTION

        Move Snort Meta into an alternate meta key

    AUTHOR

    christopher.ahearn@rsa.com

    VERSION

  2015-09-12 - Added values from emerging threats and community rules to the

  sigfind table

  2015-09-11 - Added sigfind table to give us a chance at copying just the meta

  we want

  2015-09-09 - Initial development

    DEPENDENCIES

        For 10.5 and above, you may want to change the Snort parser risk.xxxxx keys from

        'Enabled' to 'Transient'

    META KEYS

    Add meta key(s) to index-concentrator-custom.xml:

    <key description="Rules" format="Text" level="IndexValues" name="rule" valueMax="100000" />

--]]

-- Write meta into the following meta key(s)

lua_snort:setKeys({

  nwlanguagekey.create("rule",nwtypes.Text), -- our alternate meta key

})

--[[

Using a table that contains various rule groups of Snort rules.  This was the only

way I could figure out that would just copy the snort rule meta from risk.xxxxx without

getting all the other risk.xxxxx meta for a given session.  This was pulled from Snort

Emerging Threats and Community rules using some grep and awk foo against the rules files.

Note the capitalization as that will be an important identifier.

If you have other rule groups or unique identifiers, add to the list and make sure you

include the trailing comma after true.

--]]

local sigfind = ({

  ["ET POLICY"] = true, -- Emerging Threat Rules

  ["ET ATTACK_RESPONSE"] = true, -- Emerging Threat Rules

  ["ET CINS"] = true, -- Emerging Threat Rules

  ["ET CNC"] = true, -- Emerging Threat Rules

  ["ET COMPROMISED"] = true, -- Emerging Threat Rules

  ["ET DROP"] = true, -- Emerging Threat Rules

  ["ET TOR"] = true, -- Emerging Threat Rules

  ["GPL ATTACK_RESPONSE"] = true, -- Emerging Threat Rules

  ["APP-DETECT"] = true, -- Community Rules

  ["BLACKLIST"] = true, -- Community Rules

  ["BROWSER-FIREFOX"] = true, -- Community Rules

  ["BROWSER-IE"] = true, -- Community Rules

  ["BROWSER-OTHER"] = true, -- Community Rules

  ["BROWSER-PLUGINS"] = true, -- Community Rules

  ["BROWSER-WEBKIT"] = true, -- Community Rules

  ["EXPLOIT-KIT"] = true, -- Community Rules

  ["FILE-IDENTIFY"] = true, -- Community Rules

  ["FILE-IMAGE"] = true, -- Community Rules

  ["FILE-JAVA"] = true, -- Community Rules

  ["FILE-MULTIMEDIA"] = true, -- Community Rules

  ["FILE-OTHER"] = true, -- Community Rules

  ["FILE-OFFICE"] = true, -- Sourcefire Rules

  ["FILE-FLASH"] = true, -- Sourcefire Rules

  ["FILE-PDF"] = true, -- Community Rules

  ["INDICATOR-COMPROMISE"] = true, -- Community Rules

  ["INDICATOR-OBFUSCATION"] = true, -- Community Rules

  ["INDICATOR-SCAN"] = true, -- Community Rules

  ["INDICATOR-SHELLCODE"] = true, -- Community Rules

  ["MALWARE-BACKDOOR"] = true, -- Community Rules

  ["MALWARE-CNC"] = true, -- Community Rules

  ["MALWARE-OTHER"] = true, -- Community Rules

  ["MALWARE-TOOLS"] = true, -- Community Rules

  ["NETBIOS"] = true, -- Community Rules

  ["OS-LINUX"] = true, -- Community Rules

  ["OS-MOBILE"] = true, -- Community Rules

  ["OS-OTHER"] = true, -- Community Rules

  ["OS-SOLARIS"] = true, -- Community Rules

  ["OS-WINDOWS"] = true, -- Community Rules

  ["POLICY-MULTIMEDIA"] = true, -- Community Rules

  ["POLICY-OTHER"] = true, -- Community Rules

  ["POLICY-SOCIAL"] = true, -- Community Rules

  ["POLICY-SPAM"] = true, -- Community Rules

  ["PROTOCOL-DNS"] = true, -- Community Rules

  ["PROTOCOL-FINGER"] = true, -- Community Rules

  ["PROTOCOL-FTP"] = true, -- Community Rules

  ["PROTOCOL-ICMP"] = true, -- Community Rules

  ["PROTOCOL-IMAP"] = true, -- Community Rules

  ["PROTOCOL-NNTP"] = true, -- Community Rules

  ["PROTOCOL-POP"] = true, -- Community Rules

  ["PROTOCOL-RPC"] = true, -- Community Rules

  ["PROTOCOL-SERVICES"] = true, -- Community Rules

  ["PROTOCOL-SNMP"] = true, -- Community Rules

  ["PROTOCOL-TELNET"] = true, -- Community Rules

  ["PROTOCOL-TFTP"] = true, -- Community Rules

  ["PROTOCOL-VOIP"] = true, -- Community Rules

  ["PUA-ADWARE"] = true, -- Community Rules

  ["PUA-OTHER"] = true, -- Community Rules

  ["PUA-P2P"] = true, -- Community Rules

  ["PUA-TOOLBARS"] = true, -- Community Rules

  ["SERVER-APACHE"] = true, -- Community Rules

  ["SERVER-IIS"] = true, -- Community Rules

  ["SERVER-MAIL"] = true, -- Community Rules

  ["SERVER-MSSQL"] = true, -- Community Rules

  ["SERVER-MYSQL"] = true, -- Community Rules

  ["SERVER-ORACLE"] = true, -- Community Rules

  ["SERVER-OTHER"] = true, -- Community Rules

  ["SERVER-WEBAPP"] = true, -- Community Rules

  ["SQL "] = true, -- Community Rules ... 1 extra space added on purpose

  ["X11 "] = true, -- Community Rules ... 1 extra space added on purpose

})

-- Reset Parser Variables

function lua_snort:sessionBegin()

  snort = nil

end

-- Check threat source to ensure we are only looking at sessions with Snort rules.

function lua_snort:snortMeta(index,value)

  if value == 'snort rule' then

  snort = 1

  -- Enable nw.logInfo for parser debugging

  --nw.logInfo("*** SNORT RULE MATCH ***")

  end

end

--[[

Copy meta from risk.xxxxx into new meta key ONLY if our threat source match above is

correct.  Then compare it with our sigfind table looking for a match.  If it matches,

write it as meta into the alternate meta key.  If not, move along.

--]]

function lua_snort:msgMeta(index,meta)

  if snort == 1 then

  -- Enable nw.logInfo for parser debugging

  --nw.logInfo("*** RISK META: " .. meta .. " ***")

  for i,j in pairs(sigfind) do

  local foundfirst, foundlast = string.find(meta, i)

  if foundfirst ~= nil then

  -- Enable nw.logInfo for parser debugging

  --nw.logInfo("*** SNORT RULE TABLE MATCH: " .. i .. " ***")

  nw.createMeta(self.keys["rule"], meta)

  end

  end

  end

end

-- declare what tokens and events we want to match

lua_snort:setCallbacks({

    [nwevents.OnSessionBegin] = lua_snort.sessionBegin,

    [nwlanguagekey.create("threat.source", nwtypes.Text)] = lua_snort.snortMeta,

    [nwlanguagekey.create("risk.info", nwtypes.Text)] = lua_snort.msgMeta,

    [nwlanguagekey.create("risk.suspicious", nwtypes.Text)] = lua_snort.msgMeta,

    [nwlanguagekey.create("risk.warning", nwtypes.Text)] = lua_snort.msgMeta,    

})

---------------------------------------

Save it as snort.lua, copy it to your /etc/netwitness/ng/parsers directory and restart the nwdecoder service.  You may want to add a custom index to your concentrators index-concentrator-custom.xml to account for the new 'rule' meta key.

The results:

While there were some other risk.xxxxx meta in the second 'Snort Rule Match', only the Snort signature was registered as meta.

I hope this helps.

Chris

[RSAAdmin]

Thanks Chris. Tried this in my test lab and it works a treat!

[Anonymous]

I am currently testing one that would work without the rule group (sigfind) table.

This way it's one less thing to maintain.

Chris

Sent from my iPhone

[RSAAdmin]

Just in case it helps anyone else, I added a key called snort.rule to put my meta into.

I needed to change the lines

  nwlanguagekey.create("rule",nwtypes.Text), -- our alternate meta key

to

  nwlanguagekey.create("snort.rule",nwtypes.Text), -- our alternate meta key

And also:

nw.createMeta(self.keys["rule"], meta)

to

nw.createMeta(self.keys["snort.rule"], meta)

[Anonymous]

Not happy with having to maintain a table, this was the version of the parser I was developing to help with that.  This focuses on the order in which snort meta is written vs comparing it to a predefined table.

---------------------------------------

local lua_snort = nw.createParser("lua_snort", "Move Snort Meta into an alternate meta key")

--[[

    DESCRIPTION

        Move Snort Meta into an alternate meta key

    AUTHOR

    christopher.ahearn@rsa.com

    VERSION

  2015-09-14 - Removed sigfind table and changed parser by looking at order in

  which Snort meta is written instead.  This means doing meta

  callbacks against alert.id and threat.category as well as

  threat.source and risk.xxxxx.  By checking the order in which the

  meta is written, we can ensure the risk.xxxxx meta is only coming

  from a snort rule and can be written into the 'rule' meta key.

  2015-09-12 - Added values from emerging threats and community rules to the

  sigfind table

  2015-09-11 - Added sigfind table to give us a chance at copying just the meta

  we want

  2015-09-09 - Initial development

    DEPENDENCIES

        For 10.5 and above, you may want to change the Snort parser risk.xxxxx keys from

        'Enabled' to 'Transient'

        Make sure that classification is enabled on the snort configuration.  Without it,

        threat.category meta may not get written, which is a key component for this parser

    META KEYS

    Add meta key(s) to index-concentrator-custom.xml:  

    <key description="Rules" format="Text" level="IndexValues" name="rule" valueMax="100000" />

--]]

-- Write meta into the following meta key(s)

lua_snort:setKeys({

  nwlanguagekey.create("rule",nwtypes.Text), -- our alternate meta key

})

-- Reset Parser Variables

function lua_snort:sessionBegin()

  snort = nil

  snortorder = nil

end

-- Check threat source to ensure we are only looking at sessions with Snort rules.

function lua_snort:snortMeta(index,value)

  if value == 'snort rule' then

  -- delete the 'snortorder' table if it exists

  if snortorder then

  for k in pairs(snortorder) do

  snortorder = nil

  end

  end

  -- create the 'snortorder' table

  snortorder = {}

  -- insert into the 'snortorder' table

  table.insert(snortorder, "threat.source")

  -- set a tracking variable

  snort = 1

  -- Enable nw.logInfo for parser debugging

  --nw.logInfo("*** SNORT THREAT SOURCE MATCH ***")

  end

end

function lua_snort:alertidMeta(index,meta)

  if snort == 1 then

  -- insert into the 'snortorder' table

  table.insert(snortorder, "alert.id")

  -- Enable nw.logInfo for parser debugging

  --nw.logInfo("*** SNORT ALERT ID MATCH ***")

  -- update the tracking variable

  snort = 2

  end

end

function lua_snort:threatcatMeta(index,meta)

  if snort == 2 then

  -- insert into the 'snortorder' table

  table.insert(snortorder, "threat.category")

  -- Enable nw.logInfo for parser debugging

  --nw.logInfo("*** SNORT THREAT CATEGORY MATCH ***")

  -- update the tracking variable

  snort = 3

  end

end

function lua_snort:riskMeta(index,meta)

  if snort == 3 then

  -- insert into the 'snortorder' table

  table.insert(snortorder, "risk")

  -- check the order in which the 'snortorder' entries were made

  -- entries must be in this specific order

  if snortorder[1] == "threat.source" then

  if snortorder[2] == "alert.id" then

  if snortorder[3] == "threat.category" then

  if snortorder[4] == "risk" then

  -- write meta

  nw.createMeta(self.keys["rule"], meta)

  -- Enable nw.logInfo for parser debugging

  --nw.logInfo("*** SNORT RULE MATCH ***")

  -- clear out the table

  for k in pairs(snortorder) do

  snortorder = nil

  end

  -- reset the tracking variable

  snort = nil

  end

  end

  end

  end

  end

end

-- declare what tokens and events we want to match

lua_snort:setCallbacks({

    [nwevents.OnSessionBegin] = lua_snort.sessionBegin,

    [nwlanguagekey.create("threat.source", nwtypes.Text)] = lua_snort.snortMeta,

    [nwlanguagekey.create("alert.id", nwtypes.Text)] = lua_snort.alertidMeta,

    [nwlanguagekey.create("threat.category", nwtypes.Text)] = lua_snort.threatcatMeta,

    [nwlanguagekey.create("risk.info", nwtypes.Text)] = lua_snort.riskMeta,

    [nwlanguagekey.create("risk.suspicious", nwtypes.Text)] = lua_snort.riskMeta,

    [nwlanguagekey.create("risk.warning", nwtypes.Text)] = lua_snort.riskMeta,       

})

---------------------------------------

Save it as snort.lua, copy it to your /etc/netwitness/ng/parsers directory and restart the nwdecoder service.  You may want to add a custom index to your concentrators index-concentrator-custom.xml to account for the new 'rule' meta key.

Make sure you have classifications enabled on the snort.conf configuration.

Hope this helps.

Chris