1 ACCEPTED SOLUTION
Accepted Solutions
6MAY: EDITED lua parser to remove the bad arguments in the nw.getPayload function
I've fielded this type of ask from a number of different customers recently, and the only solution we can offer at this time (especially for CEF logs) is to extract this type of information via custom lua parser.
And just for some details on why this is the only solution...
- the CEF parser relies on key=value fields within the log to allow for flexible parsing of logs that have those fields in random/arbitrary order (different vendors, different platforms, etc...can all be parsed by the CEF parser thanks to this flexibility)
- but that flexibility is limited when trying to extract substrings from within CEF values
- so we need lua to help out with those substrings
The first consideration in writing the lua parser for these situations is how to trigger (or callback) the lua. We don't want this lua parser to trigger on unrelated logs, as that will have a negative performance impact on the log decoder. So the best way I have found to ensure custom lua parsers fire only against the logs that I want, is to:
- modify the associated parser (CEF, in this case) to create a custom and unique metakey that will only be created by that specific log source/log event
- you'll also need to add that custom/unique metakey to the table-map-custom.xml
- this custom/unique metakey can be set to Transient, meaning it will be available for parsers, app rules, and feeds on the Decoder, but will be discarded and not pulled into the Concentrator
- this is the custom_msg_temp key in the lua below
- have your lua parser trigger anytime a meta value is created in that custom and unique metakey
There's lots of resources online that you can use to write and understand lua...some of which are... (https://community.rsa.com/t5/rsa-netwitness-platform-threat/a-treatise-on-writing-packet-parsers-for-the-rsa-netwitness/ta-p/568861; https://www.lua.org/manual/5.1/; https://replit.com/languages/lua).
Here's what that lua parser could look like:
local parserName = "swift_alliance_access_cef_lua"
local parserVersion = "2021.04.30"
--[[
This parser requires a custom meta key.
Log Decoder:
cef-custom.xml
<VendorProducts>
<Vendor2Device vendor="Aruba Networks" product="ClearPass" device="swift_alliance_access" group="Analysis"/>
</VendorProducts>
<ExtensionKey cefName="msg" metaName="msg">
<device2meta device="swift_alliance_access" metaName="custom_msg_temp"/>
</ExtensionKey>
table-map-custom.xml
<mapping envisionName="custom_msg_temp" nwName="custom_msg_temp" flags="Transient" format="Text"/>
--]]
local cefCustomLua = nw.createParser(parserName, "swift_alliance_access_cef_lua -- parse out substrings from swift_alliance_access CEF logs.")
-- register the meta keys to add new values to
cefCustomLua:setKeys({
nwlanguagekey.create("ec.subject"),
})
function cefCustomLua:extractInfo(token,first,last)
--set the pattern we want to match against with a capture group to return the substring value we are looking for
local patternMatch = "msg=(.-) : Modified with password"
local logPayload = nw.getPayload()
-- start a "try/catch" block (error trapping)
local status, err = pcall(function()
if logPayload then
local rawLog = logPayload:tostring()
--ensure we have the raw log and that it is a string
if rawLog ~= nil and type(rawLog) == "string" then
local valueMatch = string.match(rawLog, patternMatch)
--ensure we matched the substring, and if we did create our meta
if valueMatch ~= nil then
nw.createMeta(self.keys["ec.subject"], valueMatch)
end
end
end
end, logPayload) -- end "try/catch" block
debugParser = false -- set debug mode for this parser
if not status and debugParser then
-- if we have an error in the parsing, write it to /var/log/messages with a string we can grep against
nw.logFailure("swift_alliance_access_cef_lua error: " .. tostring(err))
end
end
-- look for the existence of meta keys OR look for specific text in the raw session/log
cefCustomLua:setCallbacks({
[nwlanguagekey.create("custom_msg_temp")] = cefCustomLua.extractInfo,
})
And a quick look at the result of the pattern match:
6MAY: EDITED lua parser to remove the bad arguments in the nw.getPayload function
I've fielded this type of ask from a number of different customers recently, and the only solution we can offer at this time (especially for CEF logs) is to extract this type of information via custom lua parser.
And just for some details on why this is the only solution...
- the CEF parser relies on key=value fields within the log to allow for flexible parsing of logs that have those fields in random/arbitrary order (different vendors, different platforms, etc...can all be parsed by the CEF parser thanks to this flexibility)
- but that flexibility is limited when trying to extract substrings from within CEF values
- so we need lua to help out with those substrings
The first consideration in writing the lua parser for these situations is how to trigger (or callback) the lua. We don't want this lua parser to trigger on unrelated logs, as that will have a negative performance impact on the log decoder. So the best way I have found to ensure custom lua parsers fire only against the logs that I want, is to:
- modify the associated parser (CEF, in this case) to create a custom and unique metakey that will only be created by that specific log source/log event
- you'll also need to add that custom/unique metakey to the table-map-custom.xml
- this custom/unique metakey can be set to Transient, meaning it will be available for parsers, app rules, and feeds on the Decoder, but will be discarded and not pulled into the Concentrator
- this is the custom_msg_temp key in the lua below
- have your lua parser trigger anytime a meta value is created in that custom and unique metakey
There's lots of resources online that you can use to write and understand lua...some of which are... (https://community.rsa.com/t5/rsa-netwitness-platform-threat/a-treatise-on-writing-packet-parsers-for-the-rsa-netwitness/ta-p/568861; https://www.lua.org/manual/5.1/; https://replit.com/languages/lua).
Here's what that lua parser could look like:
local parserName = "swift_alliance_access_cef_lua"
local parserVersion = "2021.04.30"
--[[
This parser requires a custom meta key.
Log Decoder:
cef-custom.xml
<VendorProducts>
<Vendor2Device vendor="Aruba Networks" product="ClearPass" device="swift_alliance_access" group="Analysis"/>
</VendorProducts>
<ExtensionKey cefName="msg" metaName="msg">
<device2meta device="swift_alliance_access" metaName="custom_msg_temp"/>
</ExtensionKey>
table-map-custom.xml
<mapping envisionName="custom_msg_temp" nwName="custom_msg_temp" flags="Transient" format="Text"/>
--]]
local cefCustomLua = nw.createParser(parserName, "swift_alliance_access_cef_lua -- parse out substrings from swift_alliance_access CEF logs.")
-- register the meta keys to add new values to
cefCustomLua:setKeys({
nwlanguagekey.create("ec.subject"),
})
function cefCustomLua:extractInfo(token,first,last)
--set the pattern we want to match against with a capture group to return the substring value we are looking for
local patternMatch = "msg=(.-) : Modified with password"
local logPayload = nw.getPayload()
-- start a "try/catch" block (error trapping)
local status, err = pcall(function()
if logPayload then
local rawLog = logPayload:tostring()
--ensure we have the raw log and that it is a string
if rawLog ~= nil and type(rawLog) == "string" then
local valueMatch = string.match(rawLog, patternMatch)
--ensure we matched the substring, and if we did create our meta
if valueMatch ~= nil then
nw.createMeta(self.keys["ec.subject"], valueMatch)
end
end
end
end, logPayload) -- end "try/catch" block
debugParser = false -- set debug mode for this parser
if not status and debugParser then
-- if we have an error in the parsing, write it to /var/log/messages with a string we can grep against
nw.logFailure("swift_alliance_access_cef_lua error: " .. tostring(err))
end
end
-- look for the existence of meta keys OR look for specific text in the raw session/log
cefCustomLua:setCallbacks({
[nwlanguagekey.create("custom_msg_temp")] = cefCustomLua.extractInfo,
})
And a quick look at the result of the pattern match:
@MaximMarchenko I see two immediate issues here.
First, the error message "The language key 'custom_meta_swift' exceeds the maximum size of 16" is telling you that custom_meta_swift is too long for a metakey name. NW has a hard limit for metakey name lengths of 16 bytes/characters, and custom_meta_swift is 1 character too long. That will need to be changed to a shorter metakey name.
Second, that shorter metakey name will need to be the same within your table-map-custom and the lua parser's setCallbacks function (the red boxes in this screen shot):
Ahh, sorry about that, looks like I had my own stupid copy-paste mistake. Its a simple fix, just have to remove the arguments from the nw.getPayload() function on line 33:
local logPayload = nw.getPayload()
...and that's it. I should have run through this fully the first time - would have prevented me from posting that incorrectly.
