RSA ESI 1.0 Beta 3 - Parse MAC address

[ThanaPongkittil]

I've used ESI Tool 1.0 Beta 3 to parse Free Radius but in the log MAC format is 4C-7C-5F-B0-E3-ED or 4C7C5FB0E3ED which cannot parse into the variable "maddr". There are also some products that use the format 4c7c.5fb0.e3ed.

How can I parse it?

[EricPartington]

I think MAC addresses are required in the format : separated not - or period.

If your log source is sending logs in those events you will probably need to do the following:

1. create a temporary value in the table-map-custom.xml to hold this wrongly formatted MAC as text format and keep it transient
2. use the log parser to  force the incorrect format MAC into that temp key
3. use Lua to extract the value from that temp key,  reformat according to the requirements for the MAC keys (UPPERCASE:SEPARATED) and insert into the correct key

[ThanaPongkittil]

Are there other ways to do like add or change the MAC format to accept various formats?

[ThanaPongkittil]

1. I've already used NW LPT 1.0 to parse mac address to a temporary value (hardware.id) to hold the wrong format and keep it transient.

2. I've tried to use LUA to extract the value from that temp key to eth.src key as below:

--------------------------------------------------------------------------------------------------------------------------------------

--Step 1 Create Parser
local macParser = nw.createParser(“MAC_Parser”, “Reformat MAC address to seperated by colon”)

-- Step 2 Set meta keys to register values with
macParser:setKeys({
nwlanguagekey.create("eth.src")
})

-- Step 4 DO SOMETHING
function substituteColon(mac)
mac = string.gsub(mac, "-", ":")
return mac
end

function cutDotandUpper(mac)
mac = string.gsub(mac, "%.", "")
mac = string.upper(mac)
return mac
end

function macParser:reformatMAC(token, mymeta)
if string.find(mymeta, "-") then
mymeta = substituteColon(mymeta)
elseif string.find(mymeta, "%.") then
mymeta = cutDotandUpper(mymeta)
mymeta = insertColon(mymeta)
else
mymeta = insertColon(mymeta)
end

nw.createMeta(self.keys[“eth.src”], mymeta)

end

--Step 3 Set Meta-Callbacks
macParser:setCallbacks({
[nwlanguagekey.create("hardware.id")] = macParser.reformatMAC,
})

 ---------------------------------------------------------------------------------------------------------

The mac address formats are

mac1 = "80656D12EF99"
mac2 = "C4-B3-01-46-82-B3"
mac3 = "c8f2.3022.4fcf"

But I cannot make the Lua parser to show on the Log Decoder>Config>General>Parser Configuration

The way I uploaded was go to Log Decoder>Config>Parsers>Upload and then go to Explore>decoder>parsers>properties>reload>Send

How to upload the Lua parser correctly?

[WilliamMotley1]

Check your logs, there will be error messages.  The parser is calling a function insertColon() that isn't defined.  Looks like it was actually defined as subsistuteColon()?  Also, it will need to be called as,

     mymeta=self:substituteColon(mymeta)

For uploading, you can also just put the parser directly in /etc/netwitness/ng/parsers/   You'll still need to issue a parsers reload.

If I may offer a suggestion? Looks like you are trying to normalize the format to AA:BB:CC:DD:EE:FF. But your cutDotandUpper() function replaces dots with nothing (rather than hyphens). Then substituteColon() replaces hyphens with colons, but since there are no hyphens if the original meta contained dots, then no colons will be inserted.

Try this instead:

function reformatMAC(token, mymeta)
     mymeta = string.upper(mymeta)
    local reformated = {}
    for octet in string.gmatch(mymeta, "%x%x") do
        table.insert(reformated, octet)
    end
    nw.createMeta(self.keys["eth.src"], table.concat(reformated, ":"))
end

[ThanaPongkittil]

Thank you for you reply. Sorry to post uncomplete lua code. This is the complete code.

-------------------------------------------------------------------------------------------------------------------------------------

--Step 1 Create Parser
local macParser = nw.createParser(“MAC_Parser”, “Reformat MAC address to seperated by colon”)

-- Step 2 Set meta keys to register values with
macParser:setKeys({
 nwlanguagekey.create("eth.src")
})

-- Step 4 DO SOMETHING
function substituteColon(mac)
 mac = string.gsub(mac, "-", ":")
 return mac
end

function cutDotandUpper(mac)
 mac = string.gsub(mac, "%.", "")
 mac = string.upper(mac)
 return mac
end

function insertColon(mac)
 mac = mac:gsub("..", ":%0"):sub(2)
 return mac

function macParser:reformatMAC(token, mymeta)
 if string.find(mymeta, "-") then
  mymeta = substituteColon(mymeta)
 elseif string.find(mymeta, "%.") then
  mymeta = cutDotandUpper(mymeta)
  mymeta = insertColon(mymeta)
 else
  mymeta = insertColon(mymeta)
 end

 nw.createMeta(self.keys[“eth.src”], mymeta)

end

--Step 3 Set Meta-Callbacks
macParser:setCallbacks({
 [nwlanguagekey.create("hardware.id")] = macParser.reformatMAC,
})

-----------------------------------------------------------------------------------------------------------------------

The mac address formats are

mac1 = "80656D12EF99"
mac2 = "C4-B3-01-46-82-B3"
mac3 = "c8f2.3022.4fcf"

So, my idea is

   if it's mac1's format, I will insert colon to it.

   if it's mac2's format, I will substitute "-" with ":".

   if it's mac3's format, I will remove "." and capitalize it then insert colon to it.

Please, recheck my code and idea again. If it's correct, I will try to upload it to the Log Decoder.

Anyway, your suggestion is very good. Thanks for your suggestion.

[ThanaPongkittil]

I've already uploaded to /etc/netwitness/ng/parsers/ and reloaded parsers but it's not even shown in Parsers Configuration.

The way I reloaded was go to Explore>decoder>parsers>properties>reload>Send

Is it correct?

[WilliamMotley1]

Check your log - there will be error messages.  One issue I notice is that the insertColon function isn't terminated - there is no "end" statement for it.

The example function from my previous response will handle all of those formats.

[Anonymous]

I wrote a parser a few years ago that would help with the normalization.  It could help you or be used as a reference to build your own.  It's old and there may be better ways to write it, but this is what I got working about 4 years ago.

MAC formatted meta keys need to be in UPPER CASE separated by a COLON :  

We can also do a string length check to see if the initial transient meta value is 12 characters.  If thats the case, could just split on every 2 characters.

==========================================

local normalizemac = nw.createParser("Normalize_MAC", "Normalize MAC address from certain logs")

--[[ The purpose of this parser is to normalize meta from the meta callback into a
properly formatted MAC address.

Parser created 2014-03-25 by Chris Ahearn
christopher.ahearn@rsa.com
--]]

normalizemac:setKeys({
nwlanguagekey.create("eth.src", nwtypes.MAC)
})

function normalizemac:macMeta(index, mac)
-- localize or initialize table to hold seen mac addresses
local seenmac = self.seenmac or {}
-- check if this mac address has been seen before
if not seenmac[mac] then
-- nope, now it has
seenmac[mac] = true
-- copy local table back to global
self.seenmac = seenmac
-- Convert mac to upper case
local normal_mac = string.upper(string.gsub(mac, "-", ":"))
-- *check if this mac has been registered before
if not seenmac[normal_mac] then
nw.createMeta(self.keys["eth.src"], normal_mac)
end
end
end

function normalizemac:sessionEnd()
self.seenmac = nil
end

--[[ Place meta callback info here. You may need to modify the log parser to use a different
transient meta key as TEXT. This is because eth.dst is formatted as MAC which expects
AA:BB:CC:11:22:33 format. The dmacaddr key below is ONLY a placeholder. -]]

normalizemac:setCallbacks({
[nwlanguagekey.create("sourcemac")] = normalizemac.macMeta,
[nwevents.OnSessionEnd] = normalizemac.sessionEnd,
})

==========================================