Custom Parsing with Lua

[Anonymous]

For the past year and a half, I have been working with Lua on our packet and log decoders in the form of custom parsers.  During this time, I have come across some interesting ways that Lua has been able to help with organizations use cases.  This involved both the extraction of meta from raw network traffic or doing something else with previously generated meta in both packet and log decoders.

For those out in the community, are you writing your own Lua parsers?  Are there use-cases where this has become necessary?  Are there ideas out there where you feel a custom parser is necessary, but not sure how or where to start?  Have you written your own that you feel you would like to share? 

Thank you for reading and contributing.

Chris

[Anonymous]

A test parser I called test.lua

\

-- Step 1 - Create parser

local luatest = nw.createParser("lua_test", "TEST PARSER")

--[[

This is a test parser.  It is intended to help learn how to write parsers with Lua.

This parser requires a custom meta key.

Concentrator: index-concentrator-custom.xml

  <key description="Test Alert" level="IndexValues" name="test.alert" valueMax="1000" format="Text"/>

--]]

-- Step 2 - Define meta keys to write meta into

-- declare the meta keys we'll be registering meta with

luatest:setKeys({

  nwlanguagekey.create("test.alert"),

})

-- Step 4 - Do SOMETHING once your token matched

function luatest:tokenFIND(token, first, last)

  nw.createMeta(self.keys["test.alert"], "i_found_you")

end

-- Step 3 - Define tokens that get you close to what you want

-- declare what tokens and events we want to match. 

-- These do not have to be exact matches but just get you close to the data you want.

luatest:setCallbacks({

  ["cnn.com\013\010"] = luatest.tokenFIND,

})

[Anonymous]

Another example.  HTTP URL parser to build a url from http traffic.

local luaHttpUrl = nw.createParser("lua_http_url", "LUA HTTP URL", "80")

--[[

    DESCRIPTION

        Build a full url based on http traffic

    VERSION

        2014.12.15  christopher.ahearn@rsa.com - Initial development

    DEPENDENCIES

        http traffic

--]]

-- Write meta into the following meta key(s)

luaHttpUrl:setKeys({

  nwlanguagekey.create("url"),

})

function luaHttpUrl:sessionBegin()

  -- reset parser_state for the new session

  self.Path = nil

end

function luaHttpUrl:tokenRequest(token, first, last)

  -- set position to byte after the token

  current_position = last + 1

  -- get the payload

  local payload = nw.getPayload()

  -- find the end of the path

  local num_temp = payload:find(" HTTP/", current_position, current_position + 4096)

  -- if we found the end of the line

  if num_temp then

  -- we don't want to read the " "

  num_temp = num_temp - 1

  -- read up to the end of the line

  self.Path = payload:tostring(current_position, num_temp)

  end

end

function luaHttpUrl:tokenhost(token, first, last)

  -- set position to byte after the token

  current_position = last + 1

  -- get the payload

  local payload = nw.getPayload()

  -- find the end of the line

  local num_temp = payload:find("\r\n", current_position, current_position + 4096)

  -- if we found the end of the line

  if num_temp then

  -- we don't want to read the \r

  num_temp = num_temp - 1

  -- read up to the end of the line

  local host = payload:tostring(current_position, num_temp)

  -- make sure the read succeeded

  if host then

  if self.Path then

  url = "http://" .. host .. self.Path

  nw.createMeta(self.keys["url"], url)

  end

  end

  end

end

-- declare what tokens and events we want to match

luaHttpUrl:setCallbacks({

  [nwevents.OnSessionBegin] = luaHttpUrl.sessionBegin,

  ["^GET "] = luaHttpUrl.tokenRequest,

  ["^POST "] = luaHttpUrl.tokenRequest,

  ["^HEAD "] = luaHttpUrl.tokenRequest,

  ["^PUT "] = luaHttpUrl.tokenRequest,

  ["^DELETE "] = luaHttpUrl.tokenRequest,

  ["^TRACE "] = luaHttpUrl.tokenRequest,

  ["^OPTIONS "] = luaHttpUrl.tokenRequest,

  ["^PATCH "] = luaHttpUrl.tokenRequest,

  ["^CONNECT "] = luaHttpUrl.tokenRequest,

  ["^PROPFIND "] = luaHttpUrl.tokenRequest,

  ["^PROPPATCH "] = luaHttpUrl.tokenRequest,

  ["^MKCOL "] = luaHttpUrl.tokenRequest,

  ["^COPY "] = luaHttpUrl.tokenRequest,

  ["^MOVE "] = luaHttpUrl.tokenRequest,

  ["^UNLOCK "] = luaHttpUrl.tokenRequest,

  ["^LOCK "] = luaHttpUrl.tokenRequest,

  ["^SUBSCRIBE "] = luaHttpUrl.tokenRequest,

  ["^POLL "] = luaHttpUrl.tokenRequest,

  ["^Host: "] = luaHttpUrl.tokenhost,

  ["^HOST: "] = luaHttpUrl.tokenhost,

  ["^host: "] = luaHttpUrl.tokenhost,

})

[Anonymous]

Another example, this time using meta callbacks.  I have found that using Lua parsers can help processing of data extracted by Log Decoders.  That's right....Log Decoders.

Lua parsing can work on either a packet or log decoder, however I mostly use meta callbacks for a Log Decoder.

This parser attempts to normalize user accounts to lower case.

local normalizeuser = nw.createParser("Normalize_User", "Make all user accounts lower case")

--[[ The purpose of this parser is to normalize meta from the meta callback into lower

case.  With this normalization to lower-case, analysts can create a feed that might have

missed some values due to capital letters in the meta.  CASE matters.

SomeUser --> someuser

To save space, a comparison is done to determine if this lower-case was seen before or if

new meta needs to be created.  This is to save space.  Thanks DC.

Parser created 2014-03-18 by Chris Ahearn

christopher.ahearn@rsa.com

--]]

normalizeuser:setKeys({

    nwlanguagekey.create("user.dst")

})

function normalizeuser:userMeta(index, user)

    -- localize or initialize table to hold seen email addresses

    local seenUser = self.seenUser or {}

    -- check if this email address has been seen before

    if not seenUser[user] then

        -- nope, now it has

        seenUser[user] = true

        -- copy local table back to global

        self.seenUser = seenUser

        -- Convert user to lower case

        local lower_user = string.lower(user)

        -- *check if this user has been registered before

        if not seenUser[lower_user] then

  nw.createMeta(self.keys["user.dst"], lower_user)

  end

    end

end

function normalizeuser:sessionEnd()

    self.seenUser = nil

end

normalizeuser:setCallbacks({

    [nwlanguagekey.create("user.dst")] = normalizeuser.userMeta,

    [nwevents.OnSessionEnd] = normalizeuser.sessionEnd,

})

[GayathriPulla]

Hi Chirstopher, 

Is there lua parser available for converting epoch time to Human Readable Time/Local Time. We have forwarded logs in Cef format the time is in rt=value, in epoch time format. So, I just want to is there any Lua parser already available or we have to write custom lua , also , is it possible to create the one.