LUA Parser To Generate Meta based on Time Within a message

[DavidWaugh1]

I had a customer with an interesting use case.

Basically they were receiving Logs containing time stamps and they wanted to generate an alert based on this timestamp. The logs were from CyberArk and looked as follows:

1 2016-02-15T11:17:35Z SERVER1 %CYBERARK: MessageID="109";Version="9.10.0000";Message="Get File Request";Issuer="USER1";Station="127.0.0.1";File="Root\MASTER1 CA_TRL";Safe="Safe1";Location="";Category="";RequestId="2";Reason="Server patching. RFC 123456 [From 15/02/2016 08:00:00 to 15/02/2016 17:00:00 (GMT+1.00) multiple operations]";Severity="Info";GatewayStation="127.0.0.1";TicketID="";PolicyID="Standard";UserName="MASTER1 CA_TRL";LogonDomain="";Address="";CPMStatus="";Port="";Database="";DeviceType="Operating System";ExtraDetails=""

When the log was parsed meta would be generated into the following meta keys:

event.desc ='Server patching. RFC 123456 [From 15/02/2016 08:00:00 to 15/02/2016 17:00:00 (GMT+1.00) multiple operations]'

hdatetime ='2016-02-15T11:17:35Z'

What the customer wanted to do was generate an alert when the hdatetime value was greater than the time after the to value in the event description above.

In the following parser I always assume that the hdatetime field is in UTC (indicated by Z at the end of the timestamp). I also assume that the time in the Event Description is always GMT+1 = UTC plus one hour.

In order to compare times, I convert the time strings to Unix Epoch time. Normally LUA os functions could be used for this, but as these are restricted in Security Analytics, I had to borrow an algorithm from someone else.

Here is the LUA Parser

local lua_CyberArkTime= nw.createParser("CyberArkTime", "Looks to See if the time in a CyberArk message has passed")

--[[

    DESCRIPTION

    Takes a log message from a CyberArk where the Event Description contains a FROM and a TO time such as

  Server patching. RFC 123456 [From 15/02/2016 08:00:00 to 15/02/2016 17:00:00 (GMT+1.00) multiple operations]

    We compare if the TO Time has already passed and if it has write Meta into the Alert Field

  We assume that the Time in the Event Description is always GMT+1

  We assume that the Time in the Header is always UTC

    AUTHOR

    david.waugh2@rsa.com

    VERSION

       1.0 Inital Parser Request

    DEPENDENCIES

     Make Sure that in your /etc/netwitness/ng/envision/etc/table-map-custom.xml that      the hdatetime key is set to either None or Transient

        <mapping envisionName="hdatetime" nwName="hdatetime" flags="None"      format="Text"/>

--]]

function days_from_epoch(y, m, d)

-- This algorithm came from

-- http://howardhinnant.github.io/date_algorithms.html

--

    if m <= 2 then

      y = y - 1

      m = m + 9

    else

      m = m - 3

    end

    local era = math.floor(y/400)

    local yoe = y - era * 400                                           -- [0, 399]

    local doy = math.modf((153*m + 2)/5) + d-1                          -- [0, 365]

    local doe = yoe * 365 + math.modf(yoe/4) - math.modf(yoe/100) + doy -- [0, 146096]

    return era * 146097 + doe - 719468

end

function epochtime(year,month,day,hour,minute,second)

  local days=days_from_epoch(year, month, day)

  return (days*86400 + hour*60*60 +minute*60 + second)

end

-- Write meta into the following meta key(s)

lua_CyberArkTime:setKeys({

  nwlanguagekey.create("alert",nwtypes.Text),

})

function lua_CyberArkTime:sessionBegin()

  HeaderTime = nil

  EventTime = nil

  devicetype = nil

end

function lua_CyberArkTime:CheckDevice(index,dtype)

  if dtype == 'cyberark' then

  devicetype = dtype

  end

end

function lua_CyberArkTime:getEventTime(index,myEventTime)

  if devicetype then

  EventTime=myEventTime

  --nw.logInfo("EventTime ")

  --nw.logInfo(EventTime)

  end

end

function lua_CyberArkTime:getHeaderTime(index,myHeaderTime)

  if (devicetype) then

  HeaderTime=myHeaderTime

  --nw.logInfo("HeaderTime ")

  --nw.logInfo(HeaderTime)

  end

end

function lua_CyberArkTime:sessionEnd()

  if devicetype then

  --nw.logInfo("SessionEnd:EventTime ")

  --nw.logInfo(EventTime)

  --nw.logInfo("SessionEnd:HeaderTime ")

  --nw.logInfo(HeaderTime)

  if (EventTime) then

       if (HeaderTime) then

            ---Do All The Work in Here

            local FromDate,ToDate =string.match(EventTime,"From (%d%d/%d%d/%d%d%d%d %d%d:%d%d:%d%d) to (%d%d/%d%d/%d%d%d%d %d%d:%d%d:%d%d) (.*)")

            local EventDay,EventMonth,EventYear, EventHour,EventMinute,EventSecond=string.match(ToDate,"(%d+)/(%d+)/(%d+) (%d+):(%d+):(%d+)")

  local EventEpochTime=-3600+epochtime(tonumber(EventYear),tonumber(EventMonth),tonumber(EventDay),tonumber(EventHour),tonumber(EventMinute),tonumber(EventSecond))

  --nw.logInfo("EventEpochTime " .. EventEpochTime)

  local HeaderYear,HeaderMonth,HeaderDay, HeaderHour,HeaderMinute,HeaderSecond= string.match(HeaderTime,"(%d+)%-(%d+)%-(%d+)T(%d+):(%d+):(%d+)Z")

  local HeaderEpochTime =epochtime(tonumber(HeaderYear),tonumber(HeaderMonth),tonumber(HeaderDay),tonumber(HeaderHour),tonumber(HeaderMinute),tonumber(HeaderSecond))

  --nw.logInfo("HeaderEpochTime " .. HeaderEpochTime)

  if HeaderEpochTime > EventEpochTime then

  --nw.logInfo("CyberArk: Time Expired: ")

  nw.createMeta(self.keys["alert"], "CyberArk: Time Expired")

  end

  end

  end

  end

end

-- declare what tokens and events we want to match

lua_CyberArkTime:setCallbacks({

    [nwevents.OnSessionBegin] = lua_CyberArkTime.sessionBegin,

    [nwlanguagekey.create("device.type", nwtypes.Text)] = lua_CyberArkTime.CheckDevice,

    [nwlanguagekey.create("event.desc", nwtypes.Text)] = lua_CyberArkTime.getEventTime,

    [nwlanguagekey.create("hdatetime", nwtypes.Text)] = lua_CyberArkTime.getHeaderTime,

    [nwevents.OnSessionEnd] = lua_CyberArkTime.sessionEnd,      

})

[DavidWaugh1]

Some great feedback from our parser expert in house on how the parser can be improved:

This is great.  Just a couple caveats, though:

(a) Callbacks for meta from a log parser occur in no guaranteed order.  For a meta from packet parsers you can rely on callbacks occurring as meta is registered and in the order it was registered, but for meta from a log parser you can't.

(b) End events such as OnSessionEnd aren't guaranteed to match.  Most of the time they will, but sometimes they won't.

So I'd suggest something like this instead:

function lua_CyberArkTime:register()

  if deviceType and EventTime and HeaderTime then

    ... do stuff ...

  end

end

function lua_CyberArkTime:CheckDevice(index, dtype)

  if dtype == 'cyberark' then

    devicetype = dtype

    self:register()

  end

end

function lua_CyberArkTime:getEventTime(index, myEventTime)

  EventTime = myEventTime

  self:register()

end

function getHeaderTime(index, myHeaderTime)

  HeaderTime = myHeaderTime

  self:register()

end

The register() function is called for every callback.  When it has all the values it needs, it does the work.  That removes any reliance on order of callbacks, and on session end.