Indexing Log Source Event Time and Lua Parser Callbacks?

[KEVINDIENST]

Wondering if anyone on here has setup a method to index and/or parser the native log source timestamps?

We have problems in that some log sources may get backed up when our log decoders are either down or we do a long upgrade and we haven't found a solution for our analysts when the SA ingestion time and original log source time don't align. It is very frustrating for them when they can't find an event and are under stress to get incidents resolved. 

It makes it worse that we have log sources logging in different time zones and owners not being super excited about adjusting everything into UTC and verifying they have accurate NTP settings. 

I know the log decoders index the ingestion time (only thing indexed on the log decoders) and that attempting to index the raw event time from the source will have a huge performance hit, in addition to adding the burden of having to update parsers for all the log sources we support since they may all format time a little differently. 

I spoke with some RSA resources about doing this and they mentioned potentially limiting the indexing to a few critical/problematic log source types and using Lua to parse the time into small buckets (i.e. into minutes or 5-10 minute buckets instead of down to the millisecond for instance). 

Anybody else have ideas or can extrapolate on using Lua to accomplish this?

Thank you! XDudADtNBir2yWTvsAvML3S8A4BmZKPAABc5iGXZy0M=N6EPgVyqjIeX4VQaO87WEl41adMR7kmOen9eAKHNIG0=Lb4PID2jPLBSBnAF2o6daHw6uLL4Qn1o12RToTXYDMM=

[Anonymous]

I haven't looked at indexing the raw event time, but some of that, when available, is parsed into 'event.time.str'.  However, that meta key is formatted as a string.  Parsing that may be possible, but am wondering how it would be queried.

I am more curious about the first part of the problem description in log decoders being down.  Would it be possible to have all the logs flow into the log collector first?  This way, you could redirect the flow to an available log decoder if work is being done creating an outage. 

[AbhishekMahadik]

Hi Team,

I want to extract the email id from event.desc meta key so I wrote the below lua code.

Sample logs -

%CASB_log-4: Investigate : severity=informational^^service=Elastica^^activity_type=Update^^object_type=User^^ldap_memberof= changed from [&quot^^created_timestamp=2018-06-08T14:12:56^^message=Modified user: 'abc@xyz.com'^^_id=uCa2cQl9dw^^user_name=System^^inserted_timestamp=2018-06-08T14:13:45^^user=system

event.desc - Modified user: 'abc@xyz.com'

local copymeta = nw.createParser("Copyemail", "")

-- These are the meta keys that we will write meta into
copymeta:setKeys({
nwlanguagekey.create("app_name",nwtypes.Text),
})

function copymeta:sessionBegin()
event.desc = nil
end

-- This is our function. What we want to do when we match a token...
-- email meta callback.
function copymeta:myMeta(index, event.desc)
if event.desc then
local email1 = string.match(event.desc, "(?<=\')(.*?)(?=\')")
if email1 then
nw.createMeta(self.keys["app_name"], email1)
end
end
end

function copymeta:sessionEnd()
end

copymeta:setCallbacks({
[nwevents.OnSessionBegin] = copymeta.sessionBegin,
[nwlanguagekey.create("event.desc",nwtypes.Text)] = copymeta.myMeta, -- this is the meta callback key
[nwevents.OnSessionEnd] = copymeta.sessionEnd,
})

but it is giving me below error.

Nov 5 11:59:47 xxxhyb01 NwLogDecoder[41674]: [Lua] [failure] Throw in function static void nw::LuaPackage::require(lua_State*, const string&)Dynamic exception type: boost::exception_detail::clone_impl<nw::LuaError>std::exception::what: LUA_ERRRUN: [string "email.lua"]:12: ')' expected near '.'[boost::errinfo_at_line_*] = 446

please suggest.

[Anonymous]

Can you email or post the actual parser? I think something is getting lost in the web formatting.

That error message is saying that there is a missing ‘)’ somewhere around line 12 of the parser.

Chris Ahearn

RSA | Principal Consultant | Incident Response

[WilliamMotley1]

Using a dot ('.') in a variable name is a way of referencing table indices.  So "event.desc" means "table named event, index named desc".  Change all occurrences of event.desc to something like event_desc.

Edit:  not quite all occurrences.  Leave the callback as is, because the key is literally "event.desc".

[AbhishekMahadik]

Hi William,

event.desc is currently capturing the string value after a parsing.

The objective here is to capture the email id from the event.desc to use it for the correlation.

I am new to Lua coding so can you please guide how to changes it.

Regards

Abhishek 

[AbhishekMahadik]

Hi Christopher,

I have shared the complete Lua parser.

This parser only captures the email from the event.desc meta key.

As William mentioned correctly, I am seeing this issue since declaration variable with a dot.

Is there any way to define the local variable to perform the conversion?

for example -  local event1 = string event.desc()

[Anonymous]

I think you just need to change some of the variable names in your function.  Furthermore, I don't think you need the tokens or functions for sessionBegin or sessionEnd.  The one that we really need is copymeta.myMeta.

The function variables then just need to be modified.

function copymeta:myMeta(index, event.desc)
   if event.desc then
   local email1 = string.match(event.desc, "(?<=\')(.*?)(?=\')")
     if email1 then
      nw.createMeta(self.keys["app_name"], email1)
      end
   end
end

Instead of using 'event.desc' in your function, simply change the variable name to 'meta' or some other variable name.  The meta value is already part of your token match.  In this case, the meta callback of the meta key 'event.desc'.  

function copymeta:myMeta(index, meta)
   if meta then
   local email1 = string.match(meta, "(?<=\')(.*?)(?=\')")
     if email1 then
      nw.createMeta(self.keys["app_name"], email1)
      end
   end
end

You could even further refine your parser by adding additional meta callbacks so that the functions only run against a specific device.type.  In that instance, I would keep the sessionBegin token and functions and use global variables to ensure the parser was operating on the correct data.

[AbhishekMahadik]

Hi Chris,

Thanks for the suggestion.

I understand that we can replace the event.desc and use some other variable to pass the values captured in the event.desc meta key.

To do so, I need to declare the local variable as meta to capture the values coming to event.desc filed after log parsing at the decoder level.

can you please help me with the syntax of that variable?

[Anonymous]

Let's take this offline and email me directly. Once we work through this, we can post the solution.

christopher.ahearn@rsa.com<mailto:christopher.ahearn@rsa.com>

Chris

Sent from my mobile device