Event Source Monitoring - Lua

[TomiReiman]

Hello,

The Security Analytics server generates syslog messages of the following kind if you setup event source monitoring and forward the notification via Syslog:

Oct 10 12:53:16 <sa-server> The Security Analytics systems have not received log events from the following devices within  <time>: Source Type=<device.type> Source Host=<device.ip> Device Name=<did>, Source Type=<device.type> Source Host=<device.ip> Device Name=<did>, ...

The list can continue on quite a bit per one log message if the amount of devices per that device type is large. I suspect you could circumvent this by creating a monitoring entry per a device.type and device.ip pair, but that would be too laborious in most of the cases. I could not find a way to parse the message with the regular, enVision-style XML parsers due to the variable length of the message. Is it somehow possible?

Also, I tried creating a LUA parser for this, but as I have zero experience with LUA before this, I could not get it working: it just isn't generating the meta I would like and to be quite honest I am not at all surprised. Attached here is the LUA parser I ended up with. The logic is quite crude also, but the main issue is that it does not work.

Has anyone got a working parser for these messages generated by the SA server? I did not find a message definition for it in the Security Analytics parser 2.0.

I would also like to know if there is any documentation for the functions provided by the architecture. The functions starting with "nw" in this file must be defined somewhere. For example, how could I check the device.ip in the session message, or any other parameters that could help me with checking if I should run through the LUA script in the first place: e.g. do not parse the payload if this message did not originate from the SA server.

[Anonymous]

Hi Tom

Lua parsers cannot be used to parse log messages - that is done by the XML based parser files. Lua parsers can be used for additional processing of meta that is generated by the XML parsers.

As for ingesting these messages, have you tried the parser posted to the community here:

[RSAAdmin]

Resurrecting - my apologies.

These days lua actually can be used to write custom log parsers.  The biggest issue with the OP's parser is that it has no callbacks - nothing that says, "when you see this, do something".

[Anonymous]

Just an FYI, This event is now parsed in the SA log parser 2.1.63.

https://community.emc.com/docs/DOC-42795

[RSAAdmin]

That's some unexpected news, keep us updated, Bill! I will try to do my next log parser in LUA and see how it goes. ESI is nice (support for auto lua generation there and feeding it with traffic flow+log flow is a dream ), but having one approach for custom logs and traffic processing is quite convenient.

BTW I think the same parser in LUA will be like 10 times larger then in usual DML xml for a simple log source and for complex one it will be vice versa due to lua function usage (as the same fields will occur in different log types = less patterns), anyways having a choice is great!

[Anonymous]

I have some experience writing parsers with Lua and YES, they can work on both packet and log decoders and YES, they can be used to parse raw log messages.  It isn't the best way, but it can be done.  Lua offers the flexibility of scripted logic which may be necessary depending on the type of logs being parsed.

Like the packet side, you would still create a token match and then do your logic from there.  If there are log samples, we can build one and then maybe even post it back here to the community to share.

Chris

[RSAAdmin]

Thanks for the info, Chris, some questions arisen:

Do all the versions of SA support LUA for logs?

I know that LUA processing is faster then Flex for packets, how about LUA vs DML XML speed on SA?

Are there any plans to migrate logs to LUA (like packets)?

I have a client using log hybrid for zillions of logs per day, and during peaks the decoder lags a lot so I wonder if migrating the most occurring events would fix that.

As I said, I'll try to do the next requested custom parser in LUA, and if it's not ugly I'll share it

[RSAAdmin]

I'll take a stab at some of these:

1) It's been possible for a while.  I'm don't know the exact version number, though.

2)  No testing has been done.  I suspect that XML log parsing is still faster.  Flex was a different animal entirely - it was essentially a scripting language implemented in XML.  XML log parsers aren't really scripts so much as pattern matching.

3)  No.

One thing to note is that a log message being matched by a lua parser doesn't prevent it from being matched by the log parser.  This is good if you only want to extract certain portions with lua while letting the XML parser do its thing.  But if you really want to parse the entire message with a lua parser, you'll want to disable the relevant log parser.

[Anonymous]

Lua is enabled on the packet and log decoders.  I believe Lua is installed on the other core appliances, but it wouldn't really be used to my knowledge.  I don't have any specs on Lua vs XML speed testing, however Lua does offer scripting that the XML does not.  In many of the cases where I have used Lua for logs has been after I exhausted attempts to using the XML for log parsing.  There are no plans to migrate the Log parsers to Lua.

[Anonymous]

Adding an example of how I used Lua to parse out certain data from a log message.  The goal was to extract each email address in the TO field.  However, the number of email addresses was going to be unknown.