Security Analytics Log Parser 2.1.63.zip

[Anonymous]

***************************************************************************************

THE MOST CURRENT VERSION IS 2.3.99 published 8/1/17.

https://community.rsa.com/community/products/netwitness/blog/2017/08/01/draft-post 

***************************************************************************************

This version 2.1 will now parse 763 events, focusing mainly on the audit events on both 10.3 and 10.4 generated from the SA server.

This was developed and tested on 10.4.0.2 and 10.3.5.

This package includes:

  Broker/Concentrator Custom index

  Custom Table Map

  Installation Instructions 10.3/10.4

  Event Categories spreadsheet

  Variable and metakey info spreadsheet

  ESA alerts 10.3/10.4

  Investgation metakey group file

  SA Log Parser

  Reports 10.3/10.4

Please review the "Variable and Metakey info.xlsx" file to note the new metakeys.

I have changed some of the metakeys around and created new ones to keep the indexes for the metakeys small.

I have also removed some of the keys in the custom index file to prevent overriding the default values of the concentrator/broker index files.

To move from the 2.0 parser to the 2.1, please use the new custom table map and custom index files provided.

Also remove the old reports & ESA alerts and replace them with the new ones.

New parsing capabilities:

Event source monitoring – Events that are generated from the Event Source Monitoring regarding log sources not reporting will be parsed.

Archiver Monitoring – Aggregation Status,Storage Capacity/Time (sorry don't have storage connection)

Query Time – Query times are now parsed

Queue Time – Queued query times are now parsed

Queries – Queries are now parsed, so you can view the queries that users submit.

Reports:

  SA Configuration Changes

  SA Live Content Update

  SA Login Activity

  SA Monitored Systems Archiver

  SA Monitored Systems Event Sources

  SA Query Performance Summary

  SA Service Stops and Starts

  SA User and Role Management

  SA User Query Activity

ESA Alerts

SA System Related Alerts

  Capture has started

  Capture has stopped

  Database configuration issue

  Lockbox Fingerprint has changed

  Monitored Log Source Failure

  RabbitMQ is stopping

  Device has been taken offline

  Device has been taken online

  System has stopped aggregation

SA User Related Alerts

  Initiated stop aggregation

  Initiated start aggregation

  User has created or deleted a group or role

  User has created or deleted a user account

  Has deleted a feed

  Has issued a parser reload

  Has merged application rules

  Has modified device configuration

  Has modified a user account or role

  Has replaced application rules

  Has requested an appliance reboot

  Has restarted service using GUI

  Has started capture

  Has stopped capture

  Has uploaded a log parser

  Has uploaded a feed

Security Analytics Log Parser 2.1.63.zip

[Anonymous]

Thanks!

[MikhailKalishuk]

great job, leonardc! thanks a lot.

[Anonymous]

You are welcome!

[Anonymous]

You are Welcome!

[Anonymous]

Nice!

[Anonymous]

Not a big deal but figured I would mention it for future releases.  In the excel doc Variable and Metakey info the monitored_device is actually linked to device.monitored, not monitored.device. 

Thanks again!

[Anonymous]

Ok, thanks for the heads up.  I will get it corrected.

[RSAAdmin]

Hi I know this is probably a bit silly but I applied the concentrator-custom-index.xml to my packet concentrator and it broke it. This in hindsight was probably a bit silly, but maybe worth mentioning in the nodes that this applies to the log concentrator only.

Correction:

The issue was that  I already had the meta ip.srcport defined and had it as a different type. - Text rather than Uint16

[Anonymous]

Thanks for the update on what the issue was.

[Anonymous]

Hi leonardc,

I would like to thank you for your work.

I still wonder, how is possible, that parser for SA must be created by custommers. It's shame...

[Anonymous]

This was created by RSA PS not a customer.

[Anonymous]

You are welcome, and Sean is correct, as I work for RSA in Security Analytics Professional Services.

Thanks...

[RSAAdmin]

I noticed that this parser doesnt support debug logs, are there plans to incorporate these logs into this parser?

[Anonymous]

I work on this in my spare time and I can only work with samples I have or can generate on my test system.  If you can send me some sanitized samples and provide me some info on how they were generated that would be great.  I will add them in the next release.  Just send me a private message and we can work out the sample delivery details.

[RSAAdmin]

Is the zip link broken? I click download it got me to the sign on page and I click EMC single signon but it didn't work 😞

[MikhailKalishuk]

No, it's correct. By my expirience, you just have to login there with your EMC Community Account (that was used to login here, not EMC SSO).

[RahulLohar]

Hi,

We are running with SA release 10.4.1 and need to know whether the new release of SA log parser going to be launched soon or should I deploy the current version i.e. 2.1.63. If 10.4.1 version's events can be parsed using 2.1.63 version then only its better to go ahead with this.

Apart from this, I am eager to know whether it logs the centOS platform events as well like service start/stop, management activities done through CLI not through console.

[RSAAdmin]

Hi I am using the parser with 10.4.1 and it is working fine.

[Anonymous]

The current version works with 10.4.1.  The CentOS events would be parsed by the rhlinux parser.

Most SA service starts and stops from the CLI generate log events and are parsed, they just lack some additional events that provide context around that restart/start/stop.  The GUI service restart shows that it was user initiated and gives the user an opportunity to provide a reason for the restart.  Java based services like jetty, report engine, esa have their own logs and do not send to /var/log/messages, so these events are not seen nor parsed.  However some of these services now have audit features and have not yet been fully explored to see what events come from enabling the auditing.

There is no ETA for a newer version of the SA parser at this time, as I have said before this is something I do in my spare time.  Since it summer and it is daylight longer, that reduces my time on the computer and increases my other outside activities.

[Anonymous]

Hello Guys,

I have a doubt, last week I applied an SecurityAnalytics 2.0 parser on my SA. Now here I have one doubt.

I have two sites, i.e., Prod & DR with different SA & Log Decoder appliances, so I had applied the parser first on the DR Log Decoder and also applied the Broker & Concentrator (DR) Index file, as we use 1 broker for investigation purspose, to view all the data from every concentrator.

Now do I need to apply the same parser on the Prod Log Decoder and do the same on Concentrator also, if I"ll do it did I get the "rsasecurityanalytics" device type twice or only once.

Kindly suggest.

Regards,

Deepanshu Sood.

[Anonymous]

Deepanshu,

In a multi-site (multiple log decoders) it is a good idea to keep the parsers/table maps consistent on all log decoders.  The same goes with the indexes on the Concentrators/Brokers.  You will only see one device type in Investigation but you will see different log decoder sources for the device type rsasecurityanalytics.

Typically in a multiple site configuration the parser works best when the Security Analytics logs are sent to different log decoders.

For example you have two sites, Site1 and Site 2.  The SA Devices should be configured to as follows:

     Site1 SA Devices should send logs to the Site2 Log Decoder.

     Site2 SA Devices should send logs to the Site1 Log Decoder.

This way if any of the systems you are logging from have a problem in Site1 the events will be captured by Site2, processed and made available to the ESA (assuming you have one for each site or one that is looking at both sites) for alerting.  The same goes for Site2 sending to Site1.

leonardc

[Anonymous]

Hi #leonardc,

Hope you are doing well.

Well this time I am facing some issues regarding the parser for RSA Security Analytics.

All the SA components are going under multiple device type, like mentioned below

rhlinux (>1,000 - 16%)  - unknown (172)  - bigip (21)

And also if I opened and checked the logs under these meta keys, the logs are not clear so I am not able to understand the event like which activity it's reffering too. for more understanding kindly refer to the below mentioned logs.

Aug 20 03:01:01 DR-SA-LDEC01 anacron[25690]: Anacron started on 2015-08-20

Aug 20 03:17:01 DR-SA-LDEC01 anacron[25690]: Job `cron.daily' started

Aug 20 03:17:08 DR-SA-LDEC01 anacron[25690]: Job `cron.daily' terminated

Aug 20 03:17:08 DR-SA-LDEC01 anacron[25690]: Normal exit (1 job run)

Aug 19 11:53:56 DR-SA-LDEC01 nw[2902]: [WindowsStringsManager] [warning] unable to find operating system mapping for Windows 8.1 Enterprise; using default mapping (warning suppressed 188 times)

Aug 19 11:10:01 DR-SA-LDEC01 CROND[6810]: (root) CMD (/usr/lib64/sa/sa1 1 1)

So all these logs are very unclear to understand that what does this mean, and also the categoriztaion of this not going in the right fileds.

Pls suggest. Thanks a lot.

Regards,

Deepanshu Sood.

[Anonymous]

Deepanshu,

It is normal for the SA devices to show up as multiple device types as there are Linux CentOS logs along with other application logs, including SA application logs.  The three common device types that the SA systems should have in Investigation is "rhlinux", "rsasecurityanalytics" and "unknown".  The SA parser parses a subset of the SA application logs, as I don't have samples for every possible log that SA can generate, so there will be the "unknown" device type as well.  A general rule is that most SA application logs in /var/log/messages have the "nw[#####}" after the date and hostname.  The remaining logs are generally Linux CentOS operating system log entries, crond and anacron for example are most likely Linux scheduled jobs running to automate the management of the system.  There can be other entries but the most common is what I have described.  It is difficult to keep the "unknown" device type from the system as log entries get added or modified with updates and patches.

When the parser has no exact match for a log event, the log decoder finds the closest match using the parsers that it has enabled and when there is no match at all, it gets classified as "unknown".  You will find that there some odd matches like "crossbeam" is a common one I find that matches on some SA logs.  Unfortunately it does not give you any meta when this happens and basically is not much different than getting put in the "unknown" device type.

In short, if the parser has a good match for the log event, you get meta, if you have a bad match (incorrect device type) or the "unknown" device type you don't get any useful meta that you can use in investigation.

To resolve "unknown" log events, open a ticket with support and ask about how to get new "unknown" log events added to the parsers in Live.  You will need to be able to provide information regarding what generated the log event and the reason for the event.  You can use google to help with determining what generated the log event and what it means.  This only applies to parsers that come from Live, not custom parsers that were developed outside of the RSA Live Content, like this parser.  In that case you would need to contact the author of the parser and see if they can update the parser, or you also have the option to update the parser yourself if you are familiar with editing parsers.  In all cases you will need to know where the log event came from and the context/reason for this event to occur so that it can be properly classified and the values properly assigned to metakeys.

Thanks...

Leonardc

[Anonymous]

Hi,

force the device type in the devicetbl.xml file on the logdecoder prior to update to 10.5.0,1.

Beside nokiaipso or infobloxnios, the unknown messages for the logdecoder and the concentrator are now also dispatched as device.type = securityanalytics and rsa_security_analytics_monitored_devices. (BTM: never heard about them).

I have now 6 device type for the same ip address.

Kind regards

Gab.

[Anonymous]

Use with caution, forcing the device ip gives it the desired device type, however, forcing the device type does not mean the event will be parsed properly.  If the parser does not have an entry for the event, there will be no meta assigned to it.  It's still the same "unknown" event with a different name and no meta.

[TravisGrosch]

Has anyone used this on 10.5?

[RSAAdmin]

Yes, it works fine.

I use it in parallel with the CEF Global Audit Logging feature in 10.5 too.

Global Audit Logging Overview - RSA Security Analytics Documentation

[Anonymous]

As David said, it still parses events like queries, ESA rule creation/Deletions, ESA rule sync, database, index activites, some user logon/off activity, subscription updates and a few others.

Most of the ESA alerts have been replaced by the health and wellnes policies.

In short it still provides some useful information.  I have not had the time to fully evaluate it on 10.5.

[RSAAdmin]

Hi another customer has noticed that the auditing has changed in 10.5 and so not all messages that were produced by a 10.4 system are generated in the same way in a 10.5 System.

Examples of the messages are those produced are those beginning SA-AUDIT.

Dont know if this is just a 10.5 config issue or something else?

[Anonymous]

Hello,

Well this parser is helping me to so much to identify those event sources which are having some problem in logging, like SQL/datbases servers.

In the same way this parser is detecting those database machines which are not logging to SA, can we check the other types of collection method machines which are not logging or throwing some error messages?

Is it possible to understand with the help of this parser, so that I can create a report by understand which machines are not logging.

Pls advise.

Regards,

Deepanshu Sood.

[Anonymous]

Hello,

Looks like the parser doesn't work on 10.5.1, they changed the logging from

nw[<hinfo>]

to

nwBroker[<hinfo>]

nwLogDecoder[<hinfo>]

nwConcentrator[<hinfo>]

The logs turned to unknowns in an instant

[RSAAdmin]

No Problem!

You can get round this by editing the parser

Do a search and replace on the parser, replacing

nw[<hinfo>]:

with

Nw<hservice>[<hinfo>]:

I posted a modified one here

[Anonymous]

UPDATE:

I am currently working on a 2.3.x version that will address the new header on 10.5.1 as well as add new events.  No ETA for it's release at the moment.  Just wanted to let you know that there is an update in the works.

Thanks....

[HuanZhou]

any update for 10.6.1? 

[Anonymous]

I am doing some testing with 10.6.1.  Is there anything in particular you are wanting to parse in 10.6.1 logs?

[HuanZhou]

Was creating some reports for SA 10.6.1 Event Source Monitoring Alerts, also VLC errors reports (like odbc/windows errors), will let you know if any logs not able to parse.

Also now we can set the syslog forwarding from device - system - Hosts Tasks - Set Syslog Forwarding to forward the nw related logs. 

[huanzhou1]

Tested, it show unknown for the VLC logs. 

[Anonymous]

THE MOST CURRENT VERSION IS 2.3.99, which works with 10.6.x.