RSA Netwitness Suite Log Parser 2.3.99

[Anonymous]

Overview

This version will now parse over 1,400 events from the devices, however the parser does not parse audit events that are generated in the "Administration-->Security" user interface.  Those events are handled by the Global Audit, Global Notification settings and parsed by the CEF parser.  However, if you made modifications to the "Security" settings on the individual device, that event will be parsed by this parser.

This version was developed and tested on 10.6.2.0 using available log samples from 10.4.x thru 10.6.2.0.

Improvements

New Headers have been added to accommodate the log format change in 10.5.1 and above.

Logs from the Virtual Log Collector are now parsed, particularly Windows Collection Errors.

Error/Failure Logs are consolidated under the Event Category Name of "System.Errors"

Puppet Logs are parsed

Collectd Logs are parsed

Added "maxValues" kb 00031300 modification

Custom Index reduction in size and maxValues adjusted accordingly

Overall cleanup of some variable/index clutter

Improved accuracy for parsing of Query and Queue Times

Duration added for Query Times, they are now converted to seconds under the "duration.time" metakey

Contents

This package includes:

   Custom Log parser

   Custom Index for Concentrator*

   Custom Table Map*
   Event Categories Spreadsheet

*I have revised the custom index and table map to reflect the new changes in the default settings for 10.6.2.  If you are using a prior version to 10.6, you may need to add some additional index keys to the custom index.

Parser Content

Content, such as reports and dashboards, written by me for this parser will be published separately and links will be added here.  Currently content for Index operations, queries, cancelled queries, system errors, configuration changes, security changes, service restarts, and content updates for feeds/parsers are being tested on an enterprise system at the time of this writing.  These will start appearing in the next few days.

https://community.rsa.com/community/products/netwitness/blog/2017/08/03/report-valuemax-has-been-reached 

Installation

Log Decoder

Remove the prior version of the parser

1. SSH into each log decoder as "root" that has the prior version.
2. Remove the old parser directory
   rm -r /etc/netwitness/ng/envision/etc/devices/rsasecurityanalytics/
   You should see the prompts like below:
   [root@logdecoder60 SA_Logs]# rm -r /etc/netwitness/ng/envision/etc/devices/rsasecurityanalytics/
   rm: descend into directory `/etc/netwitness/ng/envision/etc/devices/rsasecurityanalytics'? y
   rm: remove regular file `/etc/netwitness/ng/envision/etc/devices/rsasecurityanalytics/rsasecurityanalytics.ini'? y
   rm: remove regular file `/etc/netwitness/ng/envision/etc/devices/rsasecurityanalytics/rsasecurityanalyticsmsg.xml'? y
   rm: remove directory `/etc/netwitness/ng/envision/etc/devices/rsasecurityanalytics'? y

Download and unzip parser

1. Download the parser file "rsasecurityanalytics_2.3.99.zip" from the bottom of this page.
2. Unzip the file using Winzip, or 7zip.
   The unzipped parser file name will be "rsasecurityanalytics.envision"

Upload the parser on the Log Decoder

1. Login to the Web Interface as "admin" or user who is a member of the "Administrators" Role.
2. Choose "Administration-->Services" from the navigation menu in the upper left corner of the screen.
3. Locate the Log Decoder and click on the gear icon, located at the far right of the screen.
4. Hover over "View", then click "Config".
5. Click on the "Parsers" Tab.
6. Click on the "Upload" icon in the upper left portion of the window.
7. Click on the "+" in the upper left of the "Upload Parsers" dialog box.
8. Navigate to the folder where the "rsasecurityanalytics.envision" is located and select it.  Click "Open"
9. Click on "Upload"
10. Click on the "X" in the upper right corner of the dialog box or click "Cancel"

Remove prior version custom table map entries

1. On the same screen as above, Click on the "Files" Tab
2. On the left side of the screen click on the dropdown and select "table-map-custom.xml".
3. Locate the section related to the custom table entries for the log parser typically labelled
   RSA Security Analytics Log Parser Revision 2.1.63 xx/xx/xx
4. Remove that section.
5. Replace with new table map entries from the table-map-custom.xml file.
6. Click "Apply"

Load the new log parser and custom table map.

1. On the same screen as above, click on "Config" just above the "App Rules" Tab.
2. Click on "System"
3. Click on "Stop Capture" at the top left of the screen.
4. Wait for capture to stop.
5. Click on "Shutdown Service" at the top center of the screen.
6. On the "Confirm Shutdown" dialog, type "RSA Security Analytics Parser update"
7. Click "OK"

Concentrator

Update The Concentrator Custom Index

1. Login to the Web Interface as "admin" or user who is a member of the "Administrators" Role.
2. Choose "Administration-->Services" from the navigation menu in the upper left corner of the screen.
3. Locate the Concentrator and click on the gear icon, located at the far right of the screen.
4. Hover over "View", then click "Config".
5. Click on the "Files" Tab
6. On the left side of the screen click on the dropdown and select "index-concentrator-custom.xml".
7. Locate the section related to the custom table entries for the log parser typically labelled
   RSA Security Analytics Log Parser Revision 2.1.63 xx/xx/xx
8. Remove that section.
9. Replace with new custom index entries from the index-concentrator-custom.xml file.
10. Click "Apply"

Load The New Custom Index.

1. On the same screen as above, click on "Config" just above the "Correlation Rules" Tab.
2. Click on "System"
3. Click on "Stop Aggregation" at the top left of the screen.
4. Wait for aggregation to stop.
5. Click on "Shutdown Service" at the top center of the screen.
6. On the "Confirm Shutdown" dialog, type "RSA Security Analytics Parser update"
7. Click "OK"

ALL Appliances

Configure Rsyslog to Forward Logs

1. SSH into each NetWitness Appliance.
2. Modify the /etc/rsyslog.conf file.  
   vi /etc/rsyslog.conf
3. Press the letter "i" or the "Insert" key.  You should see "-- INSERT --" at the bottom left of your screen.
4. Scroll to the bottom of the file and look for the following line:
   #*.* @@remote-host:514
5. Remove the "#" and change "remote-host" to the destination Log Decoder or Virtual Log Collector (VLC).
   *.* @@<Log Decoder or VLC IP Address Here>:514
6. Press the  "ESC" key
7. You should see a colon ":" in the lower left of the screen.
8. Save the file by typing ":wq"
   :wq
9. Restart the Rsyslog service.
   service rsyslog restart
10. Rsyslog is now forwarding logs to the Log Decoder or VLC.

rsasecurityanalytics_2.3.99.zip

[JoeGumke]

thanks leonard.

How do users normalize the JSON logs that the appliances send back into itself out of the box?

[Anonymous]

Joseph,

You are welcome.  I don't know of a way to normalize the internally ingested JSON logs off the top of my head.  Can you post or send me a sample log so that I can look further into it?

Thanks...

Leonard 

[someone]

Great article, thanks Leonard.

I suggest to RSA Content team to put some effort and manage the own product parsers and not let customers rely on a PS engineer with spare time or having to make their own parsers. It's ironic for a SIEM product.

Official own product parsers never supported the latest version of the product, making them pointless. Lastly, this should be added on Live to make the life of the user easier.

[JoeGumke]

hey leonard,

I see that your custom map files add quite a bit of content. How do users prevent duplicated events from being added to their map entries?

[Anonymous]

Joseph,

Great question!  There is some due diligence expected of the system content admin to check for duplicates in the custom table maps and custom index files.  If there are duplicates the last one to load is the one that will be used by the system.

I know I have a fair amount of content in the system I work with and every time something needs to be added to those custom files I take the time to check for duplicates.  The same also goes for application rules.  I recently reviewed several application rules on a decoder and found multiple rules doing the same thing but under different names.  I would highly advise employing some sort of content management process/system.  What I use now is basically versioned folders with the "highly commented" custom files inside of it.  Each modification gets a new folder with a new version number along with the matching comments in the custom files.

Thanks...

Leonard

[JoeGumke]

thanks leonard.

When will RSA implement this feature? RSA can't expect admins to deploy their own content management system, when the SIEM should already have one (blindly managing/deploying content as it is). We need RSA to implement the ability for the tool to already check for duplicated records. We have several hundred records and several admins working our system, and it would be infeasible for us to manually go through and check it this way.

[RonaldRoskens]

Is there a new version or documentation for this that works under 11.2?

[Anonymous]

This version works on the 11.x platform.  Of course the navigation in the UI is slightly different for installation.  I have not had time to update the page to reflect it yet.