This website uses cookies. By clicking Accept, you consent to the use of cookies. Click Here to learn more about how we use cookies.
Accept
Reject
Register Now
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
NetWitness Community Blog
Subscribe to the official NetWitness Community blog for information about new product features, industry insights, best practices, and more.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

NetWitness Retention Script: Reporting The Numbers

LeonardC
Trusted Contributor Trusted Contributor
Trusted Contributor
‎2021-07-07 06:00 PM

Overview

This article covers how to ingest the log output from the retention script you can download from NetWitness Retention Script: Understanding the Numbers. Below are the required steps and modifications you will need to perform on the NetWitness Platform to properly meta and report on the output from the retention script.

Zip File Contents*

*Download the zip file attached at the end of the article.

Concentrator

Custom Index Entries.txt - Concentrator custom index entries

Log Decoder

cef.envision - Parser custom xml file, you can upload via the "Parser" tab on the Log Decoder Config screen.

cef-custom.xml - Actual custom xml file you can upload directly to the cef directory.

cef-customizations-live-package.zip - RSA Live deployment package used to deploy the cef-custom.xml file

log decoder custom index entries.txt - Custom Log Decoder Custom entries that will be copy/pasted into Log Decoder custom index file.

Log Decoder Table Map entries.txt - Custom Log Decoder Table Map entries to be copy/pasted into Log Decoder custom table map file.

Reports

Retention.zip - Report Engine rules, reports, lists to be imported into the Reporting Engine.

Investigate

RetentionMetagroup.jsn - Metagroup file to be imported into Investigation.

Prerequisites

  • NetWitness Retention Script must be installed and running on a daily schedule prior to any Retention Reporting schedule
  • Logs must be either forwarded via rsyslog or via the retention script configuration (recommended)
  • The retention script has successfully executed at least 1 time and the logs received at the Log Decoder/VLC
  • All the retention script log meta should be visible in Investigate before scheduling reports

Configuration Changes Summary

Log Decoder

Log Parser Modification

A customization file (cef-custom.xml) for the CEF Log parser will need to be added to the /etc/netwitness/ng/envision/etc/devices/cef directory, or if you already have a custom file there you will merge the two files together to get the new meta from the retention script.

cef-custom.xml contents:

<DEVICEMESSAGES>
    <ExtensionKeys>
        <ExtensionKey cefName="retention" metaName="retention"/>
        <ExtensionKey cefName="collection" metaName="collection"/>
    </ExtensionKeys>
</DEVICEMESSAGES>

Log Decoder Custom Index Entries

There will need to be an entry added for the numeric formatted meta.

log-decoder-index-custom.xml contents:

<!-- *** Index 1.0 05/04/2020 *** -->

<!-- *** Custom metakeys for Retention *** -->
<key description="Retention" name="retention" format="UInt32" level="IndexNone"/>

Log Decoder Table Map Entries

These entries will allow us to leverage the new metakeys from the CEF custom file.

table-map-custom.xml contents:

<!-- Added for Retention Script -->
<mapping envisionName="retention" nwName="retention" flags="None" format="UInt32"/>
<mapping envisionName="cs_oldfilectime" nwName="file.oldest" flags="None" format="Text"/>
<mapping envisionName="collection" nwName="collection" flags="None" format="Text"/>

Retention Log Script Log Forwarding

Edit the retention script to send the logs to the VLC or Log Decoder IP address.

Concentrator

Concentrator Index Modification

The new metakeys we have created will need to be in the index for use in Investigation and Reporting

concentrator-index-custom.xml contents:

<!-- *** Index keys added for retention script *** -->
<key description="Retention" name="retention" format="UInt32" level="IndexValues" valueMax="500"/>
<key description="Oldest File Time" name="file.oldest" format="Text" level="IndexValues" valueMax="10000"/>
<key description="Collection" name="collection" format="Text" level="IndexValues" valueMax="500"/>

Reporting Engine

Custom Retention Reports

Custom Retention reports will need to be imported via the "Retention.zip" file, located in the download file at the bottom of this post.

Investigation

Custom Metakey Group

You can import the custom metagroup file RetentionMetagroup.jsn for Investigation to create a meta group dedicated to just viewing Retention Meta.

Detailed Installation Instructions

Log Decoder

Log Parser Installation from RSA Live

Deploy the latest CEF Parser from Live.  The older versions of the CEF parser will not work properly.

LeonardC_0-1623419749210.png

 

LeonardC_4-1621950307197.png

LeonardC_6-1621950476549.png

LeonardC_5-1621950358411.png

 

CEF Customization File

Install the CEF Customization file (cef-custom.xml).  If you already have one of these files, STOP!  You will need to merge the contents of your custom file and the contents of the custom file used in this article.  The steps below will REPLACE your existing file.  If you are not sure, check the directory location /etc/netwitness/ng/envision/etc/devices/cef to see if a "cef-custom.xml" file exists. If you have a cef-custom.xml file or are not sure, skip the procedure below to load the cef.envision parser file until you can determine if you need to merge other custom settings.  Read this article for more information on CEF Parser Customization.

LeonardC_0-1621952654368.png

LeonardC_1-1621952999046.png

LeonardC_2-1621953064834.png

LeonardC_3-1621953495541.png

LeonardC_4-1621953546920.png

LeonardC_0-1622053949913.png

LeonardC_7-1621955652581.png

 

Verify CEF Parser is enabled

LeonardC_8-1621951208893.png

LeonardC_9-1621951293584.png

 

Add Custom Entries Log Decoder Custom Index File

LeonardC_3-1622054310277.png

 

Add Custom Entries to the Log Decoder Table Map Custom File

LeonardC_1-1621966794128.png

LeonardC_2-1622054216671.png

 

LeonardC_10-1621891877399.png

LeonardC_11-1621891905716.png

 

Concentrator

Add Custom Entries to the Concentrator Custom Index File

LeonardC_3-1621966958319.png

LeonardC_4-1621967158205.png

LeonardC_5-1621967319255.png

LeonardC_6-1621892352451.png

LeonardC_7-1621892377526.png

 

Verify Meta

To verify the meta in the system, the retention script will need to have been executed after all the custom configuration steps above have been completed.  You can wait for it to occur on your cronjob schedule or execute it manually.  There is one caveat to running the script multiple times within a 24 hour period, it may skew your trending numbers in the report for that day. 

Device Type

rsa_netwitness_custom_script

Metakeys populated

event.type
event.desc
service.name
obj.name
event.computer
ip.addr
retention
obj.type
filename
ip.orig
collection

Investigate Screenshot

LeonardC_1-1623436441115.png

 

 

Report Engine

Import Reports to Reporting Engine

LeonardC_6-1621967545314.png

LeonardC_8-1621967914666.png

LeonardC_7-1621967856815.png

LeonardC_9-1621967974537.png

 

Schedule Dynamic Device List Reports

Starting with the Log Decoder, setup the schedules to build the device lists that the other reports will use for the retention reports.  Perform this task for each device you have in your environment.

For example, if you only have Log Decoders and Concentrators, then you would have schedules on the following reports:

RE-RETLIST-01 - Log Decoder Dynamic List for Retention Reports

RE-RETLIST-03 - Concentrator Dynamic List For Retention Reports

RE-RETLIST-04 - Concentrator Aggregation Stack Dynamic List For Retention Reports

You only need one daily schedule for all Log Decoders and all Concentrators in your environment.  Hybrid devices are broken up into their respective service types (Log Hybrid = Log Decoder, Concentrator) so they are treated the same as the non hybrid device when it comes to scheduling.

LeonardC_10-1621968098383.png

LeonardC_18-1621892844461.png

Setup a Daily report Schedule.  Note the time at which this report is scheduled (23:00).  We want the ALL of the Dynamic List reports to run at the same scheduled time, as we have to create the dynamic list FIRST before all the other reports are scheduled to run.  For example, the timeline would look like:

22:00 - Retention Script Cronjob executes retention script

23:00 - Dynamic List Reports Run to Generate Dynamic Lists

23:30 - All Other Retention Reports Are Scheduled to Run, Utilizing the Dynamic Lists Generated at 23:00

Choose whatever time you like when you schedule the reports, just make sure you have the Dynamic List Reports running 30 minutes BEFORE the other scheduled reports.

LeonardC_19-1621892862423.png

LeonardC_20-1621892879957.png

The list names will all line up with the report name.  For Example:  Report - "RE-RETLIST-01 - Log Decoder Dynamic List for Retention Reports" will populate List - "LI-RETENTION-01 - Log Decoders".  So basically "Report 01" will line up with "List 01", "Report 02" will line up with "List 02", etc...

LeonardC_12-1621970269461.png

In the "Rule" and "Column" there is only one item to select for any of the Dynamic List Reports.  Be sure to "Overwrite Existing List" to keep it accurate.

LeonardC_13-1621970482951.png

LeonardC_23-1621892935647.png

Repeat the schedule steps for each type of device in the environment, don't forget to schedule the aggregation reports.

 

Schedule the Concentrator/Archiver Aggregation Stack Retention Reports

The aggregation reports will have a report for each stack (typically a Concentrator/Archiver and the Decoder(s) it is aggregating from).  This report is dependent upon the Dynamic List Reports functioning properly.  

RE-Retention-01 and RE-Retention-02 are the same, except RE-Retention-01 has an explanation of the numbers in the report.

LeonardC_3-1621976988975.png

LeonardC_15-1621971693136.png

 

Schedule a Daily report to run 30 minutes AFTER the Dynamic List Reports were scheduled to run and enabling an "Iterative Report"

LeonardC_5-1621977245490.png

 

Select the "LI-RETENTION-04 Concentrator Aggregation Stack" List for Concentrators

LeonardC_18-1621972150114.png

 

LeonardC_19-1621972268980.png

LeonardC_0-1621975891339.png

For an Archiver report repeat the same steps except use the "RE - RETENTION - 05 Archiver Aggregation Stack Report" and the "LI-RETENTION-05 - Archiver Aggregation Stack" List.

 

Schedule Trending Reports for Concentrator/Decoders

The trending reports will show the retention over time for the particular device.  The steps below show how to set it up for the Concentrator.

LeonardC_0-1622041602900.png

LeonardC_1-1622041634883.png

LeonardC_5-1622054838231.png

 

LeonardC_3-1622041666438.png

LeonardC_4-1622054687658.png

Repeat the same steps for the other Log Decoder or Decoder devices.

 

Schedule Trending Reports for Archivers (Beta)

This trending report will show the Archiver retention trending data.  This has not been thoroughly tested with collection rules with different retention times.  Verify the trending output before trusting it.  Due to the Report Engine limitations this is the only option available within the NetWitness Platform at this time.  The aggregation stack report will be the most accurate.  Below are the steps to schedule a report for the Archivers.

LeonardC_5-1622043353732.png

LeonardC_6-1622043382272.png

LeonardC_6-1622055025274.png

 

LeonardC_8-1622043454170.png

LeonardC_7-1622055071365.png

LeonardC_8-1622055116063.png

LeonardC_12-1622043549717.png

LeonardC_9-1622055170416.png

 

Install the Investigation Meta Group

For best results, Logout and login back into the UI, so that Investigate can register the new custom metakeys before you import the metakey group.

LeonardC_8-1621892567136.png

LeonardC_9-1621892591399.png

 

LeonardC_12-1621892651710.png

LeonardC_13-1621892671224.png

LeonardC_14-1621892721917.png

 

Logging meta and reporting is complete.  Enjoy!

 

Troubleshooting Empty Reports

  1. Ensure the script is scheduled to run as a cronjob once every 24 hours (no more, no less).  The script will take approximately 3-5 seconds per stack to run, calculate your total runtime and make sure your reports are scheduled after the script completely executes.
  2. Make sure you have meta in Investigate from the completed script run. (no meta = empty report)  Check CEF parser and  associated index/tablemap/parser customizations required to create the proper meta.
  3. Verify the report schedules that create the lists have an output to the proper dynamic list and are scheduled to run BEFORE the other retention reports. (no lists = empty report)  Check the lists in the Reporting Engine to see if they are populated with the proper hosts/ip addresses.

 

Report Sample Output Screenshots

Concentrator Aggregation Stack Report

LeonardC_0-1623424514120.png

LeonardC_1-1623424726729.png

 

Individual Concentrator Report

LeonardC_2-1623424871158.png

LeonardC_3-1623424950711.png

LeonardC_4-1623425070663.png

LeonardC_5-1623425150472.png

 

 

Retention Reporting.zip
Retention Reporting.zip
2 Comments
© 2022 RSA Security LLC or its affiliates. All rights reserved.