Reporting Overview

[#1]

Reporting Overview

Reporting is a collection of data as a result of monitoring the network traffic, which can be used for further analysis. In NetWitness you can run a report against NetWitness Database core services to identify the network activities. For example, if you want to identify the Top Source Countries and Destination Countries, or top Threat and Risk trends that help monitor any changes to the normal categories or monitor the users and services that may potentially have malicious activities etc.

The reporting typically consist of: Reports and Charts. You can report on the log, packet and endpoint data collected, and customize the reports and charts to enhance the visual appearance. You can create real-time reports for historical data. You can create charts and dashlets, that can be added in the real-time chart dashlets ​as well.

Reporting Engine

Reporting relies on the Reporting Engine to provide data for the reports, alerts and charts. Hence, you must configure the Reporting Engine as a service to NetWitness before you can generate the reports. You must also specify the data source in the Reporting Engine from which the data is extracted.

The data that you can report or alert depends on the configuration of Reporting Engine and the data sources that you specify as part of the rule definition.

Note: Make sure you have access to the components in the Reporting.

Note: Make sure you have access to the required data sources. Only privileged users with access to sensitive information have the permission to certain data sources. To manage access control to data sources, see the "Add a Role and Assign Permissions for Warehouse Analytics" However, for the existing reports, alerts and charts, if the user role or permissions are modified for the data sources, then it is not applicable unless you manually update the permissions.

Note: Reporting is accessible based on the role based access, defined for the user.

Report

A report is a combination of rules and other formatting objects such as headers and HTML-formatted notes that describe and identify data pertaining to a particular area of interest. Reports are defined and managed in the Build Report page and can be scheduled to run on an adhoc or timely basis. Once a report is run, results are stored centrally and can be automatically sent over email, SFTP, URL, and NFS to users, viewed via the NetWitness web interface, downloaded as PDF and CSV files.

A report consists of the following:

Property	Description	Example
Report Name Note: For Name field, the icon to extend the column size is not displayed at the end of the column field. You have to hover the mouse a little to the left side to see the icon for extending the column.	Used to identify the report to schedule them at a later time.	Report1
Text	Pre-defined text fields used within a report to make the report more meaningful to the user.	Header1, Comment
Rules	The rules (queries) used to create a report.	select user.dst where ip.src=10.10.10.1

Note: In the Reporting user interface, the displayed date or time is always according to the user-selected time zone profile.

Rule

A rule is the basic and essential building block in the Reporting. You must create a rule which can be used in a Report, Chart or Alert.

A rule represents a unique query that detects and summarizes the requested information within a collection of network data.

The rule syntax is very similar to that of Standard Query Language (SQL) where you can use the select clause, where clause, sort and group options and limits for the result set. A rule consists of the following:

Property	Description	Example
Name	The name of the rule.	Windows System Account Activity
Select	List of meta types that are returned in the result set. The list of meta types is provided in the Meta Library. Meta Library in the Rule Builder is continually synchronized with the index configuration of the NetWitness host to which NetWitness is connected. The number of meta types that this property can represent depends on how the rule is to be sorted. If the Sort by property is 'None' or Custom, a rule can have more than one select field, for example, for each match, include the ip.src, ip.dst, size, time in the rule result. If a rule is set to be sorted, either by session count, session size, or packet size, then there can only be one field on which to select.
Where	A clause that is the base query for the rule.	alert='cleartext_ftp_password'
Then (Rule Actions)	A series of functions that manipulate the original result set of a rule in order to make the output in a report more meaningful or add additional functionality other than querying and displaying data.	lookup_and_add ('username','ip.src',10);
Sort By	Determines how the data in the result set is sorted. The various possibilities are: Total Value Column Name	Total
Limit	Designates how large a result set can be for the given rule. Users must note that if a result set is sorted by count or size, the limit represents the top (or bottom) N values to be returned. If the result set is not sorted, the first N values are returned.	20

Note: In the User Interface (UI), the date or time displayed depends on the time zone selected by the user.

Rule Types

There are different rule types in the Reporting. Rule types designate the source of data for the report rule. Following are the rule types:

Rule Type	Description
NetWitness Database (NetWitness DB)	The NetWitness database extracts the meta from a Reporting Engine configured to use a Concentrator, Broker and Archiver as the data sources and provides the meta for rules.
Warehouse Database (Warehouse DB)	The Warehouse database, also referred to as the NetWitness Warehouse, warehouses large volumes of data. The Warehouse is designed so that you can retrieve large volumes of data easily and efficiently. The Warehouse also extracts the meta from the Reporting Engine.
Respond Database (Respond DB)	The Respond database contain alerts and incidents generated from different services and you can create a report on those alerts and incidents.
Unused Rules	The Unused Rules is an option that helps in listing out the rules that are not used by any chart, alert and, report. For more information, see Filter Unused Rules.

Note: In the User Interface (UI), the date or time displayed depends on the time zone selected by the user.

List

A list is a variable that refers to a series of comma-separated values (CSV). You can insert a list into a rule or use it as an argument to a rule action. Lists can act as placeholders for other values, which you can populate and update as needed.

You can create, manage and view lists that can be used to define rules for Reporting and Alerting. You can also filter and delete the unused lists.

Lists cannot be empty or have duplicate or blank values.

Note: If you are defining a report with a rule which has lookup_and_add in the Then clause and direct the report output to a list, the list is not populated with the result.
For example, if you create a rule with ip.src in the Select clause and lookup_and_add ('ip.dst','ip.src', 10) in the Then clause, the report displays the result, but if you have redirected the output to a list, the list will be empty

Chart

Chart is a tabular or grid representation of data. It consists of the following:

Property	Description	Example
Chart Name	Identifies the chart.	Chart1
Rule Basis	Identifies the rule path chosen from the folder hierarchy.

Any NetWitness DB rule in the Reporting Engine system which is not sorted by none can be used to instantly create a chart. In NetWitness, the chart interval can be adjusted from the chart definition panel itself. Each time a chart runs, it stores its result data locally in the Reporting Engine, so that it can be reviewed in either the Dashboard View or Chart View without any performance considerations. You can also filter the unused charts from Chart View.

Note: In the Reporting user interface, the output for the field where Date and Time are displayed is always according to the user-selected time zone profile.

Note: The Reporting Engine (RE) will automatically check for the available disk space before you execute a Test Rule, Report, Chart and Alert. If the RE disk space (in percentage) is less than the minimum disk space threshold (default value is 5), the RE will stop the current execution and an error message 'Available disk space of Reporting engine home is <5%, please clean up the space to proceed further' is displayed. Additionally, you may also configure the minimum disk space threshold by using the following path: RE>Config>General>System Configuration>Mini disk space threshold in %.

Reporting Guidelines

This section lists recommended guidelines to enhance the execution time of your reporting entities such as rules, reports, alerts, charts, and lists. The guidelines are provided for the following:

• NWDB Rules
• Timeout Configuration for NWDB Rules
• Lookup and Add rule action
• List value Reports

NWDB Rules

If the reporting entities such as report, alert, or chart contain NWDB rules (in most cases where the query contains Group By) takes a long time to execute, you may do the following:

1. Refine the Where clause:

   You may limit the number of sessions scanned by using or refining the Where clause (especially when you use the Group By option). For example, consider the following rule.

   

   If you use a Where clause as mentioned above, the number of sessions aggregated is huge. To avoid this, you can filter only required sessions by specifying the list of IP addresses or creating a List (list of IP Address) that contains relevant IP addresses.

   Note: The NWDB rule where clause is appropriately quoted if the syntax has an invalid quote. For example, in case of an invalid meta, or missing separator, the status and the error message is updated appropriately.

   
   

2. Using indexed Meta keys in the Where clause:

   To understand if the Meta is indexed or not, mouse hover the Meta List present on the right panel. If the Value Type is INDEX_VALUE, then the Meta is indexed. The Value Type is INDEX_KEY or INDEX_NONE if the Meta is not indexed.

   
   Below is a snapshot of a Meta key that is indexed.

   

3. Configure the Timeout option:

   If the query is taking a long time and fails due to timeout issues, you can configure the timeout for the NWDB rule executions. For more information, see below section "Timeout Configuration for NWDB Rules".

4. Schedule the queries to run at different times:

   If multiple query aggregates are concurrently executed and timeout occurs, you may schedule the queries to run at different times without much overlap.

Timeout Configuration for NWDB Rules

Note: It is a good practice to check the statistics of the Reporting Engine and the NWDB data sources before you make any changes to the configuration. For more information, see the "Monitor Service Details" topic for Reporting Engine and "Monitor System Statistics" topic in the System Maintenance Guide.

If NWDB rule execution fails due to timeout, you may get the following errors on the View a Report panel:

• Reporting Engine timeout error

  “Data source ‘10.31.x.x Concentrator' did not respond within the configured time 30 minutes for the ‘/sdk/values' request.”

• NWDB timeout error

  "Error occurred while fetching data from source '10.31.x.x Concentrator'. {Timeout message from NWDB}”

In such cases, you need do the following:

• Reporting Engine timeout

  In case of Reporting Engine timeout, you may set the timeout to a longer duration so the long running queries can be executed. For more information on setting the NWDB Queries Time Out and NWDB Info Queries Time Out option for the Reporting Engine, see "Step 2. Configure Reporting Engine Settings" topic in the Reporting Engine Configuration Guide. NetWitness recommends you set the NWDB Query Time Out to zero minutes (implies no timeout) and NWDB Info Queries Time Out to 60 minutes.

• NWDB timeout

  In case of NWDB timeout, you may need to configure the query.level.timeout and max.concurrent.queries parameters for the NWDB data source based on the recommendations in the Core Database Tuning Guide to fine tune the queries.

  The following figure is an example of Explorer view where you can set the parameters for NWDB data source.

  

• Schedule Reports at different times

  If the NWDB core devices are heavily utilized, you may schedule the reports to run at different times without overlap.

• Split the Report

  If you have many rules in a Report, split the report into multiple reports with each report containing logical set of rules. If you have multiple rules, all rules will begin to execute at the same time based on available threads, therefore you may group the rules logically into separate reports.

Lookup_and_Add Rule Action

If a rule that consists of single or multiple lookup_and_add rule actions, takes a long time to execute the report, it is because each of the rule action triggers multiple lookup queries on the NWDB data source resulting in longer execution time.

To improve the report execution time, you may do the following:

• Refine the Where clause in the following:

  • Rule that contains the lookup_and_add rule action
  • lookup_and_add rule action

• Set Limits

  You must set appropriate limits for the rule and rule actions. If the limit is high it will result in many queries being triggered and hence the report will take a long time to execute.

• Set the boolean aggregate parameter

  If you do not want the aggregate value such as sum(meta), count(meta) etc. for the lookup values, set the boolean aggregate parameter to false in the lookup_and_add rule action. For more information, see the "NWDB Rule Syntax" section in Rule Syntax.

  lookup_and_add(string select, string field, int limit, boolean inherit, string extraWhere, boolean aggregate)

  Consider the rule with lookup_and_add rule action:

  

  
  The output is displayed:

  

• Each lookup_and_add rule action triggers by default two concurrent lookup queries on the data source. NetWitness recommends that you retain the default setting, however if you want to increase the value you may want to ensure the value of Max # of Concurrent LookupAndAdd Queries parameter in Reporting Engine is less than the Max Concurrent Queries value in the NWDB data source configuration.

  If the NWDB data source is shared across other services, then you may retain a low value for the Max # of Concurrent LookupAndAdd Queries parameter in Reporting Engine as increasing it will impact the queries from other services. For more information, see "Reporting Engine General Tab" topic in Reporting Engine Configuration Guide.

• If you are interested only in unique values and not accurate aggregates, then set the Session Threshold to a non-zero value for the NWDB rule. For more information, see "Create a Rule Using NetWitness Data Source" section in Configure a Rule. The higher the value, the longer is the rule execution. If the value is set to zero it will take a longer time but will provide accurate aggregates.

  Consider a rule with lookup_and_add rule action and Session Threshold set to 10.

  

  The output is displayed:

  

List Value Reports

Use a Refined List:

In case of List value reports (for any data source type), individual reports will be generated for each value in the list. Therefore, more the number of values in the list the longer the reports will take to execute. Hence, you must use a refined list to generate such reports.

Access Control for Reporting

Reporting Module provides you the option to set up access control for all the components in the module. In NetWitness, you can define different roles and specify the access control for each of the role from the System Security module. You can define the access control to be provided for the Reporting module for each role. For more information, see "Step 1: Review the Pre-Configured NetWitness Platform Roles" and "Step 2: (Optional) Add a Role and Assign Permission" topics in the System Security and User Management Guide.

In the Reports module, you can modify the role permissions or access to the following Reporting objects:

The Following is an example of the hierarchy of the object groups, objects and dependents. This is an illustration of the Report Groups and Reports hierarchy.

Report Groups and Reports Hierarchy

Permission for Object Groups

• You must have the Read & Write permission to set the permissions for the Object Group, Objects, or Dependents. The dependents with “No Access” permission are grayed out and dependents with “Read-Only” permission are indicated with an icon.
• When you set the permission for the Object Group, the Objects and Dependents in the Object Group do not inherit the permission automatically. You must select the "Apply these permissions to sub-groups and <Objects> in this group" option to achieve this. For example, if you do not want Operators roles to access reports in Report Group A, then you must set the permission on Group A to No access for the Operator role and select the "Apply these permissions to sub-groups and Reports in this group" option.
• When you set the permissions for the Object Group and select the "Apply these permissions to sub-groups and <Objects> in this group" option, the dependents such as rules or schedules in the objects do not inherit the permissions automatically. You must use the "Apply Read-only permission to Rules in the <Object>" option to apply the permission to the rules.
• When you set the permissions for the Objects, you must ensure that the Objects in hierarchy should always have a permission that is less than or equal to the one above in the hierarchy for the permission to be applied. For example, if the reports in a Report Group have Read & Write permission and you apply a Read-Only or No Access permission at the Report Group level and select the "Apply these permissions to sub-groups and Reports in this group" option, then the permission on the rules will remain unchanged.
• The permissions are cascaded from top to down in the hierarchy and not vice-versa. For example, if you apply a permission to a rule, it does not change the permission of the Report that contains the rule.

Permission for Objects or Dependents

• You must have the Read & Write permission to set the permissions for the Objects or Dependents.
• You can specify the permission for multiple objects at once instead of setting the permission for each object.
• When you set the permission for the Object, the dependents in the Object do not inherit the permission automatically. You must select the "Apply Read-only permission to Rules in the <Object>" option to achieve this.

When you apply the permission to dependents the permission is applied based on the existing permission for the role. For example, consider an Analyst and a Operator with the following permissions for the different dependents (Report A object has Rule AA, Rule AB, and Rule AC as dependents).

Object or Dependent	Analyst	Operator
Report A	Read & Write	No Access
Rule AA	Read & Write	No Access
Rule AB	Read and Write	Read and Write
Rule AC	Read-only	No Access

When the Analyst applies a Read & Write permission for the Operator role and selects the option "Apply Read-only permission to Rules in the <Object>", the permissions is set for the different dependents as follows:

Modify the Permissions

• Group Level: Set the permissions at the Object Group level and for all the object and entities in the Group. For example, if you have 80 reports in the Administrators Reports group and you do not want anyone except the Administrator to add or modify these reports, you can set the permission for all the other roles at the group level to Read-Only and select the option to apply it to all the reports and sub-groups in the report group.
• Multiple Objects:Select multiple objects and specify the access for all the selected objects. For example, if you have 10 reports in the Network Traffic sub group with sensitive information that you do not want anyone to access, select the 10 reports and then set the permission for all the roles as "No Access".
• Single Object: Select only the object and specify the permission. For example, select the Network Traffic Report and specify the Read-Write permission for the Security Analyst role or select the Login Failure Alert and specify the Read-Write permission for a Security Analyst role.

Object or Dependent	Operator (Before Permission is applied)	Operator (After Permission is applied)
Report A	No Access	Read & Write
Rule AA	No Access	Read-only
Rule AB	Read and Write	Read & Write
Rule AC	No Access	Read-only

Roles and Permissions for Reporting Module

Although NetWitness has five pre-configured roles, you can add custom roles. For example, in addition to the pre-configured Analysts role, you can add custom roles for AnalystsEurope and AnalystsAsia.

Role	Permission
Administrators	Full system access
Operators	Access to configurations but not to data
Analysts	Access to data but not to configurations
SOC_Managers	Same access as Analysts and an additional permission to handle incidents
Malware_Analysts	Access to malware events only

Depending on the user role, you can set the following access permissions to access the Reporting module components (Rules, Reports, Charts, Alerts, Lists):

• Create
• Delete
• Export
• Manage
• View

Note: You must enable all these permissions for a user role to be able to define, delete, manage and view each of the Reporting modules. You must also have appropriate permissions for the data source to be listed, while defining the reports, charts, or alerts. For more information, see "Configure Data sources Permissions" topic in the Reporting Engine Configuration Guide.

For a detailed list of permissions and how to add a role and assign permissions, see "Role Permissions" and "Step 2. (Optional) Add a Role and Assign Permissions" topics in the System Security and User Management Guide.

Previous Page	Next Page