Troubleshooting NetWitness Investigate

This section provides information about possible issues when using NetWitness Investigate.

Navigate View and Legacy Events View Issues

 

Behavior

A meta key that normally returns values in the Navigate view returns values, but has a Not Indexed message following the meta key name. For example, the in this figure the Service Type meta key is followed by the message: Service Type[service] Not Indexed.

netwitness_notindexmsg.png

Issue

When you first set up the environment or very rarely after performing a data reset on the broker due to other issues, you see meta keys as Not Indexed when they are indexed at meta key or meta values level.

Explanation

To fix the issue on a Broker, log out of NetWitness Platform XDR and then log in again. Valid sessions will be displayed.

 

Message Not indexed; will experience longer than usual load times. in the Manage Meta Groups dialog.
Issue

Meta keys in the Manage Meta Groups dialog are marked by a red exclamation point, and the error message is displayed. This can occur when investigating a Broker or Decoder while adding a meta group with meta keys that are not indexed in the index file or the custom index file for the service.

For a Broker, it could mean that the Broker has not begun aggregating data from a Concentrator. In this case the Broker will not have the contents of the custom index file from the aggregate services and the keys will not be indexed.

For a Decoder, it means that the meta keys are not indexed in the Decoder index or custom index file.

Explanation

To fix the issue on a Broker, log out, log in, and restart the Broker service so that it can aggregate the meta key information from connected Concentrators. To fix the issue on a Decoder, edit the custom index file to index the meta keys, log out, log in, and restart the Decoder service.

 

Behavior

When downloaded from the Event Reconstruction view, logs and metadata are always in text format irrespective of the format selected in the Legacy Events view.

Issue

When you download metadata or a log in the Event Reconstruction view, the format that you selected in the Legacy Events view is not used. The exported data is always in text format.

Explanation

Download metadata and logs from the Legacy Events view if you want to use a format other than text format.

 

Events View Issues

 

Message Applicable for hosts with 4.x Endpoint agents installed, please install the NetWitness Endpoint Thick Client.
Issue When you click Pivot to Endpoint in the Events view, no data is displayed and the message is displayed.
Explanation Version 4.4 of the NetWitness Endpoint Thick Client must be installed on the same server, the NWE meta keys must exist in the table-map.xml file on the Log Decoder, and the index-concentrator-custom.xml file on the Concentrator. The NWE Thick Client is a Windows only application. Complete setup instructions are provided in the NetWitness Endpoint User Guide for Version 4.4.

 

Behavior Download jobs are in a Waiting state or Failed state in the Jobs tray during and after upgrading the software to Version 11.4.
Issue If you had download jobs running while your administrator was upgrading the software, you may see a job in a Waiting state while the upgrade is in progress and then in a Failed state after the upgrade is complete. You cannot resume or cancel the failed job.
Explanation To delete the failed jobs, select the failed jobs in the Jobs tray and clicknetwitness_ic-delete.png.

 

Message Event counts in the Filter Events panel and the Events panel may sometimes differ when showing results for the same query.
Issue The Filter Events panel uses only index data to produce counts of events, which is less accurate than the Events panel. The Events panel results are filtered for exact matches on data retrieved from the meta database, which takes longer to process.
Explanation At worst the difference is in false positives in the Filter Events panel, not false negatives; so you will not miss an event.

 

 

Message Event Analysis requires all core services to be NetWitness 11.1. Connecting prior versions of services to the 11.1 NetWitness Server results in limited functionality (see "Investigate in Mixed Mode" in the Physical Host Upgrade Guide).
Issue When attempting to investigate a service that has not been updated to Version 11.1 in the Event Analysis view, the informational message is displayed.
Explanation When an analyst opens the Event Analysis view in mixed mode (that is, some services are upgraded to 11.1 and later, and some are still on 11.0.0.x or 10.6.x), Role-Based Access (RBAC) is not applied uniformly. This affects viewing and downloading content, and validation of filters in the interactive breadcrumb. You will see this informational message when you open Events. As you select a service, services that are not up to date are displayed in a red box, with the message that the service is not up to date. When your administrator has upgraded all connected services to 11.1 and later, these features work as expected.

 

Message Forbidden. You cannot access the requested page.
Issue When attempting to access the Events view, the view opens with the message.
Explanation Your administrator has prevented access to the Events view using role and permissions.

 

Behavior If you can download an event in the Events view, but get a 0-byte file, the administrator may have restricted access to the content.
Issue Role-Based Access Controls applied by your administrator allowed you to download an event for which you did not have permission; therefore, the file download was empty.
Explanation If you believe you should have access to the event, contact your administrator.

 

Message

Insufficient permissions for the requested data.

Issue While attempting to access an event in the Events view, the message is displayed.
Explanation You have entered an event ID for an event that you do not have permission to view. The administrator may have placed some restrictions to limit access by role and permissions.

 

Message Invalid session ID: <<eventId>>
Issue No sessionId matches the sessionId that you queried.
Explanation The reason for an invalid session ID can vary. Perhaps you edited the session ID manually, and no such session exists. Another case may be when you query a Broker, and the aggregated data has not been refreshed, you may see this error for a session that no longer exists.

 

Behavior Investigation Profiles and built-in column groups are not present in 11.1 Event Analysis.
Issue Post upgrade to NetWitness v11.1, the default column groups - Endpoint Analysis, Outbound SSL and Outbound HTTP are not added under column groups. Also, a few of the Investigation Profiles are missing post upgrade.
Explanation

It is observed that this issue occurs only when you have created a custom column group with the name which is same as one of the new 11.1 OOTB custom column group name. For example, if you create a custom column group in 11.0 with name RSA Endpoint Analysis then after upgrade to 11.1. Due to the same name already existing in 11.1, OOTB column groups and built-in profiles will not be available in the UI.

To fix this, change the name of custom column group to something other than one of the OOTB column groups and restart the jetty server by using the following command on the NetWitness server:

systemctl restart jetty

 

Message Memory limit of <XXXXXX> GB reached, controlled by setting max.query.memory
Issue The query that you submitted failed because the result set was too large, and the memory limit set by max.query.memory was reached.
Explanation To avoid this error, try to further limit results by narrowing the time range, adding filters, and decreasing the number of columns in the column group. You can also ask an administrator to limit the number of events returned.

 

Behavior No text data was generated during content reconstruction. This could mean that the event data was corrupt or invalid, or that an administrator has disabled the transmission of raw endpoint events in the Endpoint server configuration. Check the other reconstruction views.
Issue When you reconstruct an event as text in the Events view, no data is displayed and the message is displayed.
Explanation If you do not see the raw text in other Events views or Legacy Events view reconstructions, and you believe the data is not corrupted or invalid, your administrator has likely disabled transmission of raw endpoint events on the NetWitness Endpoint server. Contact your administrator for additional information.

 

Message

Rule Syntax error: Unrecognized key "<meta key or meta entity name>"

Syntax error: Unrecognized key "<meta key or meta entity name>"

Issue

While querying a service, the matching events are not listed and the message is displayed in the query console and the Events view.

netwitness_eventanalysisentities.png

Explanation The query you entered is querying a meta entity that is not configured properly. All upstream devices connected to the Broker being queries should have the same entity configuration. This error indicates that the Broker is operating with mismatched entity definitions. Ask your administrator to review the configuration described in "Index Customization" in the Core Database Tuning Guide.

 

Message

Selected Column Group is no longer available. The default summary column group has been selected instead.

Issue If you had set a preferred column group before the 11.4 upgrade, on your first visit to the Events view, the flash message is displayed even when the column group is available or is the default group (summary). This issue was resolved in Version 11.4.1.
Explanation This is a one-time occurrence. If you reload the Events view, the message is not displayed.

 

Message

Session is unavailable for viewing.

Issue While querying an event ID, the reconstruction is not displayed and the message is displayed.
Explanation The query you entered is trying to look at restricted data, for example, if you are allowed to see only log data and you are using a link to network data .

 

Message The query on channel <channel-number> was auto-canceled by the system for exceeding time usage limits. Check timeout values. Query running time was 00:05:00 (HH:MM:SS)
Issue If you continually get this timeout message, first check the query console to determine if there are issues around time it takes for a service to respond, index error messages, or other warnings that may need to be addressed to increase query response time.
Explanation If there are no messages indicating any specific warnings, ask your administrator to increase the Core Query Timeout from 5 minutes to 10 minutes as described in the System Security and User Management Guide.

 

Message The session id is too large to be handled:<<eventId>
Issue The session id that you typed in, or got from the Legacy Events view or Navigate view is too large.
Explanation If you manually typed the sessionId or edited a sessionId in the Events view, you may have created an integer that is too large for Events to process.

 

Behavior

When reconstructing network events with a large number of packets (>250) in the Events view > Packets panel, with the option to display only payloads enabled and the packets per page setting higher than the default (100), the current browser tab may become unresponsive for up to 45 seconds as it is working to render the payloads.

Issue

Depending on the amount of resources (memory and CPU) on the client machine and the number of packets in the event there may be a performance lag when displaying only payloads in packet reconstruction.

Explanation

To limit the amount of data processed in a reconstruction of a single event, change the Packets per Page setting in the footer to a lower value.

netwitness_pagpktperpg.png

 

Behavior When working in the Version 11.4 Events view, the Query Profile drop-down menu and Column Group drop-down menu do not function.
Issue You do not have permission to read columns groups and profiles. The default column group , Summary List, is applied to the Events list, and you cannot change the column group, create a column group, or delete a column group.
Explanation This occurs only when the administrator has created a custom role for you instead of assigning the default Analyst role. Ask your administrator to enable column group read and profile read permission for your role.

 

Issue No matching Endpoint data available on Investigate > Events view > Host tab.
Explanation

The Endpoint data may not be available due to any of the following:

  • No Endpoint Deployment – You must install Endpoint Log Hybrid, see “NetWitness Endpoint” in the Physical Host Installation Guide.

  • Endpoint data is not captured for the host associated with the selected network event- Make sure that NetWitness Endpoint Agent is installed, and expanded network visibility is configured to track the network events. To enable expanded network visibility, see “Creating Groups and Policies” in the NetWitness Endpoint Configuration Guide.

    Note: For Expanded Network Visibility to work, ensure the service user account used for aggregating Endpoint Log Decoder data to Endpoint Concentrator is assigned with the decoder.manage permission. For more information on how to assign roles and permissions, see "Services Security View - Aggregation Role" in the Hosts and Services Getting Started Guide for NetWitness Platform XDR.

  • Concentrators or Endpoint services are offline or very slow – You must check the status (online or offline) of the services on Health and Wellness. If the service is online, you must check the Endpoint server logs and (Endpoint) Concentrator logs for details.

  • Endpoint data is rolled over for the host associated with the selected network event - The Endpoint data may be rolled over due to data retention period configured. You must configure the data retention period to retain the endpoint data for a longer period. For more information, see “Configure Data Retention” in the Data Privacy Management Guide.

Investigate Events Reporting Issues

Reporting Engine Service Unavailable

Message I see the following error message while generating a report: The Reporting Engine service may be offline or inaccessible. Try starting the service.
Issue

This scenario occurs if the reporting engine service becomes offline or inaccessible due to the following reasons:

  • The administrator might have turned off the service.

  • The server on which the service is installed is not accessible.

Note: The offline service will be shown (circle with red) indicator.

Explanation

To fix the issue, you need to perform the following actions:

  1. Go to AdminIcon.png (Admin) > Services.

  2. In the Services list, select the Reporting Engine Service.

  3. Click actions_button.png > Start.

 

 

Insufficient Permissions

Message I see the following error message while generating a report: You do not have the required permissions to generate a report. Contact your administrator to request access.
Issue

This scenario occurs when an analyst tries to create or schedule a report without the required permissions.

Note: By default, only administrators will have access.

Explanation

To fix the issue, analysts must contact their administrators to request access permissions. Once the permissions are provided, the analysts can try generating the report later. For more information, see Configure Data Source Permissions in the NetWitness Reporting Configuration Guide.

 

 

Unsupported Custom Columns

Message

I see the following error message while generating a report: The following custom columns are not supported to generate a report. Remove them and try again.

  • <column name 1>

  • <column name 2>

Issue

This scenario occurs because the Reporting Engine Service does not support a few custom columns for generating the report on the Investigate > Events page.

Explanation

Perform the following actions to remove the custom columns from the list.

  1. In the Events view, click the (settings_icon.png) Settings icon.

    An available list of custom columns is displayed in a pop-up window.

  2. Unselect the unsupported custom columns from the list and try generating a new report.

 

 

Datasource is not Configured or Unavailable

Message

I see the following error message while generating a report: The datasource <service name> is not configured in the Reporting Engine. Add the datasource and try again.

Issue

This scenario occurs when you select a datasource that is not configured in the Reporting Engine.

Note: When the name of the datasource on the Investigate page differs from the name on the reporting page, an error message is displayed.

Explanation

To fix the issue, perform the following steps:

  1. Go to AdminIcon.png (Admin) > Services.

  2. In the Services list, select the Reporting Engine service.

  3. Click actions_button.png > View > Config.

    The Services Config View of Reporting Engine is displayed.

  4. Select the Sources tab.

  5. Click add.png and select Available Services.

    The Available Services dialog is displayed.

  6. Select the required service (for example, Log Decoder) and click OK.

    The service authentication dialog box is displayed.

    Note: The services with the Trust Model enabled must be added individually. You are prompted to provide a username and password for the selected service.

  1. Enter the Username and Password for the service.

  2. Click OK.

    The selected service is listed in the Aggregate Services pane.