Troubleshooting NetWitness Investigate
This section provides information about possible issues when using NetWitness Investigate.
Navigate View and Legacy Events View Issues
| Behavior |
A meta key that normally returns values in the Navigate view returns values, but has a Not Indexed message following the meta key name. For example, the in this figure the Service Type meta key is followed by the message: Service Type[service] Not Indexed. |
| Issue |
When you first set up the environment or very rarely after performing a data reset on the broker due to other issues, you see meta keys as Not Indexed when they are indexed at meta key or meta values level. |
| Explanation |
To fix the issue on a Broker, log out of NetWitness Platform XDR and then log in again. Valid sessions will be displayed. |
| Message | Not indexed; will experience longer than usual load times. in the Manage Meta Groups dialog. |
| Issue |
Meta keys in the Manage Meta Groups dialog are marked by a red exclamation point, and the error message is displayed. This can occur when investigating a Broker or Decoder while adding a meta group with meta keys that are not indexed in the index file or the custom index file for the service. For a Broker, it could mean that the Broker has not begun aggregating data from a Concentrator. In this case the Broker will not have the contents of the custom index file from the aggregate services and the keys will not be indexed. For a Decoder, it means that the meta keys are not indexed in the Decoder index or custom index file. |
| Explanation |
To fix the issue on a Broker, log out, log in, and restart the Broker service so that it can aggregate the meta key information from connected Concentrators. To fix the issue on a Decoder, edit the custom index file to index the meta keys, log out, log in, and restart the Decoder service. |
| Behavior |
When downloaded from the Event Reconstruction view, logs and metadata are always in text format irrespective of the format selected in the Legacy Events view. |
| Issue |
When you download metadata or a log in the Event Reconstruction view, the format that you selected in the Legacy Events view is not used. The exported data is always in text format. |
| Explanation |
Download metadata and logs from the Legacy Events view if you want to use a format other than text format. |
Events View Issues
| Message | Applicable for hosts with 4.x Endpoint agents installed, please install the NetWitness Endpoint Thick Client. |
| Issue | When you click Pivot to Endpoint in the Events view, no data is displayed and the message is displayed. |
| Explanation | Version 4.4 of the NetWitness Endpoint Thick Client must be installed on the same server, the NWE meta keys must exist in the table-map.xml file on the Log Decoder, and the index-concentrator-custom.xml file on the Concentrator. The NWE Thick Client is a Windows only application. Complete setup instructions are provided in the NetWitness Endpoint User Guide for Version 4.4. |
| Behavior | Download jobs are in a Waiting state or Failed state in the Jobs tray during and after upgrading the software to Version 11.4. |
| Issue | If you had download jobs running while your administrator was upgrading the software, you may see a job in a Waiting state while the upgrade is in progress and then in a Failed state after the upgrade is complete. You cannot resume or cancel the failed job. |
| Explanation | To delete the failed jobs, select the failed jobs in the Jobs tray and click . |
| Message | Event counts in the Filter Events panel and the Events panel may sometimes differ when showing results for the same query. |
| Issue | The Filter Events panel uses only index data to produce counts of events, which is less accurate than the Events panel. The Events panel results are filtered for exact matches on data retrieved from the meta database, which takes longer to process. |
| Explanation | At worst the difference is in false positives in the Filter Events panel, not false negatives; so you will not miss an event. |
| Message | Event Analysis requires all core services to be NetWitness 11.1. Connecting prior versions of services to the 11.1 NetWitness Server results in limited functionality (see "Investigate in Mixed Mode" in the Physical Host Upgrade Guide). |
| Issue | When attempting to investigate a service that has not been updated to Version 11.1 in the Event Analysis view, the informational message is displayed. |
| Explanation | When an analyst opens the Event Analysis view in mixed mode (that is, some services are upgraded to 11.1 and later, and some are still on 11.0.0.x or 10.6.x), Role-Based Access (RBAC) is not applied uniformly. This affects viewing and downloading content, and validation of filters in the interactive breadcrumb. You will see this informational message when you open Events. As you select a service, services that are not up to date are displayed in a red box, with the message that the service is not up to date. When your administrator has upgraded all connected services to 11.1 and later, these features work as expected. |
| Message | Forbidden. You cannot access the requested page. |
| Issue | When attempting to access the Events view, the view opens with the message. |
| Explanation | Your administrator has prevented access to the Events view using role and permissions. |
| Behavior | If you can download an event in the Events view, but get a 0-byte file, the administrator may have restricted access to the content. |
| Issue | Role-Based Access Controls applied by your administrator allowed you to download an event for which you did not have permission; therefore, the file download was empty. |
| Explanation | If you believe you should have access to the event, contact your administrator. |
| Message |
Insufficient permissions for the requested data. |
| Issue | While attempting to access an event in the Events view, the message is displayed. |
| Explanation | You have entered an event ID for an event that you do not have permission to view. The administrator may have placed some restrictions to limit access by role and permissions. |
| Message | Invalid session ID: <<eventId>> |
| Issue | No sessionId matches the sessionId that you queried. |
| Explanation | The reason for an invalid session ID can vary. Perhaps you edited the session ID manually, and no such session exists. Another case may be when you query a Broker, and the aggregated data has not been refreshed, you may see this error for a session that no longer exists. |
| Behavior | Investigation Profiles and built-in column groups are not present in 11.1 Event Analysis. |
| Issue | Post upgrade to NetWitness v11.1, the default column groups - Endpoint Analysis, Outbound SSL and Outbound HTTP are not added under column groups. Also, a few of the Investigation Profiles are missing post upgrade. |
| Explanation |
It is observed that this issue occurs only when you have created a custom column group with the name which is same as one of the new 11.1 OOTB custom column group name. For example, if you create a custom column group in 11.0 with name RSA Endpoint Analysis then after upgrade to 11.1. Due to the same name already existing in 11.1, OOTB column groups and built-in profiles will not be available in the UI. To fix this, change the name of custom column group to something other than one of the OOTB column groups and restart the jetty server by using the following command on the NetWitness server: systemctl restart jetty |
| Message | Memory limit of <XXXXXX> GB reached, controlled by setting max.query.memory |
| Issue | The query that you submitted failed because the result set was too large, and the memory limit set by max.query.memory was reached. |
| Explanation | To avoid this error, try to further limit results by narrowing the time range, adding filters, and decreasing the number of columns in the column group. You can also ask an administrator to limit the number of events returned. |
| Behavior | No text data was generated during content reconstruction. This could mean that the event data was corrupt or invalid, or that an administrator has disabled the transmission of raw endpoint events in the Endpoint server configuration. Check the other reconstruction views. |
| Issue | When you reconstruct an event as text in the Events view, no data is displayed and the message is displayed. |
| Explanation | If you do not see the raw text in other Events views or Legacy Events view reconstructions, and you believe the data is not corrupted or invalid, your administrator has likely disabled transmission of raw endpoint events on the NetWitness Endpoint server. Contact your administrator for additional information. |
| Message |
Rule Syntax error: Unrecognized key "<meta key or meta entity name>" Syntax error: Unrecognized key "<meta key or meta entity name>" |
| Issue |
While querying a service, the matching events are not listed and the message is displayed in the query console and the Events view. |
| Explanation | The query you entered is querying a meta entity that is not configured properly. All upstream devices connected to the Broker being queries should have the same entity configuration. This error indicates that the Broker is operating with mismatched entity definitions. Ask your administrator to review the configuration described in "Index Customization" in the Core Database Tuning Guide. |
| Message |
Selected Column Group is no longer available. The default summary column group has been selected instead. |
| Issue | If you had set a preferred column group before the 11.4 upgrade, on your first visit to the Events view, the flash message is displayed even when the column group is available or is the default group (summary). This issue was resolved in Version 11.4.1. |
| Explanation | This is a one-time occurrence. If you reload the Events view, the message is not displayed. |
| Message |
Session is unavailable for viewing. |
| Issue | While querying an event ID, the reconstruction is not displayed and the message is displayed. |
| Explanation | The query you entered is trying to look at restricted data, for example, if you are allowed to see only log data and you are using a link to network data . |
| Message | The query on channel <channel-number> was auto-canceled by the system for exceeding time usage limits. Check timeout values. Query running time was 00:05:00 (HH:MM:SS) |
| Issue | If you continually get this timeout message, first check the query console to determine if there are issues around time it takes for a service to respond, index error messages, or other warnings that may need to be addressed to increase query response time. |
| Explanation | If there are no messages indicating any specific warnings, ask your administrator to increase the Core Query Timeout from 5 minutes to 10 minutes as described in the System Security and User Management Guide. |
| Message | The session id is too large to be handled:<<eventId> |
| Issue | The session id that you typed in, or got from the Legacy Events view or Navigate view is too large. |
| Explanation | If you manually typed the sessionId or edited a sessionId in the Events view, you may have created an integer that is too large for Events to process. |
| Behavior |
When reconstructing network events with a large number of packets (>250) in the Events view > Packets panel, with the option to display only payloads enabled and the packets per page setting higher than the default (100), the current browser tab may become unresponsive for up to 45 seconds as it is working to render the payloads. |
| Issue |
Depending on the amount of resources (memory and CPU) on the client machine and the number of packets in the event there may be a performance lag when displaying only payloads in packet reconstruction. |
| Explanation |
To limit the amount of data processed in a reconstruction of a single event, change the Packets per Page setting in the footer to a lower value. |
| Behavior | When working in the Version 11.4 Events view, the Query Profile drop-down menu and Column Group drop-down menu do not function. |
| Issue | You do not have permission to read columns groups and profiles. The default column group , Summary List, is applied to the Events list, and you cannot change the column group, create a column group, or delete a column group. |
| Explanation | This occurs only when the administrator has created a custom role for you instead of assigning the default Analyst role. Ask your administrator to enable column group read and profile read permission for your role. |
| Issue | No matching Endpoint data available on Investigate > Events view > Host tab. |
| Explanation |
The Endpoint data may not be available due to any of the following:
|
Investigate Events Reporting Issues
Reporting Engine Service Unavailable
| Message | I see the following error message while generating a report: The Reporting Engine service may be offline or inaccessible. Try starting the service. |
| Issue |
This scenario occurs if the reporting engine service becomes offline or inaccessible due to the following reasons:
Note: The offline service will be shown (circle with red) indicator. |
| Explanation |
To fix the issue, you need to perform the following actions:
|
Insufficient Permissions
| Message | I see the following error message while generating a report: You do not have the required permissions to generate a report. Contact your administrator to request access. |
| Issue |
This scenario occurs when an analyst tries to create or schedule a report without the required permissions. Note: By default, only administrators will have access. |
| Explanation |
To fix the issue, analysts must contact their administrators to request access permissions. Once the permissions are provided, the analysts can try generating the report later. For more information, see Configure Data Source Permissions in the NetWitness Reporting Configuration Guide.
|
Unsupported Custom Columns
| Message |
I see the following error message while generating a report: The following custom columns are not supported to generate a report. Remove them and try again.
|
| Issue |
This scenario occurs because the Reporting Engine Service does not support a few custom columns for generating the report on the Investigate > Events page. |
| Explanation |
Perform the following actions to remove the custom columns from the list.
|
Datasource is not Configured or Unavailable
| Message |
I see the following error message while generating a report: The datasource <service name> is not configured in the Reporting Engine. Add the datasource and try again. |
| Issue |
This scenario occurs when you select a datasource that is not configured in the Reporting Engine. Note: When the name of the datasource on the Investigate page differs from the name on the reporting page, an error message is displayed. |
| Explanation |
To fix the issue, perform the following steps:
|
