Troubleshooting

This section provides information about possible issues when using NetWitness Endpoint.

Agent Communication Issues

Issue

Agent Last Seen Time column is not updated in the UI.
Explanation

The issue could be due to any one of the following:

  • Agent is inactive
  • Agent data is not processed if the Endpoint.Health.Overall-Health statistic shows Unhealthy due to which all the agent data including agent last seen time is not updated.
Resolution

See the resolution for these statistics in the Health and Wellness Issues section.

 

Issue

Agent is unable to communicate with the Endpoint Server.
Explanation

This could be due to one of the following reasons:

    •  

    • Agent is inactive.

 

  • Endpoint Server settings is incorrect in the agent packager or policy configuration, or not available for communication.
  • Endpoint Server or Nginx Server is not running .
  • Firewall or IP table rules are blocking the connection between the host and Endpoint Server.
Resolution
  • Check if the Endpoint Server and Nginx Server are reachable.
  • If the Endpoint Server settings are incorrect, uninstall the agent, download the agent packager, and reinstall the agent.
  • Update firewall or IP table rules, if required.
Issue

Agent takes a long time to scan.
Explanation

Sometimes, the NetWitness Endpoint scan takes a long time to complete. This is because of the CPU usage by other antivirus programs (such as Windows Defender, McAfee, Norton, and so on) that may be installed on the agent machines.
Resolution

It is recommended to whitelist the <service.exe> (name provided in the packager, by default, the service name is NWEAgent.exe) file in the antivirus suite.

 

Issue

You want to change the responsiveness of the Agent.
Explanation

Depending on your installation, you can adjust Beaconing intervals to change how responsive your agents are.
Resolution

If resources are not a concern, you can lower the HTTPS Beacon Interval and UDP Beacon Intervals. If resources are a concern and responsiveness of the agent is not, you can increase these intervals.

 

Issue Agent is unable to generate network tracking events in Insights mode.
Explanation Verify that Windows Management Instrumentation (WMI) service is running.
Resolution
  • Run Services.msc and look for Windows Management Instrumentation (WMI) service.
  • Go to properties and change the Startup type to Automatic.

Packager Issues

Message

Failed to load the client certificate.
Issue

Incorrect certificate password.
Explanation

While generating the agent installer, the certificate password does not match with the one provided while downloading the agent packager from the UI.
Resolution

Specify the correct certificate password.

Health and Wellness Issues

Endpoint Issues

Behavior The health check of the Endpoint.Health.Overall-Health statistic shows Unhealthy.
Issue

Endpoint Server service or required resources are not available or not in a usable state. This could be due to one of the following reasons:

  • Unable to forward Endpoint meta data to the Log Decoder.

  • Endpoint Log Hybrid disk usage reaches the specified limit.

  • Mongo DB is down or excessive read and write errors during processing.
Resolution

Disk Usage and Mongo Issues

Behavior

The health check of the Data.Application.Connection-Health Application, Data Store Disk Usage or Data Persistence for Endpoint Server shows Unhealthy.
Issue
  • Data.Application.Connection-Health Application or Data Persistence shows Unhealthy, if Mongo service is down or fails due to authentication.
  • Data Store Disk Usage shows Unhealthy, if Endpoint Server Mongo storage size has exceeded the threshold. By default, the server automatically delete the old data when it reaches 80% of the disk space.
Resolution
  • For Data.Application.Connection-Health Application or Data Persistence issue, you must check the Endpoint server logs (/var/log/netwitness/endpoint-server/endpoint-server.log) and Mongo logs (/var/log/mongodb/mongod.log), and:
    • If the issue is due to authentication, you must reissue the certificate. For more information, see "Service Certificate Reissue" section in the System Maintenance Guide.
    • If the issue is due to Mongo service is down, you must restart the Mongo.
  • For Data Store Disk Usage issue, you must increase the storage or configure data retention settings to clear the old data. For more information, see Configuring Data Retention Policy.

Log Decoder Issues

Behavior Endpoint metadata is not available in the Investigate > Navigate or Events view.
Issue

The health check of the Log Decoder Buffer and Meta Forward shows Unhealthy in the Health and Wellness.
Explanation

The issue could be due to any of the following reasons:

  • Log Decoder capture is not started.
  • Concentrator aggregation is not started.
  • Log Decoder connection issue.
  • Log Decoder buffer usage is beyond the specified limit.
Resolution

Make sure that:

  • Capture is enabled on the Log Decoder.
  • Aggregation is enabled on the Concentrator.
  • Meta forwarding is configured properly.

Note: Make sure Capture Autostart is enabled in the Service Config view for Log Decoder and Aggregate Autostart is enabled in the Service Config view for Concentrator.

File Log Policy Issues

Invalid Policy or Bad Connection Issues

Issue

Policies can be invalid for a variety of reasons. Some examples:

  • No sources found if the policy is enabled.
  • Invalid or missing typespec file
  • No destination is reachable for a file log policy event source type

Additionally, if capture is stopped on the destination Log Decoder, Endpoint Agents will send an error to the Endpoint Server saying that they failed to connect.

Also, if there is a lot of data to be processed for Agents collecting File data (when File Policy is enabled) , there is a possibility that Log Decoder buffer becomes full. If this happens, the Log Decoder cannot process any requests from the Agents communicating via EPS.
Explanation

The system is dynamic in nature, which means its state can change: event sources can lose their connection, typespec files can be altered or deleted, and other changes can occur that can invalidate a previously valid policy.
Resolution

To help identify the specific issue, check the log file on the Endpoint Server that reports the error:

/var/log/netwitness/endpoint-server/endpoint-server.audit.log

Relevant errors will be listed as FileLogError in the log file.

If you experience this issue, you can do the following:

  1. Try to identify and target higher-value data, thus limiting the total amount of data being processed.
  2. Enable throttling in the File policy to smooth out the peaks in usage.
  3. If you really do need to process more data on a regular basis, consider server-side hardware upgrades.

Reset File Collection Bookmarks

Issue

If the system is not configured correctly, NetWitness might collect logs and not be able to parse them. Or, files might get sent, but for some reason, not make it to the Log Decoder (for example if communication is via UDP and there is a network connectivity issue).

In these and other cases, you can reprocess these "missing" log files.
Explanation

For whatever reason, you may need to reprocess logs from the beginning of the file.
Resolution

Reset bookmarks for an event source type using the procedure described here: Reset File Collection Bookmarks.

Missing Log Collectors and Event Sources in the User Interface

Issue

Some log collectors or event sources seem to be missing from the list of available items.
Explanation

The Filter drop-down menus (types, log collectors, and log decoders) only show values that are in the event sources database, rather than all possible values. For example, if you have a log collector that has not yet collected any logs, then it is missing from the list.
Resolution

Collect logs from a specific log collector and event source, and then they should appear as items in the appropriate menu.

Relay Server Issues

Test Connection Issues

Issue

Relay Server test connection failed.
Resolution
  1. Check if the hostname or IP and port of the Relay Server are correct.
  2. Make sure that the hostname or IP of the Relay Server is resolvable from the Endpoint Server. Perform the following:
    1. In the Endpoint Log Hybrid console, verify if the Relay Server is reachable using the following command:
      nc -zvw3 <relayhost> <relayport>
      If the Relay Server is not reachable contact your Administrator.
    2. If the Relay Server is reachable, verify if the correct Relay Server installer is used by getting the Endpoint Server revision ID from the Relay Server host (/var/log/relay-install.log) and check the Endpoint Server RPM on Endpoint Log Hybrid using the following command:
      rpm -qa | grep <Endpoint Server Revision ID>
    3. Make sure if the Relay Server is installed and running.
      • Verify the Relay Server installation logs using the following command:
        /var/log/relay-install.log
      • Verify the status of Relay Server using the following command:
        systemctl status rsa-nw-relay-server

 

Issue

Test fails when installing relay server in cloud, using CentOS 7 configuration
Resolution
  1. Check if you have entered the suggested port numbers.
  2. If you have entered any other port number than the suggested one
  3. Make sure that the hostname or IP of the Relay Server is resolvable from the Endpoint Server. Perform the following:
    1. In the Endpoint Log Hybrid console, verify if the Relay Server is reachable using the following command:
      nc -zvw3 <relayhost> <relayport>
      If the Relay Server is not reachable contact your Administrator.
    2. If the Relay Server is reachable, verify if the correct Relay Server installer is used by getting the Endpoint Server revision ID from the Relay Server host (/var/log/relay-install.log) and check the Endpoint Server RPM on Endpoint Log Hybrid using the following command:
      rpm -qa | grep <Endpoint Server Revision ID>
    3. Make sure if the Relay Server is installed and running.
      • Verify the Relay Server installation logs using the following command:
        /var/log/relay-install.log
      • Verify the status of Relay Server using the following command:
        systemctl status rsa-nw-relay-server
   

 

Issue Relay Server installer generation fails with an error message ‘Unable to download the installer. Retry after sometime’.
Explanation Dependencies of the Relay Server are not resolved or downloaded completely.
Resolution

You must retry the download after 5-10 minutes. If the download still fails even after all dependencies are downloaded in the Endpoint Server, contact the NetWitness Customer Support.

Note: You can check ‘Finished downloading all Relay Server dependencies’ message in the Endpoint Server logs at /var/log/netwitness/endpoint-server/endpoint-server.log, to see if the dependencies are downloaded. If the download fails due to yum related issues, then you must clean yum repo using the command yum clean all and restart the Endpoint Server.

 

Issue

After the removal of DNSMasq in 12.1 and later versions, test connection fails between the Endpoint server and relay server.
Resolution

For the test connection to succeed, perform the following.

  1. SSH to Endpoint server.

  2. Edit the /etc/nginx/conf.d/relay.conf file and go to resolver nw-node-zero ipv6=off; line.

  3. Replace nw-node-zero with the nameserver IP or hostname.

  4. Run the following command to restart nginx service.

    systemctl restart nginx

  5. Try Test Connection again.

Installation Issues

Issue Relay Server installation fails due to missing or corrupted dependencies.
Resolution

Re-download the installer dependencies, perform the following:

  1. Go to netwitness_adminicon_25x22.png (Admin) > Endpoint Server service > select netwitness_actiondd.png > View > Explore.
  2. In the Endpoint server configuration, make sure endpoint.relay.installer.download-on-restart boolean is set to true (by default it is true).
  3. Restart the Endpoint server using the following command:
    systemctl restart rsa-nw-endpoint-server
    Fresh dependencies will be downloaded to the local directory in the Endpoint Server. This may take few minutes.
  4. Download the Relay Installer.
  5. Run the Relay Server Installation Script.
    For more information, see (Optional) Installing and Configuring Relay Server.

YARA Issues

Issue Failure in saving YARA configuration.
Explanation

The rule-folder path is under some Linux user home-directory such as /root or /home/user1.
Resolution

Choose other paths such as /var or /tmp etc. or change the owner of the directory to 'netwitness' user.

OPSWAT Issues

Files can not be scheduled for scan

Issue Some files cannot be scheduled for the scan. See Endpoint Configuration Guide > Troubleshooting
Explanation

The issue could be due to any one of the following:

1. One of the endpoint servers is either down or not responsive.

2. OPSWAT is not configured in one or more of the endpoint servers.
Resolution

1. Ensure all the endpoint servers are up and running.

2. Ensure OPSWAT is configured in all the endpoint servers.

OPSWAT not configured on all endpoint servers

Issue OPSWAT is not configured on appropriate Endpoint servers.
Explanation

The issue could be due to any one of the following:

1. OPSWAT is not configured in any of the endpoint servers.

2. One or more endpoint servers are down.
Resolution

1. Ensure OPSWAT is configured in all the endpoint servers.

2. Ensure all the endpoint servers are functioning.
   

OPSWAT scan results not getting updated

Issue OPSWAT scan is scheduled successfully, but no results are getting updated.
Explanation

The issue could be due to any one of the following:

1. Either you have selected unsupported file extensions, or some files are larger than the configured file size limit.

2. OPSWAT server is either down or not responsive.
Resolution

1. Ensure only the files with supported extensions are selected, and file size has not exceeded the configured maximum file size.

2. Contact OPSWAT support for additional information.