Troubleshoot Log Collection

This topic describes the format and content of Log Collection Troubleshooting. NetWitness informs you of Log Collector problems or potential problems in the following two ways.

  • Log files.
  • Health and Wellness Monitoring views.

Junk Syslog Messages

The remote log collector has been made looser in regards to how it handles syslog messages. This was done to reduce the number of dropped messages due to missing parts of the header, or for other minor formatting errors. However, this might also allow syslog event messages that contain junk to get through the parser. If you see such messages in the system, you can add a syslog collection filter to remove events that are sending these messages.

Log Files

If you have an issue with a particular event source collection protocol, you can review debug logs to investigate this issue. Each event source has a Debug parameter that you can enable (set parameter to On or Verbose) to capture these logs.

Caution: Only enable debugging if you have a problem with this event source and you need to investigate this problem. If you have Debug enabled all the time it will adversely affect the performance of the Log Collector.

Health and Wellness Monitoring

Health and Wellness monitoring makes you aware of potential hardware and software problems in a timely manner so that you can avoid to outages. NetWitness recommends that you monitor the Log Collector statistical fields to make sure that the service is operating efficiently and is not at or near the maximum values you have configured. You can monitor the following statistics (Stats) described in the netwitness_adminicon_25x22.png (Admin) > Health & Wellness view.

Sample Troubleshooting Format

NetWitness returns the following types of error messages in the log files for.

Log Messages

timestamp failure (LogCollection) Message-Broker Statistics:...

timestamp failure (AMQPClientBaseLogCollection):...
timestamp failure (MessageBrokerLogReceiver):...

Possible Cause

The Log Collector cannot reach the Message Broker because the Message Broker:

  • has stopped running

  • has erroneous connection settings

Solutions
  1. <use the="the" systemctl="systemctl" command="command" on="on" console="console" to="to" check="check" status="status" of="of" message="message" broker="broker" shell="shell" console.="console.">returns the following if the message broker is not running:</use>

prompt$ systemctl status rabbitmq-server

rabbitmq start/running, process 10916

  1. Start the RabbitMQ Message Broker on event-broker node in the Explore view:

    netwitness_lc_troublshoot.png

Troubleshooting Logstash

Issue

Unable to re-configure or edit the custom configuration in Logstash.
Possible Cause

While re-configuring the events source configuration, if you update the Event source host's IP and save it, Logstash still connects to the old IP.

For example, suppose the Elasticsearch IP is not reachable due to some reason, and you updated the correct IP, but Logstash connects to the old IP (which is not reachable).
Solution

You must restart the Logstash service:

systemctl restart logstash

Issue

NetWitness Export connector UI does not validate user aggregation permissions.
Possible Cause

Users without aggregation permission are also allowed to configure NetWitness Export Connector in the UI.
Solution

Make sure you configure the NetWitness Export Connector Event source with the right aggregation permissions.

Issue

NetWitness Export Connector UI does not validate user credentials.
Possible Cause

NetWitness Export Connector UI displays test configuration successful even if you have configured Export Connector without appropriate aggregation permissions.
Solution

Make sure you configure the NetWitness Export Connector Event source using the correct credentials.

Issue

Logstash event source test configuration fails with memory allocation an error "There is insufficient memory for JVM".
Possible Cause

Sometimes test configuration fails due to insufficient memory available for JVM.
Solution

You must change the JVM -xms to a lower value at /etc/logstash/jvm.options. Perform the following:

  1. Go to /etc/logstash/jvm.options and change -Xms8g to a lower value as per the memory configuration of the appliance.

  2. Restart the Logstash service using the following command:
    service logstash restart

Issue

Issue with Logstash Performance.

Possible Cause

If EPS is more than 25K, you must increase JVM to get better performance.
Solution

You must change the JVM -xmx to a higher value at /etc/logstash/jvm.options. Perform the following:

  1. Go to /etc/logstash/jvm.options and update JVM to -Xmx32g.

  2. Restart the Logstash service using the following command:
    service logstash restart