Commands Used for Troubleshooting

NwConsole provides the following commands that are helpful when troubleshooting NetWitness:

  • whatIsWrong: Provides a snapshot of a service's configuration, stats, and failure and warning logs for a specified period of time in the past.
  • dbcheck: Performs consistency checking of database files.
  • topQuery: Helps pinpoint queries that are taking an excessively long time to run.
  • netbytes: Troubleshoots the network connections on the current host.
  • netspeed: Troubleshoots the connection between the host computer running NwConsole and the remote computer connected to it using the login command.

The following sections, as well as the NwConsole help and topic information (man) pages, provide additional information.

whatIsWrong

When a service is not working correctly, the reason is usually somewhere in the logs that the service has written. You can use the whatIsWrong console command to obtain a snapshot of a service's configuration, stats, and failure and warning logs (with surrounding context logs) for a specified past period of time, which defaults to the previous seven days. You can save the results of running whatIsWrong into a specified plain-text file. The output of this command can be a useful starting point to help determine what is currently wrong with a service.

To use the whatIsWrong console command, log on to the service to troubleshoot using the login command, and run the whatIsWrong command.

Note: Use help whatIsWrong to see all of the available parameters, including the number of days or hours to look back for events, the pathname to store results, whether or not to append or overwrite the results file, and the delimiter to use for log fields. You can also limit the number of most recent logs used to find context, and you can specify how many context logs per warning or failure log to retrieve.

Whenever you receive a request for logs for a Core service, you should run the whatIsWrong command first and use the results collected as a starting point.

dbcheck

The dbcheck command is used to perform consistency checking of database files (session, meta, packets, logs, stats, and so on). This might be necessary when a service cannot start because of errors in the consistency of the database files. Normally a service would automatically recover and correct any consistency issues on startup, but there are times when this does not occur. When a service starts (like Decoder), it typically does not read or open most database files in order to start quickly. It assumes most files are in a consistent state and only does a cursory check of the most recently written files. If there are problems, dbcheck can perform those consistency checks, but ONLY if the service is not running.

Caution: Do not attempt to run this command while a service is running.

For example, you can check a single file:

dbcheck /var/lib/netwitness/decoder/packetdb/packet-000000001.nwpdb

You can also use wildcards to check multiple files:

dbcheck /var/lib/netwitness/decoder/metadb/meta-00000002*.nwmdb

topQuery

The topQuery command can help pinpoint queries that are taking an excessively long time to run. This command parses the audit logs of a service and returns the top N longest running queries for the specified time period.

The easiest way to run it is to log on to the service (usually a Broker or Concentrator) and type topQuery. The default behavior is to return the top 100 longest running queries for the last seven days.

Type help topQuery for the list of parameters. Here are some additional examples with explanations:

  • topQuery hours=12 top=10
    This command returns the top 10 queries for the last 12 hours.
  • topQuery time1="2020-03-01 00:00:00" time2="2020-03-14 00:00:00"
    This command returns the top 100 queries between March 1, 2020 and March 14, 2020. Times are in UTC, not local.
  • topQuery input=/var/log/messages output=/tmp/top20.txt top=20 user=sauser1
    Instead of connecting to a service, it parses the Syslog audit messages for the top 20 queries in the last seven days, but only for queries executed by user sauser1. It writes the top 20 queries to /tmp/top20.txt instead of the console screen. The parameter user is a regex, so you can specify multiple usernames by writing something like user="(sauser1|sauser2)".

netbytes

The netbytes command is very useful for troubleshooting the network connections on the current host. It displays continuous send and receive statistics for all network interfaces. Once executed, you must press Ctrl-C to exit this command, which also exits NwConsole.

netspeed

The netspeed command is used to troubleshoot the connection between the host computer running NwConsole and the remote computer connected to it through the login command. You must supply the amount of bytes to transfer and it will time the speed of the connection. The netspeed command is very useful for troubleshooting aggregation performance issues that might be network related.

login somedecoder:50004 admin ...

netspeed transfer=4g

To troubleshoot the connection between a Concentrator and a Decoder, SSH into the Concentrator, run NwConsole, and then log on to the Decoder and run netspeed. The output from the command gives you an indication of the maximum network throughput. If it is much less than the standard 1 Gbps interface, it could indicate a network issue.