Advanced NetWitness Configuration

Grok Filter Plugin

You can use Grok to parse incoming logs and extract valuable meta information. For details, see Grok filter plugin: match. The meta extracted using grok is part of the full event package sent to NetWitness, where it can be mapped to NetWitness meta.

Resources:

Logstash Input and Filter plugins

See the following URLs:

You can use the beats input plugin for Logstash to receive events from beats sources, including Filebeat & Auditbeat: Beats Input Plugin

You can use the drop filter plugin for Logstash to filter out unwanted logs being passed through Logstash: Drop Filter Plugin

Filter out unwanted logs

You can also filter out unwanted logs using the drop-event processor for Filebeat & Auditbeat.

Configure heartbeat plugin to send test logs to NetWitness

The Heartbeat plugin can be used to send a test message to verify connectivity between logstash and NetWitness. For details, see the Heartbeat Plugin configuration guide: Heartbeat Input Plugin.

Example of Heartbeat Plugin

...

heartbeat {
  id => "sample_plugin"
  #interval => 60
  count => 3
  message => "sequence"
  add_field => {
  "[@metadata][nw_type]" => "logstash_testlog"
  "msg" => "This is test log from some eventsource"
}

...

Configure logstash to persist events in case of failure (Recommended)

By default, Logstash uses in-memory queues to buffer events. The size of these in-memory queues is not configurable. If there is a machine failure, or if the service is forcibly stopped, the contents of these queues are lost. To protect against data loss in these situations, Logstash supports persistent queues that are stored on disk and thus can survive failures. For details, see Logstash Persistent Queues.

NetWitness Codec Advanced Configuration

By default, the NetWitness codec sends the complete JSON event as payload to the NetWitness Log Decoder. If the NetWitness nw_type device parser type has a custom payload format, you must configure the NetWitness codec plugin to recognize this custom format.

The payload_format and payload_format_failover mappings use nw_type as the key. The payload_format mapping is searched first for the device type (nw_type). If the device type is not set, or no format is specified for nw_type, or the system fails to make all configured variable substitutions, the complete JSON output is used as the payload. If the primary format from the payload_format mapping fails, the payload_format_failover mapping is tried. If that also fails, the complete JSON output is used as the payload. You can use The format can use Logstash event field syntax for this custom configuration.

The following code snippet shows example of adding meta with custom payload formats.

Add meta with custom payload formats

output {
   if [@metadata][nw_type] { # Only targeted NetWitness items
   tcp {
      id => "netwitness-tcp-output-conf-output"
      host => "127.0.0.1"
      port => 514
      ssl_enable => false
      codec => netwitness {
         # Payload format mapping by nw_type.
         # If nw_type is absent or formatting fails,
         # JSON event is used as the payload
         payload_format => {
            "apache" => "%APACHE-4-%{verb}: %{message}"
         }
         # Failover format, if above format fails
         # If nw_type is absent or formatting fails,
         # JSON event is used as the payload
         payload_format_failover => {
            "apache" => "%APACHE-4: %{message}"  # When verb is missing
            }
         }
      }
   }
}

Troubleshoot Installation Issues

If you encounter any issues during installation of Logstash, see Logstash Installation and Setup.

Configure NetWitness to Collect Events

You need to start capture on the Log Decoder to which your are sending your Logstash data.

To start or restart network capture on a Log Decoder:

  1. Log in to NetWitness and click the ADMIN icon > Services.
    The Services view is displayed.
  2. Select a Log Decoder service.
  3. Under netwitness_actions_button.png (actions), select View > System.

  4. In the toolbar, click netwitness_startcapturedr.png.

    Note: If the toolbar is displaying the Stop Capture (netwitness_stopcapture.png) icon, then capture has already been started.

By default, Log Decoders support events that are up to 32 KB in size. If your events are getting truncated on the Log Decoder, use the following procedure to change the event size:

  1. Change LogDecoder REST config at http://LogDecoder_IP:50102/decoder/config, where LogDecoder_IP is the IP address of your Log Decoder.
  2. Set pool.packet.page.size to 64 KB.
  3. Restart the Log Decoder: this is required after you change the pool.packet.page value.

Note: If you are collecting events larger than 64 KB in size, follow instructions above in the Filter out unwanted logs section. You can drop unwanted logs or fields for a specific event source, to reduce the size of the incoming data.