Configure File Event Sources in NetWitness

This topic tells you how to configure the File collection protocol:

Note: In NetWitness 11.4 and later, you can perform File Log collection for many event sources using Endpoint Agents, thus simplifying the collection process. For details, see the NetWitness Endpoint Configuration Guide. For a list of which event sources are supported, see the section "Currently Supported File Log Event Source Types."

Configure a File Event Source

To configure a File Event Source:

  1. Go to netwitness_adminicon_25x22.png (Admin) > Services from the NetWitness menu.
  2. Select a Log Collection service.
  3. Under Actions, select netwitness_ic-actns.png > View > Config to display the Log Collection configuration parameter tabs.

  4. Click the Event Sources tab.

    12.1_chooseCollectionMethod_1122.png

  5. In the Event Sources tab, select File/Config from the drop-down menu.

  6. In the Event Categories panel toolbar, click netwitness_ic-add.png.

    The Available Event Source Types dialog is displayed.

  7. Select a file event source type and click OK.

    The newly added event source type is displayed in the Event Categories panel.

  8. Select the new type in the Event Categories panel and click netwitness_ic-add.png in the Sources toolbar.

    The Add Source dialog is displayed.

  9. Add a File Directory name and modify any other parameters that require changes. For details, see File Collection Parameters below.

  10. To get the public key and enter it into the dialog box, do the following:

    1. Select and copy the public key from the Event Source by running: cat ~/.ssh/id_rsa.pub
    2. Paste the public key in the Eventsource SSH Key field.
  11. Click OK.

You need to restart file collection for your changes to take effect.

Stop and Restart File Collection

After you add a new event source that uses file collection, you must stop and restart the NetWitness File Collection service. This is necessary to add the key to the new event source.

File Collection Parameters

The following table provides descriptions of the File Collection source parameters.

The following table describes the Basic configuration parameter for File collection.

Note: Required parameters are marked with an asterisk. All other parameters are optional.

Name Description
File Directory*

Collection directory (for example, Eur_London100) into which the File event source places its files. Valid value is a character string that is conforms to the following regular expression:

[_a-zA-Z][_a-zA-Z0-9]*

This means that the file directory must start with a letter followed by numbers, letters, and underscores. Do not modify this parameter after you start collecting event data.

After you create the collection, the Log Collector creates the work, save, and error sub-directories under the collection directory.
Address* IP address of the event source. Valid value is an IPv4 address, IPv6 address, or a hostname including a fully-qualified domain name.
File Spec Regular expression. For example, ^.*$ = process everything.
File Encoding

Character encoding used by the syslog senders to this port. Defaults to UTF-8.

Note: It is safe to leave this as UTF-8, since UTF-8 handles ASCII characters as well, and most senders have their encoding set to UTF-8.

NetWitness has tested the following values:

  • EUC-KR
  • SJIS
  • GB3212/GBK
  • ISO_8859-1 (German)
  • ISO_8859-7 (Greek)

Cancel

Closes the dialog without making adding an event source type.

OK

Adds the parameters for the event source.

The following table describes the Advanced configuration parameter for File collection.

Name Description
Ignore Encoding
Conversion Errors

Select the check box to ignore encoding conversion errors and ignore invalid data. The check box is selected by default.

Caution: This may cause parsing and transformation errors.
File Disk Quota

Determines when to stop saving files regardless of the Save On Error and Save On Success parameter settings. For example, a value of 10 indicates that when there is less than 10% available disk left, the Log Collector stops saving files to reserve enough space for your estimated normal collection processing.

Caution: Available disk refers to a partition where the base collection directory is mounted. If the Log Decoder server has a 10TB disk size and 2TB is allocated to base collection directory, then setting this value to 10 causes log collection to stop when less than 0.2TB (10% of 2TB) of space is left. It does not mean 10% of 10TB.

Valid value is a number in the 0 to 100 range. 10 is the default.
Sequential Processing

Sequential processing flag:

  • Select the check box (default) to process event source files in collection order.
  • Do not select the checkbox to process event source files in parallel.
Save On Error Save on error flag. Check the checkbox to retain the eventsource collection file when the Log Collector it encounters an error. The check box is selected by default.
Save On Success Save eventsource collection file after processing flag. Check the checkbox to save the eventsource collection file after processing it. The check box is not selected by default.
Eventsource SSH Key

SSH public key used to upload files for this event source. Please refer to the Generate Key Pair on Event Source and Import Public Key to Log Collector section in the Install and Update the SFTP Agent Guide for instructions on generating keys.

Note: If File collection is stopped, NetWitness does not update the authorized_keys file with the SSH public key that you add or modify in this parameter. You must restart File collection to update the public key.
You can add or modify the value of the public key in this parameter in multiple File event sources without File collection running, but NetWitness will not update the authorized_keys file until File collection is restarted.
Manage Error Files

By default, the Log Collector uses the File Disk Quota parameter to ensure that the disk does not fill up with error files. If you set this parameter to true, you can specify one of these:

  • Maximum space allotted to error files in the Error Files Size parameter.
  • Maximum number of error files allowed in Error Files Count parameter.

A reduction percent is also specified, which tells the system how much to reduce when the maximum is reached.

Select the check box to manage error files. The check box is not selected by default.
Error Files Size

Only valid if the Manage Error Files and Save On Error parameters are set to true.
Specifies to what extent NetWitness saves error files. The value that you specify is the maximum total size of all the files in the error directory.

Valid value is a number in 0 to 281474976710655 range. You specify these values in either Kilobytes, Megabytes, or Gigabytes. 100 Megabytes is the default. If you change this parameter, the change does not take effect until you restart collection or restart the Log Collector service.
Error Files Count

Only valid if the Manage Error Files and Save On Error parameters are set to true. Maximum number of error files allowed in the error directory. Valid value is a number in 0 to 65536 range. 65536 is the default.

If you change this parameter, the change does not take effect until you restart collection or restart the Log Collector service.
Error Files Reduction %

Percent amount by size or count of the error files that the Log Collector service removes when the maximum size or count has been reached. The service removes the oldest files first.

Valid value is a number in the 0 to 100 range. 10 is the default.
Manage Saved Files

Select the check box to manage saved files. The check box is not selected by default.
By default, the Log Collector uses the File Disk Quota parameter to ensure that the disk does not fill up with saved files. If check this check box, you can specify one of these:

  • Maximum space allotted to saved files in the Saved Files Size parameter.
  • Maximum number of saved files allowed in Saved Files Count parameter.

A reduction percent is also specified, which tells the system how much to reduce when the maximum is reached.
Saved Files Size

Only valid if the Manage Saved Files and Save On Success parameters are set to true.
Maximum total size of all the files in the save directory. Valid value is a number in the 0 to 281474976710655 range. You specify these values in either Kilobytes, Megabytes, or Gigabytes. 100 Megabytes is the default.

If you change this parameter, the change does not take effect until you restart collection or restart the Log Collector service.
Saved Files Count

Only valid if the Manage Saved Files and Save On Success parameters are set to true. Maximum number of saved files in the save directory. Valid value is a number in 0 to 65536 range. 65536 is the default.

If you change this parameter, the change does not take effect until you restart collection or restart the Log Collector service.
Saved File Reduction %

Percent amount by size or count of the saved files that the Log Collector service removes when the maximum size or count has been reached. The service removes the oldest files first.

Valid value is a number in the 0 to 100 range. 10 is the default.
Debug

Caution: Only enable debugging (set this parameter to On or Verbose) if you have a problem with an event source and you need to investigate this problem. Enabling debugging will adversely affect the performance of the Log Collector.

Enables or disables debug logging for the event source.
Valid values are:

  • Off = (default) disabled
  • On = enabled
  • Verbose = enabled in verbose mode ‐ adds thread information and source context information to the messages.

This parameter is designed to debug and monitor isolated event source collection issues. The debug logging is verbose, so limit the number of event sources to minimize performance impact.

If you change this value, the change takes effect immediately (no restart required).