Configure Hot, Warm, and Cold Storage

This topic provides instructions for Administrators on how to configure total hot, warm, and cold storage on an Archiver.

An Archiver host has hot storage pre-configured to the defaults. Administrators can configure total hot, warm, and cold storage to meet their specific business requirements. An Archiver must have total hot storage configured, but warm and cold storage configurations are optional. NetWitness does not manage cold storage.

Prerequisites

Ensure that you have:

  1. Installed the Archiver host in your network environment.
  2. Installed and configured Log Decoder in your network environment.
  3. Added Archiver as a Core service to your NetWitness deployment.
  4. Added Log Decoder services as a data source for Archiver.
  5. Installed and configured a DAC or other physical storage in your network environment.
  6. Determined your log retention and storage requirements.

Procedures

Configure Total Hot Storage for an Archiver

  1. Go to netwitness_adminicon_25x22.png (Admin) > Services, click netwitness_filericon.png, and select Archiver.

  2. In the Actions column, click netwitness_ic-actns.png > View > Config.

    The Services Config view of Archiver is displayed with General tab open.

  3. Click the Data Retention tab, in the Total Hot Storage section, click netwitness_ic-actns2_21x21.png to configure total hot storage.

    netwitness_121_cnfhotstorage_1122.png

  4. In the Hot Storage Mount Points dialog, add the mount points attached to the Archiver host that you want to include in Total Hot Storage.

    These are the paths to high performance storage, such as DAC storage and SAN. Do not add collections or subdirectories to the mount points.

    To add a mount point, click netwitness_ic-add.pngand type the path to the mount point.

    netwitness_addhotmp.png

  5. Verify that your mount point paths are correct and click Save.

    NetWitnesswill automatically create metadb, packetdb, sessiondb, and index directories for each collection defined on the Archiver:

    <storageLocation>/<CollectionName>/metadb
    <storageLocation>/<CollectionName>/packetdb
    <storageLocation>/<CollectionName>/sessiondb
    <storageLocation>/<CollectionName>/index

    For example, if your mount point is /var/netwitness/archiver, then the following directories will be created for each of your collections:

    /var/netwitness/archiver/<CollectionName>/metadb
    /var/netwitness/archiver/<CollectionName>/packetdb
    /var/netwitness/archiver/<CollectionName>/sessiondb
    /var/netwitness/archiver/<CollectionName>/index

    After the Archiver service is restarted, data will start being saved to your defined collections. Ensure that your log retention collections are correct before restarting the Archiver service.

Caution: After data has been saved to a mount point, it cannot be removed from the user interface.

Configure Total Warm Storage for an Archiver

(Optional) The procedure to configure Total Warm Storage for an Archiver is the same as for Total Hot Storage, except that you click netwitness_ic-actns2_21x21.png in the Total Warm Storage section and add the mount points that you want to use for warm storage, which are the physical paths to warm storage, such as Network Attached Storage (NAS).

netwitness_warmstor.png

Configure Total Cold Storage for an Archiver

(Optional) The procedure to configure Total Cold Storage for an Archiver is the same as for Total Hot Storage, except that you click netwitness_ic-actns2_21x21.png in the Total Cold Storage section and you add only one mount point for cold storage. NetWitness does not manage cold storage.

You must include the collection name format specifier %n somewhere in the cold storage mount point path name to avoid filename collisions between collections.

netwitness_addcoldstor.png

The following format specifiers are allowed in the path:

Format Specifier Description
%n collection name (required)
%y year the data moved to cold storage
%m month
%d day
%h hour
%##r block of hours for the current day. For example, if you want three 8 hour blocks, you can set it to %8r. The first 8 hours of the day returns 0, the second 8 hours returns 1, and last 8 hours of the day returns 2.

Changes take effect immediately.

For example, if you have a collection named compliance and you create the following cold storage path:

/sa-cold-storage/%n/%y-%m-%d/

NetWitness creates a directory each day with the following format:

/sa-cold-storage/compliance/2015-11-20/

Hot, Warm, and Cold Tier Storage Features

The following table describes features of the Hot, Warm, and Cold Tier Storage dialogs.

Feature Description
netwitness_ic-add.png Adds a mount point.
netwitness_ic-delete.png Removes a mount point. You cannot delete a mount point that is in use unless you delete the associated collections.
netwitness_iconcheckbox_20x20.png Select the mount points that you want to include for the Total Hot, Warm, and Cold Storage. You can only select one mount point for Total Cold Storage.
Mount Point

Shows the path to the attached physical storage. For example: /var/netwitness/archiver/database0, which is the location of the hot storage DAC.

Do not add collections or subdirectories to the mount points. NetWitness will automatically create metadb, packetdb, sessiondb, and index directories for each collection defined on the Archiver:

<storageLocation>/<CollectionName>/metadb
<storageLocation>/<CollectionName>/packetdb
<storageLocation>/<CollectionName>/sessiondb
<storageLocation>/<CollectionName>/index

For example, if your hot storage mount point is /var/netwitness/archiver, then the following directories will be created for each of your collections:

/var/netwitness/archiver/<CollectionName>/metadb
/var/netwitness/archiver/<CollectionName>/packetdb
/var/netwitness/archiver/<CollectionName>/sessiondb
/var/netwitness/archiver/<CollectionName>/index

For Cold Storage, you must include the collection name format specifier %n somewhere in the cold storage mount point path name to avoid filename collisions between collections.
Storage Size Shows the size of the attached storage. The Data Retention tab shows the total amount of storage for your reference.

Collections

The Collections section lists all of your storage collections along with Total Storage for Hot and Warm Storage.

netwitness_col1.png

Collections Features

The following table describes the icons and columns of the Collections section. You can hide some of the columns based on your requirements.

Feature Description
netwitness_add_21x21.png Opens the Collections dialog, in which you can add a storage collection.
netwitness_ic-delete.png Removes the selected collection. Deleting the collection permanently removes all stored data from the collection, but the empty data directories remain.
netwitness_ic-edit.png Opens the Collections dialog, in which you can edit the selected collection.
netwitness_ic-refresh_17x17.png Refreshes collection information.
netwitness_iconcheckbox_20x20.png Selects a collection. For example, you can select a collection for editing or removal.
Collection

Shows the name of your collection, such as Default, Compliance, MediumValue, and LowValue. You can create multiple collections with different criteria for retaining logs. If you do not create any collections, the Default collection is used.

If a collection has errors, the collection name and the columns with errors appear in red text.

Usage / Hot Storage Shows the current hot storage usage and the maximum hot storage for the collection. When the size of the logs reach the maximum hot storage amount, the logs are removed or they roll to the next available storage tier (warm or cold).
Usage / Warm Storage Shows the current warm storage usage and the maximum warm storage for the collection. When the size of the logs reach the maximum warm storage amount, the logs are removed or they roll to available cold storage.
Cold Storage Indicates whether cold storage is enabled or disabled. A solid colored green circle indicates that cold storage is enabled (netwitness_greencir.png). An blank white circle indicates that cold storage is disabled.
Retention Shows the number of days that logs are retained before being removed or optionally moved to cold storage. No Limit indicates that log retention is not restricted by a specified number of days.
For Hot and Warm Storage, size and retention period settings for a collection can override each other based on which criterion (size or time) is satisfied first.
Velocity (last hour) Shows the number of logs captured over the last hour.
Oldest Date Shows the date and time of the last log capture.
Duration Shows how may days ago that the last log was captured. For example: 20 days.
Compression Shows the compression type used for the meta and raw data in the collection.
Hash Shows whether hash is enabled or disabled. When enabled, the hash algorithm is used to ensure the data integrity of the files being saved. By default, the only data being hashed is raw logs and the hash files are saved in the same directory as data.
# of Rules

Shows the number of rules applied to the collection.
Define at least one rule for each collection. A collection without any associated rules shows a zero in red text as a warning: netwitness_redwarn.png The collection name also appears in red text, which indicates an error in the collection.

Caution: If a collection does not have a rule, no logs will ever go into that collection.
Actions Enables you to see the rules associated with a collection in the Retention Rule section when you select <actions button> > Select Rules. In the Retention Rule section, you can change the overall priority of the collection rules.
Total Storage Shows the current total hot storage usage and the maximum total hot storage at the bottom of the Usage / Hot Storage column. It also shows the current total warm storage usage and the maximum total warm storage at the bottom of the Usage / Warm Storage column.

Any errors in the collection appear in red text. A dotted underline indicates that a tooltip is available with information about the error.

netwitness_col2.png

Collections that have editing disabled (grayed out) also have tooltips that provide information on the problem.

Retention Rules

The Retention Rules section lists all of the retention rules used for your storage collections listed in the order of rule execution.

netwitness_retrl1.png

The following table describes the features of the Retention Rule section.

Feature Description
netwitness_add_21x21.png Opens the Rule Definition dialog, in which you can add a retention rule to use in a storage collection.
netwitness_delete_21x21.png Removes the selected retention rule. In order for your log collections to gather and store log data, you must associate them with at least one retention rule.
netwitness_edit_22x22.png Opens the Rule Definition dialog, in which you can edit the selected retention rule.
netwitness_ic-refresh_17x17.png Refreshes retention rule information.
netwitness_ic-up.png Move Up

Moves the selected retention rule up in the Retention Rule priority list. Retention Rule order is very important. NetWitness evaluates the the retention rules for all of the collections in numerical order by the number listed in the Order column in the Retention Rule section.

You can also use drag and drop to reorder retention rules.

netwitness_ic-down.png Move Down Moves the selected retention rule down in the Retention Rule priority list. Retention Rule order is very important. NetWitness executes the retention rules for all of the collections in numerical order by the number listed in the Order column in the Retention Rule section.
Apply Saves the rule order change.
netwitness_revert_17x17.png Revert Reverts the rule order change.
netwitness_iconcheckbox_20x20.png Selects or shows a selected retention rule.
Order Shows the order of a rule in the overall list of retention rules.
Rule Name Shows the name of rule, such as ComplianceDevices and GeneralWindowsLogs.
Condition

Shows the conditions for the rule. These conditions specify the type of logs to include in the collection.

Define Retention Rules presents the guidelines for all queries and rule conditions in Core services.
Collection Shows Collection name and how many days that the collection is retained. For example: MediumValue (30 Days)

Collection Dialog

On the netwitness_adminicon_25x22.png (Admin) > Services > Config view > Data Retention tab of an Archiver, administrators can define the criteria for log retention and storage. In the Collection dialog, which is accessible from the Collections section, you can define individual storage collections to use for different log types. For example, you may want to create collections for compliance reasons or to selectively retain critical logs.

Procedures related to this dialog box are described in Configure Archiver Storage and Log Retention and Configure Log Storage Collections.

To access the Collection dialog:

  1. Go to netwitness_adminicon_25x22.png (Admin) > Services, click netwitness_filericon.png, and select Archiver.

  2. In the Actions column, click netwitness_ic-actns.png> View > Config.

    The Services Config view of Archiver is displayed with General tab open.

  3. Click the Data Retention tab.
  4. In the Collections section, click netwitness_ic-add.png to add or edit the rule.
    The Collection dialog is displayed.

netwitness_arccoll_546x327.png

The following table describes the fields in the Collection dialog.

Field Description
Collection Name Specify a name for your collection, such as Compliance, MediumValue, or LowValue.
Hot Storage Specify the maximum size or percentage of hot storage to use for this collection. The free space available to use for hot storage and the total hot storage are shown next to this field.
When the size of the logs reach the maximum hot storage size, the logs are removed or they roll to the next available storage tier (warm or cold).
Warm Storage (Optional) Specify the maximum size or percentage of warm storage to use for this collection. The free space available to use for warm storage and the total warm storage are shown next to this field.
When the size of the logs reach the maximum warm storage size, the logs are removed or they roll to available cold storage.
Cold Storage (Optional) Specify whether to use cold storage for this collection. If you use cold storage for the collection, logs outside of the specified size and retention limits roll over to cold storage. If you do not use cold storage, logs outside of the specified size and retention limits are removed.
Retention (Optional) Specify the number of days that logs are retained before they are removed or rolled over to cold storage.
For Hot and Warm Storage, size and retention period settings for a collection can override each other based on which criterion (size or time) is satisfied first.
Compression Specify the type of compression to use for meta and raw logs in the collection. You can compress the meta and raw logs using GZIP or LZMA to save space. GZIP is very fast at compressing and decompressing, but it does not compress as well as LZMA. LZMA offers better compression at a cost of decompression speed (roughly three times slower than GZIP). Compression ratios are highly dependent on your data.
The default compression is GZIP.
Hash Specify whether to enable or disable hash. When enabled, the hash algorithm is used to verify the data integrity of the files being saved. By default, the only data being hashed is raw logs and the hash files are saved in the same directory as data.

Note: When decreasing collection storage allocations or lowering retention time, it may take several minutes to hours for the data to move and space to become available depending on the amount of moving (rolling) data. The default times are every 20 minutes for a size roll and every six hours for a time roll.​

Rule Definition Dialog

In the netwitness_adminicon_25x22.png (Admin) > Services > Config view > Data Retention tab of an Archiver, administrators can define the criteria for log retention and storage. In the Rule Definition dialog, which is accessible from the Retention Rules section, you can define retention rules to use for your storage collections.

Procedures related to this dialog box are described in Configure Archiver Storage and Log Retention and .Define Retention Rules

To access the Rule Definition dialog:

  1. Go to netwitness_adminicon_25x22.png (Admin) > Services, click netwitness_filericon.png, and select Archiver.

  2. In the Actions column, click netwitness_ic-actns.png> View > Config.

    The Services Config view of Archiver is displayed with General tab open.

  3. Click the Data Retention tab.

  4. In the Retention Rule section, click netwitness_ic-add.png or netwitness_ic-edit.png.

    The Rule Definition dialog is displayed.

    netwitness_ruledef.png

The following table describes fields in the Rule Definition dialog.

Field Description
Name Specify a unique name for your retention rule. For example: ComplianceDevices
Condition

Specify the conditions for the type of logs that you want to include in the collection.

All sting literals and time stamps must be quoted. Do not quote number values and IP addresses.

For example:

device.group='PCI Devices' || device.group='HIPPA Devices'
Collection Select the collection on which you want to apply this rule. For example: Compliance