Configure Archiver Storage and Log Retention

This topic provides instructions for Administrators to configure storage and log retention on an Archiver.

For compliance reasons, it is often necessary to retain some logs longer than other logs. Some logs are legally sensitive and cannot be retained for a long period of time. Other logs have a requirement to be retained for years. In addition to compliance, some logs are useful for historic forensics and other logs have little to no security or operationally relevant value and can be deleted after a short time.

Because business requirements vary, NetWitness enables you to configure Collections, which are log retention sets for storing log data. For each collection, you can specify how much of the total storage space to use and how many days to retain the logs in the collection. To specify the type of logs to put in the collection, you define retention rules to associate with the collections. Retention rules for all of your collections execute sequentially in an order that you define.

To do this, you must first define the total physical storage space for your collections. NetWitness enables you to define three types of storage:

  • Hot Tier Storage: This storage contains log data that is in active use as part of the business process. Users can access these logs faster than other types of storage and they can use these logs for reporting and other tasks. Hot storage is usually Direct-Access Capacity (DAC) or SAN storage.
  • Warm Tier Storage: (Optional) This storage contains older log data aggregated by Archiver. Log data access is slower than hot storage. Users can also use these logs for reporting and other tasks. Warm storage is usually Network Attached Storage (NAS).
  • Cold Tier Storage: (Optional) This storage contains the oldest log data that is either required for the operation of the business or mandated by regulatory requirements. The logs are offline and Archiver cannot access these logs for reporting or other tasks. However, if you want to access this log data, you can restore it to the collections created on the Archiver service and then use it for reporting. Cold storage is usually offline storage, such as NAS, or temporary storage before archiving to tape. Once data moves to the Cold Tier, that data is no longer managed by Archiver. Once moved, it is incumbent on external processes to back it up or manage that Cold Tier space such that it does not reach 100% capacity. If capacity is reached, this will cause the Archiver to stop aggregation until the problem is fixed.

Archivers are preconfigured to use available hot storage and a default log collection, so you do not have to configure Archiver storage and log retention if you do not have complex log retention requirements.

Logs can move from one type of storage to another in the following ways:

  • Hot Storage > Cold Storage
  • Hot Storage > Warm Storage > Cold Storage

netwitness_hotwarmcold_250x141.png

When a collection reaches its retention limits for hot and warm storage, NetWitness deletes the log data from hot or warm storage. With cold storage configured, a copy goes into cold storage before the logs are deleted from hot or warm storage. For example, if you have a collection with Hot Storage of 1 TB, Warm Storage of 1 TB, and Cold Storage enabled, when the log data reaches 1 TB of hot storage, the oldest log data moves to warm storage. When the log data in warm storage reaches 1 TB, the oldest log data from warm storage is copied to cold storage before it is removed from warm storage.

For Hot and Warm Storage, size and retention period settings for a collection can override each other based on which criterion (size or time) is satisfied first. For example, if you have a collection with Hot Storage of 1 TB, no Warm or Cold Storage, and a Retention period of 20 days, if the Log data exceeds 1 TB after 11 days, the oldest logs over 1 TB are deleted even though the collection has a 20 day retention period.

After you create hot, warm, and cold storage, you configure your log retention storage collections. You can specify the maximum size of the Hot and Warm Storage for the collection, whether to use Cold Storage, the number of days to retain the logs in the collection, the data compression, and whether to use a hash algorithm to be able to verify the data integrity of the files being saved.

After configuring your collections, you define retention rules for your collection. These rules specify the type of logs to be stored in the collection. Each collection must have at least one retention rule associated with it in order to store log data.

Procedure

Perform the following tasks in the order shown to configure storage and log retention.

Task Reference
1. Configure total hot, warm, and cold storage. Refer to Configure Hot, Warm, and Cold Storage.
2. Configure log retention storage collections. Refer to Configure Log Storage Collections.
3. Define retention rules for the collections and determine the order of execution of the overall list of retention rules. Refer to Define Retention Rules.