Warehouse Connector Configuration

How Warehouse Connector Works

Warehouse Connector collects meta and events from Decoder and Log Decoder and writes them in AVRO format into a Hadoop-based distributed computing system. You can set up the Warehouse Connector as a service on existing Log Decoders or Decoders.

The Warehouse Connector contains the following components:

  • Data Source
  • Destination
  • Data Stream

Data Source

Warehouse Connector collects data from the data source to store it in the destination. The supported data sources are Log Decoder and Decoder.

Destination

Destination is the Hadoop-based distributed computing system that collects, manages, and enables reporting on security data. The following are the supported destinations:

  • NetWitness Warehouse (MapR) deployments
  • HortonWorks Data Platform
  • Any Hadoop-based distributed computing system that supports WebHDFS or NFS mounting of HDFS file systems.
    Example: Commercial MapR M5 Enterprise Edition for Apache Hadoop

Data Streams

A data stream is a logical connection between the data source and destination. You can have multiple streams for different subsets of data collected. You can setup streams to segregate data from multiple Decoder and Log Decoder services. You can create a stream with a single data source and destination or with multiple data sources and a single destination.

The Warehouse Connector:

  • Aggregates session and raw log data from Decoders and Log Decoders.
  • Transfers the aggregated data to supported destinations like Hadoop based deployments.
  • Serializes the aggregated data that includes both schema and data into AVRO format.

Meta Filters

Meta filters enables you to filter the meta keys that should be written into the Warehouse. For more information, see Specify Meta Filters for a Stream.

Multi-Valued Meta Keys

NetWitness Warehouse supports multi-valued meta keys. The multi-valued meta keys is the meta field with the array type. You can use the meta keys library to determine the meta fields of type array and write HIVE queries with the correct syntax for arrays. By default, the following meta keys are treated as multi-valued and are defined in the file, multivalue-bootstrap.xml located at /etc/netwitness/ng in the Warehouse Connector:

  • alias.host
  • action
  • username
  • alias.ip
  • alias.ipv6
  • email

  • device.group

  • event.class

Checksum Validation

You can validate the file integrity of the AVRO files that are transferred from the Warehouse Connector to the data destinations. You need to enable checksum validation option when you configure the Warehouse Connector.

Lockbox Support

Lockbox provides an encrypted file that Warehouse Connector uses to store and protect sensitive data. You need to create the lockbox by providing a lockbox password while configuring the Warehouse Connector for the first time.

Note: You can orchestrate Warehouse Connector on a Log Decoder or a Decoder appliance.

The following is an overview on how to install and configure the Warehouse Connector service on Log Decoder or Decoder.

  • Configuring the Warehouse Connector service on NetWitness,
  • Configuring data sources,
  • Destinations,
  • Streams for Warehouse Connector,
  • Configuring alert notifications on NetWitness.

Note: RSA NetWitness Platform has added a Health & Wellness stat for Warehouse Connector to indicate the status of its Lockbox. Also, an out-of-the-box rule has been added so that a Health & Wellness alarm is raised when the Lockbox does not exist or cannot be opened.

netwitness_wcconfg_workflow_end.png

To install and configure the Warehouse Connector service, perform the following:

  1. Install Warehouse Connector Service on a Log Decoder or Decoder or Hybrid
  2. Configure a Warehouse Connector Service
  3. Configure the Data Source for Warehouse Connector
  4. Configure the Destination
  5. Configure a Stream
  6. Monitor a Warehouse Connector
  7. Add Warehouse as a Data Source to Reporting Engine
  8. Analyze a Warehouse Report
  9. Manage a Stream