Configure the Destination Using SFTP

Make sure that you have:

  • Installed the Warehouse Connector service or virtual appliance in your network environment.
  • Added the Warehouse Connector service to NetWitness. For more information, see the "Add a Service to a Host" in the Hosts and Services Getting Started Guide.
  • For the SFTP destination type, the destination host should be listed in the /root/.ssh/known_hosts file used by the ssh service (for example, sshd) running on the Warehouse Connector.

Add Destination from Warehouse Connector Host

To add the destination host to the /root/.ssh/known_hosts file, from the Warehouse Connector host, initiate a secure connection to the destination host:

  1. Log in to the Warehouse Connector.
  2. Enter ssh root@<SAWIP> or ssh username@<SAWIP>.
  3. Select Yes and enter the password.
  4. Add the host key in the /root/.ssh/known_hosts file

     

    Note: After you upgrade Warehouse Connector to 11.0, you must make sure that the destination host is listed in the /root/.ssh/known_hosts file used by the SSH service (i.e. sshd) running on the Warehouse Connector. If you do not perform this action, the streams configured with SFTP in Warehouse Connector will not start.

 

  • If you want to use SFTP to write data into the destination using SSH key-based access, you need to configure SSH key-based access between the Warehouse Connector and the Warehouse host or Hadoop node. For more information, see Configure SSH Keys below.

    Note: If you want to enable checksum validation to validate the integrity of the AVRO files that are transferred from the Warehouse Connector to the destinations, make sure that you generate the keys without setting the passphrase and do a key exchange between Warehouse Connector and the warehouse nodes.

Configure SSH Keys

To configure ​SSH key-based access between the Warehouse Connector and the Warehouse host or Hadoop node:

  1. Generate SSH keys on the Warehouse Connector at the default location. Perform the following:

    1. SSH to the Warehouse Connector.

    2. Type the following command and press ENTER:

      $ OWB_FORCE_FIPS_MODE_OFF=1 ssh-keygen -t ecdsa -b 521

    3. The command prompts you to enter the file in which to save the generated key.

      Enter file in which to save the key (/root/.ssh/id_ecdsa):

    4. Enter the file in which you want to save the key and press ENTER.

      The command prompts you to enter and confirm the passphrase.

      Note: If you want to enable checksum validation to validate the integrity of the AVRO files that are transferred from the Warehouse Connector to the destinations, make sure that you do not set the passphrase​. Then, the below steps e, f, g, and h are not applicable.

      Enter passphrase (empty for no passphrase):
       Enter same passphrase again:

      The public key is generated and is saved in the location that you provided.

      Note: If the SSH key is not generated in the default location (/root/.ssh/id_ecdsa), you need to configure the destination for warehouse connector through Explore view. For more information, see To configure the destination through Explore view:.

    5. Change the directory by entering the following command: 
      cd /root/.ssh/

    6. Move the generated key to the below location:

      mv ~/.ssh/id_ecdsa ~/.ssh/id_ecdsa.old

    7. Type the following command and press ENTER:

      $ OWB_FORCE_FIPS_MODE_OFF=1 openssl pkcs8 -topk8 -v2 des3 -in id_ecdsa.old -out id_ecdsa

      The command prompts you to enter and confirm the passphrase.

    8. Enter the encryption passphrase.

    9. Run the following command to change the file permission:

      chmod 600 ~/.ssh/id_ecdsa
    10. Copy the generated public key to append to the remote Warehouse host or Hadoop node.
      ssh-copy-id -i ~/.ssh/id_ecdsa.pub root@<destination host ipaddress>

  2. SSH to remote Warehouse host or Hadoop node as "ssh '<user>@<ip address>", if identity key file is at default location.
    or
    SSH to remote Warehouse host or Hadoop node as "ssh '<user>@<ip address> -i <identity file path>", If identity key file is not at default location.

  3. Append the generated public key to the remote Warehouse host or Hadoop node's authorized keys list located at ~/.ssh/authorized_keys.

    Note: Make sure that you copy the public keys to the Hadoop node and while copying the public key ensure that you provide the login details of the user using which the WebHDFS destination would be added.

You can now securely communicate between Warehouse Connector and Warehouse nodes or Hadoop nodes.

Configure Warehouse Connector to use SFTP destination

Note: If the SSH key is not generated in the default location (/root/.ssh/id_ecdsa), you need to configure the destination through Explore view. For more information, see To configure the destination through Explore view:.

To configure the destination through User Interface:

    1. Log on to NetWitness
    2. Go to netwitness_adminicon_25x22.png (Admin) > Services.
    3. In the Services view, select the added Warehouse Connector service, and select netwitness_actions_icon.png > View > Config.
      The Services Config view of Warehouse Connector is displayed.
      121_WC_services_config_view_1122.png
    4. On the Sources and Destinations tab, in the Destination Configuration section, click netwitness_add.png.
    5. In the Add Destination dialog, select SFTP from the Type drop-down list.
      netwitness_wc_add_sftp_dest.png
    6. In the Name field, enter a unique symbolic name for the destination.

      Note: The Name field does not support spaces or special characters except underscore (_).

    7. In the Host field, enter the remote server IP address.
    8. In the Port field, retain the default port, 22.
    9. In the Username field, enter the SSH username.

Note: In the case of HortonWorks HD, ensure that the username is gpadmin and for password based access the password for gpadmin should be used. For passphrase-based access, the passphrase used to generate the keys for gpadmin user should be used.

  1. In the Password/Passphrase field, enter one of the following:
    • SSH password - If you are using SFTP to write data into the destination using password-based access.
    • SSH passphrase - If you are using SFTP to write data into the destination using SSH key-based access.
  2. In the Remote Path field, enter the path of the directory present on the SFTP server.
  3. Click Save.
  4. (Optional) If you want to enable checksum validation, perform the following:
    1. Go to netwitness_adminicon_25x22.png (Admin) > Services.
    2. In the Services view, select the added Warehouse Connector service, and select netwitness_actions_icon.png > View > Explore.
      The Explore view of Warehouse Connector is displayed.
      121_WC_explore_view_1122.png
    3. In the options panel, navigate to warehouseconnector/destinations/sftp/config​.
    4. Set the parameter isChecksumValidationRequired to 1.
      121_sftp_config_param_update_1122.png
    5. Restart the respective stream.

To configure the destination through Explore view:

  1. Go to netwitness_adminicon_25x22.png (Admin) > Services.

  2. In the Services view, select the added Warehouse Connector service, and select netwitness_actions_icon.png > View > Explore.
    The Explore view of Warehouse Connector is displayed.
    121_WC_explore_view_1122.png

  3. Right click on "warehouseconnector" node and select properties.
    121_Exploreview_WHC_1122.png

  4. Select "add" property and manually enter the below config parameters.
    name=<destination name> destination=sftp://<destination path> host=<destination host ipaddress:port> type=hdfs port=22 username=<username> password=<password> privKeyFile=<private key file path>

Aggregate Metas and Raw Logs for a Log Session

To aggregate raw logs and metas from Log Decoder into a single AVRO file instead of two folders.

  1. Go to ADMIN > Services.

  2. Select a Warehouse Connector service and click netwitness_actions_icon.png > View > Explore.
    The Explore view for the Warehouse Connector is displayed.
  1. Open warehouseconnector/streams/<stream name>/loader/config and in the right pane, select the export.logAndsession.avro.enabled parameter.
  2. Change the value to yes.
  3. Restart the service.
  4. Go to ADMIN > Services.
  5. Select a Warehouse Connector service and click netwitness_actions_icon.png > View > Config.

  6. On the Streams tab, select the stream that you want to reload.

  7. Click Reload.