Streams Configuration
The Streams tab for a Warehouse Connector in the Services Config view provides a way to manage stream configuration.
Workflow
What do you want to do?
Role | I want to... | Refer to... |
---|---|---|
Administrator | Install Warehouse Connector Service on a Log Decoder or Decoder | Install Warehouse Connector Service on a Log Decoder or Decoder or Hybrid |
Administrator | Configure a Warehouse Connector Service | Configure a Warehouse Connector Service |
Administrator | Configure the Data Source for Warehouse Connector | Configure the Data Source for Warehouse Connector |
Administrator |
Configure the Destination using NFS, SFTP, WebHDFS. |
Configure the Destination Using NFS |
Administrator | Configure a Stream* | Configure a Stream |
Administrator | Monitor a Warehouse Connector* | Monitor a Warehouse Connector |
Administrator |
Add Warehouse as Data Source to Reporting Engine |
For more information, see "Add Warehouse as a Data Source to Reporting Engine" in the Reporting Engine Configuration Guide. |
Administrator | Analyze a Warehouse Report | For more information, see "Step 4. Analyze a Warehouse Report" in the Warehouse Guide. |
Administrator |
Manage a Stream and Lockbox |
*You can perform this task in the current view
Related topics
Quick Look
The following figure shows the Streams tab on the Warehouse Connector Services Config view.
The Streams tab allows you to perform the following:
Features | Description |
---|---|
Add a stream. | |
Delete a stream. | |
Edit the stream. | |
Start the stream. | |
Stop the stream. | |
Finalize the stream. | |
Reload the stream. If you have added a new meta or if a new meta is added as part of content update to any of the sources, Log Decoder or Decoder, you need to reload the stream for the meta to be visible in the schema for the Reporting Engine. Reloading a stream does not have any impact on the data, but only the new meta list is fetched from the sources. |
The following table describes the fields in the Streams tab:
Parameter | Description |
---|---|
Name | Name of the stream. |
Sources | The sources associated with the stream. |
Destination | The destinations associated with the stream. |
Status | Status of the stream. |
Stream Statistics
You can view the statistics of a configured stream. Click the icon next to the name of the stream.
The following parameters are displayed in the Stream Statistics:
Section | Parameter | Description |
---|---|---|
Stream | ||
Number of files in the permanent rejected folder | Determines the number of files in the permanent rejected folder (named, permfail) in the Warehouse Connector. The permanent rejected folder contains the files that Warehouse Connector failed to write to the destination. | |
Rejected folder usage in percentage | Determines the disk usage of the rejected folder. | |
Session Rate | Determines the rate at which the session is processed by the Warehouse Connector for the source. | |
Destination | ||
Status | Indicates the status of the destination. | |
Source | ||
Status | Indicate the status of the source. | |
Sessions Behind | Determines that number of sessions that needs to be processed by the Warehouse Connector. | |
Last Session read | Determines the last session id processed by the Warehouse Connector. |
Change Stream Configuration
You can change configuration of a stream in runtime. In the Streams tab, click Change Configuration to change the configuration of the selected stream.
You can change the following parameters of the Stream Configuration:
Note: If you change the value of any parameter in stream configuration, make sure that you restart the stream.
After upgrading, if the values of Maximum Message Hold Count, Maximum Message Hold Interval and Maximum Message Hold Size are 3000000, 60 and 128 respectively, ensure that you assign the following values to the streams:
- Maximum Message Hold Count - 2400000
- Maximum Message Hold Interval - 600
- Maximum Message Hold Size - 512
You can assign these values by modifying the existing Stream configuration.
Parameter | Description | ||||||||
---|---|---|---|---|---|---|---|---|---|
Aggregation Configuration |
|||||||||
Aggregate max sessions | Determines the maximum number of sessions in a response for an aggregation request from the Warehouse Connector to the source . | ||||||||
Aggregation Interval | Determines the time between the responses from the source. | ||||||||
Loader Settings | |||||||||
Compress files on disk | Enable to compress files on disk. Supported values:
|
||||||||
Export Rollup Interval | Determines the roll-up interval for export files and also the directory structure the Warehouse Connector writes to the destination. For example: If the parameter is set to:
Recommended value is hour. |
||||||||
Maximum Message Hold Count | Determines the maximum number of sessions to store in the memory before processing.
Note: If you have deployed a Warehouse Connector Virtual Appliance, make sure that you change the default value of the parameter to 800000. |
||||||||
Maximum Message Hold Interval (Seconds) | Determines the maximum time (in seconds) to hold the sessions in memory before processing. | ||||||||
Maximum Message Hold Size | Determines the maximum size for the sessions to store in the memory before processing. | ||||||||
Remote Export Path | Determines the remote local mount point for HDFS (nfs://) and the location to export the data. | ||||||||
Page Size |
Determines the maximum pages. |
||||||||
Stream Settings | |||||||||
Auto Startup | Enable to automatically start the stream whenever the Warehouse connector process is restarted. By default, the parameter is set to off. |