This website uses cookies. By clicking Accept, you consent to the use of cookies. Click Here to learn more about how we use cookies.
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for
NetWitness Knowledge Base Archive
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for
- NetWitness Community
- NetWitness Knowledge Base Archive
- Panorama Z Connector
-
Options
- Subscribe to RSS Feed
- Bookmark
- Subscribe
- Printer Friendly Page
- Report Inappropriate Content
Panorama Z Connector
Article Number
000017073
Issue
Panorama Z Connector
What is the Panorama Z Connector and How do I use it?
What is the Panorama Z Connector and How do I use it?
Resolution
Introduction
------------
The Panorama Z Connector is an asynchronous message queue that accepts events from the NIC Collector, buffers them and then streams them to a target host. The Panorama Z connector was originally developed to send events to the Netwitness Panorama suite, however it is flexible enough to stream events to any intended target that can handle Syslog traffic (Another Envision appliance, Arcsight, etc.).
Installation
------------
For a given site, the Panorama Z Connector software should be run on each local collector for which you want to forward events. If you run the installer on the D-SRV, it installs the EDI portion instead of the NIC Collector interface. When you run the Panorama Z Connector installer on an ES appliance, it installs both the Panorama Z Connector software and the EDI code as it views an ES appliance as both an LC (local collector) and a D-SRV (Data Server).
The Panorama Z Connecter installer prompts for two values during the install process: destination IP address and TCP port number. The installer writes the values you enter into a file named analysis.rts.adapters.connector.ini that lives in the common storage directory under the path \config\collectoradapters\<nodename>. The default configuration file only contains endpoint information. If you want to make changes to the way that the Panorama Z connector streams data (IE. the format of the header, remove header entirely, etc.) you?ll have to edit the file and add the appropriate sections.
Configuration
-------------
Modifiable values for the .ini file include:
connector.event.format The default event format.
connector.exclude.ip Event source IPs to be excluded from the stream. (Empty means exclude nothing)
connector.exclude.type Envision device types to be excluded from the stream. (Empty means exclude nothing)
connector.channel.tcp.port The destination TCP port. (A value is required)
connector.channel.tcp.address The destination IP address. (A value is required)
The .ini file format is an XML structure that can be inferred from the property name. For example, to exclude events collected from IP address 1.2.3.4 in the stream, you would need to add an <exclude></exclude> container entry within the connector container and then add an <ip></ip> sub container entry for each IP address that you want to exclude.
<connector>
<channel>
<tcp>
<address>10.20.30.40</address> <!-- connector.channel.tcp.address -->
<port>514</port> <!-- connector.channel.tcp.port -->
</tcp>
</channel>
<exclude>
<ip>1.2.3.4</ip> <!-- connector.exclude.ip -->
</exclude>
</connector>
The Panorama Z Connector handles incoming event data and converts (if necessary) to an outgoing event text format. The outgoing format is configured using a format string that defines the order and the placement of the event components in the streamed output. The following markers are currently supported:
%ctime% Collection time in milliseconds since Unix epoch 1311904412000
%text% Event text. For example, "%PIX-1-101001: (PRIORITY) Failover cable OK"
%type% Envision device type ciscopix
%time% An RFC3339 timestamp 2011-07-29T01:53:32Z
%ip% Event source IP address> 1.2.3.4
%eid% enVision event identifier daf2d7d1-f58b-43fa-b4c3-d8ef1e17f8c9-0000012b299b5754-00000000
For example: if the event format is set to:
%time% [%ip%] %text%
an incoming event with the values shown in the table above will be streamed as:
2011-07-29T01:53:32Z [1.2.3.4] %PIX-1-101001: (PRIORITY) Failover cable OK Logging
In the configuration above, the "[" and "]" characters were added to the output string around the IP address variable as the intended target (an Envision appliance) expects to see the source IP address enclosed within brackets. They are not part of the normal variable declaration.
The following configuration forwards only the event text.
<connector>
<channel>
<tcp>
<address>10.20.30.40</address>
<port>514</port>
</tcp>
</channel>
<event>
<format>%text%</format>
</event>
</connector>
From the values shown in the table above, this would send:
%PIX-1-101001: (PRIORITY) Failover cable OK Logging
Typically, a value of ?%time% [%ip%] %text%? is a better choice for syslog forwarding. However, it is dependent on what the recipient expects to see in a forwarded event.
Troubleshooting
---------------
You can use NIC System message ID 400041 to monitor the operation of the connector. If required, you can also use the the keyword ZCONNECTOR to filter Panorama Z Connector specific messages from other NIC Collector messages that are issued with the same message IDs.
For example, TCP connection issues between the connector and the recipient in the last one day can be queried using the following lsdata command:
E:\nic\4000\AJDES-ES\bin>lsdata -time -1d now -devices nic(400041) -ms ZCONNECTOR -d 0
Oct 01 14:28:07 [10.100.226.55] %NIC-2-400041: Collector, Collector, -, -, -, -, Detail: 5792:
Error: ZCONNECTOR:5:1280003: Events transmission failed. Reason: A connection attempt failed
because the connected party did not properly respond after a period of time, or established
connection failed because connected host has failed to respond
Oct 01 14:28:07 [10.100.226.55] %NIC-2-400041: Collector, Collector, -, -, -, -, Detail: 5792:
Error: ZCONNECTOR:5:1280004: Event dispatch failed. Retry after (seconds) 60
See the connector log message catalog and the MessageIDs for other messages that are issued.
The Panorama Z Connector accepts incoming events as long as there is room in a buffer to hold them. If the network connection between the Envision collector and the recipient is broken, the defined buffers continue to enqueue for dispatch, but are never freed. If the connection is not re-establsihed quickly enough, the buffers will run out space to store incoming events. The Panorama Z Connector records the collection timestamp of the first event dropped and continues to drop subsequent events until one of the following two things happen:
- The connection is restored
- The Panorama Z Connector is shutdown
If the network breakage between Envision and the recipient is short-lived and the connection is restored, the buffers will free up and the Panorama Z Connector will once again start accepting events. The first event that is accepted after a drop triggers a gap notification". This notification compares the timestamp of the first dropped event to the timestamp of the first event accepted after a drop to determine the collection time window between which events were dropped.
If the NIC collector is brought down with undispatched events in the buffers, the Panorama Z Connector records the timestamp of the last event that was successfully sent and sets that as a bookmark. This bookmark is read by the Panorama Z Connector the next time it starts and is treated like the first dropped event timestamp.
Gap notifications are implemented as zero-byte files in a predefined folder that is monitored by the recovery process (EDI). The naming convention of these gap files encodes the time window information (i.e. starttime-endtime). The EDI process reads the gap notification to determine the events that need to be re-sent (were missed duirng the network outage).
This implementation guarantees that no event forwarded by the Panorama Z Connector is lost. However, this does not rule out the possibility that some events might be sent more than once. The Panorama Z connector is only responsible for the creation of gap notifications. The notifications are picked up by the EDI recovery process which queries the NIC Server for the missed events with the time bounds specified in the file name. NIC Server queries are at a minute granularity, whereas the NIC Collector timestamps recorded in the gap notifications have millisecond precision. This difference in precision may lead to a significant (depending on the EPS) number of events being sent more than once. The recipient must make use of the event identifiers to identify and remove duplicates.
------------
The Panorama Z Connector is an asynchronous message queue that accepts events from the NIC Collector, buffers them and then streams them to a target host. The Panorama Z connector was originally developed to send events to the Netwitness Panorama suite, however it is flexible enough to stream events to any intended target that can handle Syslog traffic (Another Envision appliance, Arcsight, etc.).
Installation
------------
For a given site, the Panorama Z Connector software should be run on each local collector for which you want to forward events. If you run the installer on the D-SRV, it installs the EDI portion instead of the NIC Collector interface. When you run the Panorama Z Connector installer on an ES appliance, it installs both the Panorama Z Connector software and the EDI code as it views an ES appliance as both an LC (local collector) and a D-SRV (Data Server).
The Panorama Z Connecter installer prompts for two values during the install process: destination IP address and TCP port number. The installer writes the values you enter into a file named analysis.rts.adapters.connector.ini that lives in the common storage directory under the path \config\collectoradapters\<nodename>. The default configuration file only contains endpoint information. If you want to make changes to the way that the Panorama Z connector streams data (IE. the format of the header, remove header entirely, etc.) you?ll have to edit the file and add the appropriate sections.
Configuration
-------------
Modifiable values for the .ini file include:
connector.event.format The default event format.
connector.exclude.ip Event source IPs to be excluded from the stream. (Empty means exclude nothing)
connector.exclude.type Envision device types to be excluded from the stream. (Empty means exclude nothing)
connector.channel.tcp.port The destination TCP port. (A value is required)
connector.channel.tcp.address The destination IP address. (A value is required)
The .ini file format is an XML structure that can be inferred from the property name. For example, to exclude events collected from IP address 1.2.3.4 in the stream, you would need to add an <exclude></exclude> container entry within the connector container and then add an <ip></ip> sub container entry for each IP address that you want to exclude.
<connector>
<channel>
<tcp>
<address>10.20.30.40</address> <!-- connector.channel.tcp.address -->
<port>514</port> <!-- connector.channel.tcp.port -->
</tcp>
</channel>
<exclude>
<ip>1.2.3.4</ip> <!-- connector.exclude.ip -->
</exclude>
</connector>
The Panorama Z Connector handles incoming event data and converts (if necessary) to an outgoing event text format. The outgoing format is configured using a format string that defines the order and the placement of the event components in the streamed output. The following markers are currently supported:
%ctime% Collection time in milliseconds since Unix epoch 1311904412000
%text% Event text. For example, "%PIX-1-101001: (PRIORITY) Failover cable OK"
%type% Envision device type ciscopix
%time% An RFC3339 timestamp 2011-07-29T01:53:32Z
%ip% Event source IP address> 1.2.3.4
%eid% enVision event identifier daf2d7d1-f58b-43fa-b4c3-d8ef1e17f8c9-0000012b299b5754-00000000
For example: if the event format is set to:
%time% [%ip%] %text%
an incoming event with the values shown in the table above will be streamed as:
2011-07-29T01:53:32Z [1.2.3.4] %PIX-1-101001: (PRIORITY) Failover cable OK Logging
In the configuration above, the "[" and "]" characters were added to the output string around the IP address variable as the intended target (an Envision appliance) expects to see the source IP address enclosed within brackets. They are not part of the normal variable declaration.
The following configuration forwards only the event text.
<connector>
<channel>
<tcp>
<address>10.20.30.40</address>
<port>514</port>
</tcp>
</channel>
<event>
<format>%text%</format>
</event>
</connector>
From the values shown in the table above, this would send:
%PIX-1-101001: (PRIORITY) Failover cable OK Logging
Typically, a value of ?%time% [%ip%] %text%? is a better choice for syslog forwarding. However, it is dependent on what the recipient expects to see in a forwarded event.
Troubleshooting
---------------
You can use NIC System message ID 400041 to monitor the operation of the connector. If required, you can also use the the keyword ZCONNECTOR to filter Panorama Z Connector specific messages from other NIC Collector messages that are issued with the same message IDs.
For example, TCP connection issues between the connector and the recipient in the last one day can be queried using the following lsdata command:
E:\nic\4000\AJDES-ES\bin>lsdata -time -1d now -devices nic(400041) -ms ZCONNECTOR -d 0
Oct 01 14:28:07 [10.100.226.55] %NIC-2-400041: Collector, Collector, -, -, -, -, Detail: 5792:
Error: ZCONNECTOR:5:1280003: Events transmission failed. Reason: A connection attempt failed
because the connected party did not properly respond after a period of time, or established
connection failed because connected host has failed to respond
Oct 01 14:28:07 [10.100.226.55] %NIC-2-400041: Collector, Collector, -, -, -, -, Detail: 5792:
Error: ZCONNECTOR:5:1280004: Event dispatch failed. Retry after (seconds) 60
See the connector log message catalog and the MessageIDs for other messages that are issued.
The Panorama Z Connector accepts incoming events as long as there is room in a buffer to hold them. If the network connection between the Envision collector and the recipient is broken, the defined buffers continue to enqueue for dispatch, but are never freed. If the connection is not re-establsihed quickly enough, the buffers will run out space to store incoming events. The Panorama Z Connector records the collection timestamp of the first event dropped and continues to drop subsequent events until one of the following two things happen:
- The connection is restored
- The Panorama Z Connector is shutdown
If the network breakage between Envision and the recipient is short-lived and the connection is restored, the buffers will free up and the Panorama Z Connector will once again start accepting events. The first event that is accepted after a drop triggers a gap notification". This notification compares the timestamp of the first dropped event to the timestamp of the first event accepted after a drop to determine the collection time window between which events were dropped.
If the NIC collector is brought down with undispatched events in the buffers, the Panorama Z Connector records the timestamp of the last event that was successfully sent and sets that as a bookmark. This bookmark is read by the Panorama Z Connector the next time it starts and is treated like the first dropped event timestamp.
Gap notifications are implemented as zero-byte files in a predefined folder that is monitored by the recovery process (EDI). The naming convention of these gap files encodes the time window information (i.e. starttime-endtime). The EDI process reads the gap notification to determine the events that need to be re-sent (were missed duirng the network outage).
This implementation guarantees that no event forwarded by the Panorama Z Connector is lost. However, this does not rule out the possibility that some events might be sent more than once. The Panorama Z connector is only responsible for the creation of gap notifications. The notifications are picked up by the EDI recovery process which queries the NIC Server for the missed events with the time bounds specified in the file name. NIC Server queries are at a minute granularity, whereas the NIC Collector timestamps recorded in the gap notifications have millisecond precision. This difference in precision may lead to a significant (depending on the EPS) number of events being sent more than once. The recipient must make use of the event identifiers to identify and remove duplicates.
No ratings
