Enable and Configure the Entropy ParserEnable and Configure the Entropy Parser
Beginning with NetWitness 11.0, the administrator can configure a Decoder to use a NetWitness native parser, known as the Entropy parser. When the Entropy parser is enabled, analysts have visibility into channels that are trying to blend in with other traffic, but do not follow normal protocol behavior. This helps to identify channels that do not conform to the normal environment traffic baseline, and may be worthy of investigation.
The parser creates meta keys, based on statistics collected by the native NetWitness parser, that help to identify behavior of any channel that is getting lots of network traffic. When the parser is first enabled, the analyst needs to become familiar with overall behavior for the different channels seen in a captured session to understand the frequency of bytes and the normal client and server payload. Once the normal behavior is known, analysts can use the meta keys to find behavior that does not match the expected.
By default, the Entropy parser generates 10 additional meta keys that do not add significantly to the load on a Decoder, and are useful for this specialized case. The parser is disabled by default.
Enable indexing if you have interest in exploring interesting sessions based on payload byte analysis of the packets. By default, to make indexing easier, the normal Float32 value for entropy.req and entropy.res is multiplied by 10k and stored in a UInt16 (thus giving four digits of precision, 0 to 10,000).
However, if you define the entropy.* fields in the Decoder language to be Float32, the Decoder will store it as a float with a range of 0.0 to 1.0. Take care to change the language everywhere if you decide to keep it as a Float32.
NetWitness does not recommend indexing as a Float32 because of the high unique counts due to minute changes in precision.
These are the new meta keys generated by the Entropy parser by default:
- entropy.req and entropy.res: These meta keys capture entropy using the Shannon entropy equation, which has a floating point value as a result. The floating point value of 0 to 1.000 is multiplied by 10000 and written in NetWitness as UInt 16, an unsigned integer of 0 through 10000. .
- mcb.req and mcb.res: The most common byte is simply which byte for each side (0 thru 255) was seen the most.
- mcbc.req and mcbc.res: The most common byte count is the number of times the most common byte (above) was seen in the session streams.
-
ubc.req and ubc.res: - Unique byte count is the number of unique bytes seen in each stream. 256 would mean all byte values of 0 thru 255 were seen at least once.
- payload.req and payload.res: The payload size metrics are the payload sizes of each session side at the time of parsing. However, in order to keep indexing from having high unique counts (bad for performance), the two payload size metas below are calculated this way:
Less than 1000 is the exact number of payload bytes.
1000 or greater is bucketed in increments of 1000. So a size of 5826 would be stored as 5000.
To enable and configure the Entropy parser on a Decoder:
- Go to
(Admin) > Services, select a Decoder service and click
View > Config.
The Services Config view for the selected Decoder is displayed. - The Entropy parser is disabled by default. Click the drop-down list for Entropy in the Config Value column, and select Enabled. If you want to disable some of the meta keys, click the drop-down list and select Disabled next to the meta key.
- Click Apply.
The Entropy parser is enabled and begins creating the new meta keys as configured in the Concentrator custom index file. - Navigate to the Explore View for the Decoder, and select the decoder > parsers > config node. In the parsers.options, you can set the Entropy parser payload. The default value shown in the screen capture is Entropy=payload=1000. When defining the value, the syntax is Entropy=payload="1000" The quotes are required if there is white space in the value, and it is a good practice to always use them to avoid white space issues. If you want to see the exact payload, set this parameter to "1".
The default Entropy payload is 1000, which means that if the payload count is less than 1000, the exact value is provided. If the payload count is greater than 1000, the value is rounded down to the nearest 1000. For example, a count of 3798 is rounded down to 3000. -
If you want to change the default Entropy payload rounding factor, edit the value. This change takes effect when the parser is reloaded
- In the Service Config view select the Concentrator that is aggregating traffic from this Decoder. Select View > Files and open the Custom Index file for the Concentrator. Look for the Entropy parser meta keys to see if they are included and uncommented.
By default the keys are commented out and therefore not enabled. To enable that part of the language the administrator needs to copy that part of index file into the index-concentrator-custom.xml and uncomment the key description line for each meta key. An example of the custom index file with the Entropy parser keys and instructions is shown below in Entropy Parser Configuration in the Concentrator Custom Index File. - With the Entropy meta keys enabled, they are available to analysts in Investigate, but hidden by default. To make the meta keys visible in the Investigate Values view, edit the default meta keys in the Default Meta Keys dialog so that they are open instead of hidden. You can manage these meta key the same way you manage other meta keys.
Entropy Parser Configuration in the Concentrator Custom Index FileEntropy Parser Configuration in the Concentrator Custom Index File
The following is an excerpt of the Concentrator Index file lines that the administrator must copy to the custom index file. The comments provide guidance on configuring the parser.
<!-- This section is commented out because it's only used by the Entropy parser which is disabled by default. To enable this part of the language, copy to index-concentrator-custom.xml and uncomment the keys. HOWEVER, take note that depending on how the Entropy parser is configured, the entropy.req and entropy.res format might be a Float32 instead of a UInt16. So make sure to change to the correct type if necessary.-->
<!-- Entropy parser meta - enable indexing if you have interest in exploring this for interesting sessions based on payload byte analysis of the packets. By default, to make indexing easier, the normal Float32 value for entropy.req and entropy.res is multiplied by 10k and stored in a UInt16 (thus giving 4 digits of precision, 0 to 10,000). However, if you define the entropy.* fields in the Decoder language to be Float32, it will store it as a float with a range of 0.0 to 1.0. Take care to change the language everywhere if you decide to keep it as a Float32. We do not recommend indexing as a Float32 because of the high unique counts due to minute changes in precision. -->
<!--
<key description="Entropy Request (Client)" format="UInt16" level="IndexNone" name="entropy.req" valueMax="10001"/>
<key description="Entropy Response (Server)" format="UInt16" level="IndexNone" name="entropy.res" valueMax="10001"/>
-->
<!-- The most common byte is simply which byte for each side (0 thru 255) was seen the most -->
<!--
<key description="Most Common Byte Request" format="UInt8" level="IndexNone" name="mcb.req"/>
<key description="Most Common Byte Response" format="UInt8" level="IndexNone" name="mcb.res"/>
-->
<!-- The most common byte count is the number of times the most common byte (above) was seen in the session streams -->
<!--
<key description="Most Common Byte Count Request" format="UInt32" level="IndexNone" name="mcbc.req" valueMax="500000"/>
<key description="Most Common Byte Count Response" format="UInt32" level="IndexNone" name="mcbc.res" valueMax="500000"/>
-->
<!-- Unique byte count is the number of unique bytes seen in each stream. 256 would mean all byte values of 0 thru 255 were seen at least once -->
<!--
<key description="Unique Byte Count Request" format="UInt16" level="IndexNone" name="ubc.req"/>
<key description="Unique Byte Count Response" format="UInt16" level="IndexNone" name="ubc.res"/>
-->
<!-- The payload size metrics are the payload sizes of each session side at the time of parsing. However, in order to keep indexing from having high unique counts (bad for performance), the two payload size metas below are indexed in buckets. -->
<!--
<key description="Payload Size Request" format="UInt32" level="IndexNone" bucket="true" name="payload.req" valueMax="500000"/>
<key description="Payload Size Response" format="UInt32" level="IndexNone" bucket="true" name="payload.res" valueMax="500000"/>
-->
