NetWitness 11.x Type and Pattern Validation for Variables

[#1]

Article Number

000002088

Applies To

RSA Product Set: NetWitness Logs and Network
RSA Product/Service Type: Log Decoder
RSA Version/Condition: 11.x and Above 
Platform: CentOS
 

Issue

Prior to 11.0 upgrade, type and pattern validation was not implemented for variables. As a result, any text will qualify for any of the variables. For Example, consider the following parser

Image description
As the format is not specified for any of the variables, the username in this case could be 192.168.1.1.  

 

Resolution

Typed variables (also called as VARTYPE) added in 11.0 and later versions, provide type and pattern validation for variables. While the variables are parsed, you can conditionalize them to match a certain format. For Example, consider the following parser

Image description
Using VARTYPE, you can define the username variable to contain only characters and numbers, and fail the MESSAGE pattern match. In this case, username is a typed variable.

Image description

As shown in the above example, the variable name is specified using the name attribute. The regular expression defines the criteria that username must meet the specific condition ([a-z0-9]+). If 192.168.1.1 is found where the username variable resides in a pattern, the pattern will not match.
Note: In 11.2.1, typed variables allow the capture of parsed values to other variables.
 

Typed Variables Formats

1. regex

This functionality uses the standard regular expression syntax. Use the ignorecase attribute to toggle the case sensitivity. By default, regular expressions are case sensitive. Hence, the below two statements are identical.

Image description
Image description
Note: The ignorecase attribute is optional.
In 11.5, enable regex search (instead of match) using the search attribute. By default, this attribute is set to false, resulting in regex match. Match requires the entire input to match the expression. Refer the following screenshot.

Image description
 

2. dateTime

The dateTime format was added in 11.5 and later versions. It includes a list of format specifiers for parsing timestamps. Refer the following screenshot.

Image description

3. format

You can set the criteria for a typed variable using the format attribute as well. The value of the attribute for the respective variables are predefined types that define the type the variables must have for a match to happen. For example, you can define the variable hostIP as a typed variable, if some log messages are incorrectly assigning it a value of hostname.com instead of valid IPs. Refer the following screenshot.

Image description

In this case, only IPs of the format XXX.XXX.XXX.XX will match and not the value of hostname.com.
The below table lists all the valid types of formats.

Format	Description	Example
Base64	Base 64 encoded text (11.4 and later versions)	SGVsbG8gd29ybGQhIQ== (Hello world!!).
Float32	Decimal numbers	2.71818
Float64	Decimal numbers	2.71818
HexString	Hex Encoded Text (11.3 and later versions)	48656C6C6F20776F726C642121 (Hello world!!).
IPv4	ipv4	192.168.1.1
IPv6	ipv6	2607:f0d0:1002:51::4
MAC	Physical Mac address	01:23:45:67:89:ab
Hostname	RFC-1123 compliant hostname (11.2.1 and later versions)	abc.xzy.com
Int8	Signed 8-bit integer	-128 to 127
Int16	Signed 16-bit integer	-32768 to 32767
Int32	Signed 32-bit integer	-2147483648 to 2147483647
Int64	Signed 64-bit integer	-9223372036854775808 to 9223372036854775807
URI	Universal Resource identifier (11.2.1 and later versions)	http://www.google.com/path/script?query=param
UInt8	Unsigned 8-bit integer	0 to 255
UInt16	Unsigned 16-bit integer	0 to 65535
UInt32	Unsigned 32-bit integer	0 to 4294967295
UInt64	Unsigned 64-bit integer	0 to 18446744073709551615
EMail	Valid email address (11.2.1 and later versions)	bob@company.com
TimeDuration	Format type to convert captured time values into a duration.	VARTYPE name="timeduration" duration="%H:%T:%S"> <CAPTURE index="0" variable="duration"/> </VARTYPE>
convert	It is used to convert captured values to a new formatted value. Convert supports two values, Bytes and Domain. Bytes: There is an optional parameter specifier currently supported for bytes. The list of specifier supported for Bytes are: KB2B MB2B GB2B B2KB B2MB B2GB Domain: The optional parameter specifier is not required for Domain.	Byte conversion: KB to Bytes, MB to Bytes, Bytes to KB and so on. For example: <VARTYPE name="kb_variable" convert="Bytes" specifier="KB2B"> <CAPTURE index="0" variable="bytes"> </VARTYPE> Domain name conversion: The (3)www(6)cheese(5)toast(3)com is converted to www.cheese.toast.com. For example: <VARTYPE name="src_host" convert="Domain"> <CAPTURE index="0" variable="shost"/> </VARTYPE>

Example screenshot
 
Image description
 

4. Multiple Types

Define multiple criteria for a typed variable such as hostip. For the variable hostip to match an IPv4 or IPv6, and not a hostname, define additional types. Refer the following screenshot.

Image description

In this case, a log will match if the value of hostip matches either of the two formats specified. You can add any number of criteria to a variable and define multiple formats, multiple regular expressions, and / or a mixture of both.
The below variants of encoding are also supported to specify multiple types.
Fully nested

Image description

Individually defined
Image description

5. Capture
In 11.2.1 and later versions, the <CAPTURE …> element is provided to capture sub-components of the match and assign them to additional variables.

In the following example, a regex is defined with captures specified for username and optionally the domain (if it is present). The coordinating index values are used to specify which capture index is assigned to which variable.

Image description

In the above example, with the input DOMAIN\user, index 0 would capture the entire match DOMAIN\user, index 2 would capture DOMAIN and index 3 would capture user. 
Note: Index 1 (not specified here for capture), would have a value DOMAIN\ and thus would be ignored.
The <CAPTURE ...> element can also be used for format Typed Variables. This provides a mechanism to conditionally assign a variable based on the discovered (verified) type. Refer the following screenshot.

Image description

The capture index for a format Typed Variable must be zero (0) and there must only be one capture specified. If not, all captures are ignored.
 

6. Precedence

Typed Variables support a match order to enable the application of precedence (added in 11.2.1 and later versions). Consider the following example.
 If the regex is first run on user@domain.com, it will match the user portion of the email, short circuit, and not allow for the full email signature to match.
The order attribute specifies that the Email format should be executed for variable huser before the regex format. Refer the following screenshot.

Image description

Those Typed Variables with the order attribute absent or with a value of zero (0) are executed after those with a defined value, which are executed lowest to highest.
 

7. Match Requirement

There are cases where a match may not be required. For example, in a tagval <MESSAGE> (which is inherently very structured), a <VARTYPE> may be useful for parsing additional <CAPTURE> values from a field (e.g. user and domain). However, since the message is very structured, it would be desirable to allow the message to match even if the data received is unexpected to consume the other highly structured fields of the message.
To this means, the attribute requireMatch is provided (when absent the default is true). It allows <VARTYPE> to provide <CAPTURE> functionality without invalidating a message match. It can be specified in either the <VARTYPE> or <TYPE> elements. When absent or containing a garbage value in the <TYPE> element, the value will be inherited from the parent <VARTYPE>. Refer the following screenshot.

Image description
Note: The attribute requireMatch is added in 11.3 and later versions.
When multiple format types are provided, precedence order is followed and the last format type processed wins out on requiring a match.
Note: Typed Variable processing support for TagVal messages was added in 11.3.

When multiple format types are provided, precedence order is followed and the last format type processed wins out on requiring a match.
Note: Typed Variable processing support for TagVal messages was added in 11.3.
 

8. VARTYPE Support in CEF Parser

Typed Variable support for CEF was added in NetWitness 12.0. Typed Variables can provide:

• Pattern validation and fine parsing for variables in HEADER definitions.
• Fine parsing for the metaName variables defined in the ExtensionKey mappings.

 

9. SCANNED Variable Support in CEF Parser

Some CEF sources have structured data embedded in variables. This data can be further parsed with the dynamic scanning parsers like Parse Rules. Scanned Variable support for CEF was added in NetWitness 12.0.
 For more information on custom CEF parser, refer Custom CEF Parser - NetWitness Community - 677921.

10. Support for Static Value Mapping in VARTYPE

Support for static value mapping in VARTYPE was added in NetWitness 12.1 version. Using the static value mapping, content author can transform, normalize, and categorize the logs.
Example for Transformation:
Image description
In the above example, transformation is achieved by mapping the static value ‘http’ to variable ‘protocol’ for port ‘80’.
Example for Normalization:
Image description
In the above example, normalization is achieved by tagging static values ‘logon’ and ‘logoff’ to variable ‘operation_id’ for log values ‘in/on’ and ‘out/off’ respectively.
Example for Categorization:
Image description
In the above example, categorization is achieved by tagging static values to log value as follows:

Log Value	Static Value	Variable
in	User.Activity.Successful Logins	event.cat.name
out	User.Activity.Logoff	event.cat.name

Note: Multiple static values can be assigned to the same capture by nesting multiple capture groups.
 

FAQs

The table below lists few FAQs regarding VARTYPE

Questions	Answers
What if a variable username has a type definition in one parser and another type definition in a second parser?	Type variables are applied on a per parser basis. Thus, variables of the same name can have different defined types across multiple parsers and the type criteria is applied only to patterns in that parser.
Where should <VARTYPE> elements be placed?	The <VARTYPE> elements must be the child elements of the <DEVICESMESSAGES> element. Hence, they are siblings of the <HEADER> and <MESSAGE> elements. It does not matter if the <VARTYPE> elements are placed before or after its siblings. It is even possible to have some before and after, so long as they meet the hierarchy requirements.
How do I view debug logging for typed variable evaluation?	The log level must be explicitly set for the log parsing module with LogParse=debug to view debug logging for typed variable evaluation.
What about header priority?	Header priority does not play a role in Typed Variables (or vice-versa). Header priority dictates the priority in which headers should be matched if there are multiple matches.