Log Parser Customization

On occasion, you may need to modify one or more of your log parsers. For example, you may need to fix an unknown message, or to parse certain fields differently than in the manner provided by default.

Log Parser Customization allows you to add new parser elements or modify existing ones. All customizations reside in a separate file that does not get removed or overwritten by Log Decoder upgrades or the updating parsers through the RSA Live.

Note: This feature is only available in 10.6.5 and later (including NetWitness 11.x)

Loading Order

The default parser file is loaded before the custom file (if a custom file exists). This allows users to override elements, as shown in the examples below that modify items that exist in the default file.

File Location and Naming

Log parser files are located on the Log Decoder in the following path:

/etc/netwitness/ng/envision/etc/devices

Each log parser has its own sub-folder. For example, the ciscoasa parser files are in the following folder:

/etc/netwitness/ng/envision/etc/devices/ciscoasa

Custom log parser files are located in the same folder as the corresponding system-provided files. For naming, you use the name of the XML file, followed by -custom.xml

For example, the ciscoasa parser consists of two files: ciscoasa.ini and v20_ciscoasamsg.xml. If you create a custom file, you need to name it v20_ciscoasamsg-custom.xml, and add it to the same folder, /etc/netwitness/ng/envision/etc/devices/ciscoasa.

Header and Message Duplication

When you customize a parser, make sure to duplicate only those headers and messages that you want to customize. That is, we recommend that you do not simply copy everything from the default parser file and then paste it into your custom XML file. Also, note that if you duplicate headers and messages that exist in the default parser, you will not be using the default versions, even if RSA updates them in the future.

Examples

The following sections contain examples for adding or modifying portions of a log parser.

All the examples use the Oracle Access Manager (oracleam) log parser.

Example Code

Cod examples are broken down into two areas:

Additionally, insertBefore and insertAfter describes the usage of the insertBefore and insertAfter commands, for use when adding a new item.

Common Steps

The common steps, which are the same in all of the examples, are as follows:

  1. Use an SSH tool, such as WinSCP, to navigate to the following folder on your Log Decoder:

    /etc/netwitness/ng/envision/etc/devices/oracleam

  2. Copy the oracleammsg.xml file to your local system.

  3. Note the Device Messages and Version information, which comprise the first several lines of the oracleammsg.xml file. You need to copy these lines into your custom parser file.

    <DEVICEMESSAGES

    name="oracleam"
    displayname="Oracle Access Manager"
    group="Access Control">

    <VERSION
    xml="60"
    checksum="110c39794680bdedfabb5a73339d38eb"
    revision="104"
    device="2.0"/>

  4. Using a text editor, create a file named oracleammsg-custom.xml, and add custom text, after the introductory text specified in the previous step. The specific custom text is supplied in each of the following examples.

  5. Save the custom file as oracleammsg-custom.xml, and using your SSH tool, upload it to /etc/netwitness/ng/envision/etc/devices/oracleam on your Log Decoder.

Add a New Item

When you add an item, you use a new identifier, and optionally, an insertBefore or insertAfter command.

You can add any of the following items:

Add New Header

Using a text editor, create a file named oracleammsg-custom.xml, and add the following text:

<DEVICEMESSAGES

name="oracleam"
displayname="Oracle Access Manager"
group="Access Control">

<VERSION
xml="60"
checksum="110c39794680bdedfabb5a73339d38eb"
revision="104"
device="2.0"/>
<!-- VERSION info copied from oracleammsg.xml -->

<HEADER

id1="0044"
id2="0044"
insertBefore="0005"
content="%ORACLEAM-&lt;hfld1&gt;: &lt;hdate&gt; &lt;htime&gt; *&lt;htimezone&gt; - &lt;messageid&gt; &lt;!payload:messageid&gt;" />

</DEVICEMESSAGES>

Note the insertBefore="0005" line. This instructs the system to insert the new header before existing header number 0005.

Add New Message

Using a text editor, create a file named oracleammsg-custom.xml, and add the following text:

<DEVICEMESSAGES

name="oracleam"
displayname="Oracle Access Manager"
group="Access Control">

<VERSION
xml="60"
checksum="110c39794680bdedfabb5a73339d38eb"
revision="104"
device="2.0"/>
<!-- VERSION info copied from oracleammsg.xml -->

<MESSAGE

id1="AUTHZ_SUCCESS:03"
id2="AUTHZ_SUCCESS"
eventcategory="1302000000"
insertAfter="AUTHZ_SUCCESS:01"
functions="&lt;@ec_theme:Authentication&gt;&lt;@ec_outcome:Success&gt;&lt;@event_time:*EVNTTIME($HDR,'%G/%F/%W %N:%U:%O',hdate,htime)&gt;&lt;@msg:*PARMVAL($MSG)&gt;&lt;@:*SYSVAL($MSGID,$ID1)&gt;"
content="&lt;event_type&gt; - &lt;web_method&gt; - &lt;hostname&gt;&lt;fld1&gt;- &lt;saddr&gt; - {&lt;web_domain&gt;%&lt;fld27&gt;&lt;fld2&gt;|&lt;url&gt;&lt;fld2&gt;} - cn=&lt;username&gt;,&lt;fld3&gt; - &lt;fld4&gt; - &lt;protocol&gt; - &lt;obj_type&gt;&lt;fld6&gt; - &lt;context&gt; - &lt;id&gt; - cn=&lt;fld7&gt;,cn1=&lt;fld23&gt;, uid=&lt;uid&gt;" />

</DEVICEMESSAGES>

Note the insertAfter="AUTHZ_SUCCESS:01" line. This instructs the system to insert the new message after existing message with ID AUTHZ_SUCCESS:01.

Add New Valuemap

For the remaining examples, the introductory lines are not included. Add the following code after the introductory VERSION information.

<VALUEMAP

name="getDisposition"
default="$NONE"
keyvaluepairs="0=&apos;Failure&apos;|1=&apos;Success&apos;" />

<MESSAGE

id1="AUTHZ_SUCCESS:03"
id2="AUTHZ_SUCCESS"
eventcategory="1302000000"
insertBefore="AUTHZ_SUCCESS:01"
functions="&lt;@ec_theme:Authentication&gt;&lt;@ec_outcome:Success&gt;&lt;@event_time:*EVNTTIME($HDR,'%G/%F/%W %N:%U:%O',hdate,htime)&gt;&lt;@msg:*PARMVAL($MSG)&gt;&lt;@:*SYSVAL($MSGID,$ID1)&gt;"
content="&lt;event_type&gt; - &lt;web_method&gt; - &lt;hostname&gt;&lt;fld1&gt;- &lt;saddr&gt; - {&lt;web_domain&gt;%&lt;fld27&gt;&lt;fld2&gt;|&lt;url&gt;&lt;fld2&gt;} - cn=&lt;username&gt;,&lt;fld3&gt; - &lt;fld4&gt; - &lt;protocol&gt; - &lt;obj_type&gt;&lt;fld6&gt; - &lt;context&gt; - &lt;id&gt; - cn=&lt;fld7&gt;,cn1=&lt;fld23&gt;, uid=&lt;uid&gt;" />

</DEVICEMESSAGES>

Add New Tagval

Add the following code after the introductory VERSION information.

<TAGVALMAP

pairdelimiter="^^" encapsulator="&quot;" />

<VALUEMAP

name="getDisposition"
default="$NONE"
keyvaluepairs="0=&apos;Failure&apos;|1=&apos;Success&apos;" />

<MESSAGE

id1="ORACLEAM_TVM"
id2="ORACLEAM_TVM"
eventcategory="1901000000"
tagval="true"
missField="true"
functions="&lt;@msg:*PARMVAL($MSG)&gt;&lt;@event_time:*EVNTTIME($MSG,'%W-%G-%F %H:%T:%S',fld3)&gt;&lt;@disposition:*getDisposition(fld12)&gt;&lt;@msg_id:*PARMVAL(event_type)&gt;&lt;@vid:*PARMVAL(event_type)&gt;&lt;@event_id:*STRCAT(event_type,_,disposition)&gt;&lt;@event_cat:*getEventLegacyCategory(event_id)&gt;&lt;@event_cat_name:*getEventLegacyCategoryName(event_cat)&gt;"
content="IAU_EVENTTYPE=&lt;event_type&gt;^^IAU_EVENTCATEGORY=&lt;category&gt;^^IAU_COMPONENTTYPE=&lt;event_source&gt;^^IAU_HOSTID=&lt;dhost&gt;^^IAU_HOSTNWADDR=&lt;daddr&gt;^^IAU_AGENTID=&lt;fld1&gt;^^IAU_PROCESSID=&lt;process_id&gt;^^IAU_SESSIONID=&lt;sessionid&gt;^^IAU_SSOSESSIONID=&lt;sessionid1&gt;^^IAU_APPLICATIONNAME=&lt;application&gt;^^IAU_APPLICATIONDOMAINNAME=&lt;fld2&gt;^^IAU_EVENTSTATUS=&lt;fld12&gt;^^IAU_TSTZORIGINATING=&lt;fld3&gt;^^IAU_THREADID=&lt;fld4&gt;^^IAU_INITIATOR=&lt;username&gt;^^IAU_USERID=&lt;uid&gt;^^IAU_MESSAGETEXT=&lt;event_description&gt;^^IAU_REMOTEIP=&lt;saddr&gt;^^IAU_RESOURCE=&lt;fld5&gt;^^IAU_DOMAINNAME=&lt;domain&gt;^^IAU_SERVERNAME=&lt;hostname&gt;^^IAU_INSTANCENAME=&lt;instance&gt;^^IAU_AUTHORIZATIONPOLICYID=&lt;policy_id&gt;^^IAU_AUTHENTICATIONPOLICYID=&lt;policy_id&gt;^^IAU_RESOURCEHOST=&lt;shost&gt;^^IAU_RESOURCEURI=&lt;url&gt;^^IAU_ADDITIONALINFO=&lt;fld7&gt;" />

</DEVICEMESSAGES>

Note the tagval="true" code in the message. We are adding this message that uses the new Tagval map.

insertBefore and insertAfter

As shown in some of the previous examples, the insertBefore and insertAfter commands instruct the system about where to place the new items when combining the standard and custom XML definition files, as it creates a unified parser during processing.

Note: If both insertBefore and insertAfter are defined, insertBefore will be used, and a warning will be logged. If neither is specified, the header or message is added at the end of the combined parser definition.

Modify an Existing Item

To modify an existing element, you use the same identifiers as an existing item, and change the contents. See the examples to modify any of the following items:

Modify Header

This example replaces the Header that has an ID of 0004. Add the following code after the introductory VERSION information.

<HEADER

id1="0004"
id2="0004"
content="%ORACLEAM-&lt;hfld1&gt;: &lt;hdate&gt; &lt;htime&gt; *&lt;htimezone&gt; - &lt;messageid&gt; &lt;!payload:messageid&gt;" />

</DEVICEMESSAGES>

Modify Message

This example replaces the Message that has an ID of AUTHZ_SUCCESS:01. Add the following code after the introductory VERSION information.

<MESSAGE

id1="AUTHZ_SUCCESS:01"
id2="AUTHZ_SUCCESS"
eventcategory="1302000000"
functions="&lt;@ec_theme:Authentication&gt;&lt;@ec_outcome:Success&gt;&lt;@event_time:*EVNTTIME($HDR,'%G/%F/%W %N:%U:%O',hdate,htime)&gt;&lt;@msg:*PARMVAL($MSG)&gt;&lt;@:*SYSVAL($MSGID,$ID1)&gt;"
content="&lt;event_type&gt; - &lt;web_method&gt; - &lt;hostname&gt;&lt;fld1&gt;- &lt;saddr&gt; - {&lt;web_domain&gt;%&lt;fld27&gt;&lt;fld2&gt;|&lt;url&gt;&lt;fld2&gt;} - cn=&lt;username&gt;,&lt;fld3&gt; - &lt;fld4&gt; - &lt;protocol&gt; - &lt;obj_type&gt;&lt;fld6&gt; - &lt;context&gt; - &lt;id&gt; - cn=&lt;fld7&gt;,cn1=&lt;fld23&gt;, uid=&lt;uid&gt;" />

</DEVICEMESSAGES>

Modify Valuemap

This example replaces the getDisposition Valuemap. Add the following code after the introductory VERSION information.

<VALUEMAP

name="getDisposition"
default="$NONE"
keyvaluepairs="0=&apos;Failure&apos;|1=&apos;Success&apos;|3=&apos;Test&apos;" />

</DEVICEMESSAGES>

In this example, we are assuming the device XML, oracleammsg.xml, includes a Valuemap named getDisposition, and that we are changing the existing information, for example we might be adding a new key value pair, 3=&apos;Test&apos.

Modify Tagval

This example replaces the existing Tagval. Add the following code after the introductory VERSION information.

<TAGVALMAP

pairdelimiter="^^^" encapsulator="&quot;" />

<MESSAGE

id1="ORACLEAM_TVM"
id2="ORACLEAM_TVM"
eventcategory="1901000000"
tagval="true"
missField="true"
functions="&lt;@msg:*PARMVAL($MSG)&gt;&lt;@event_time:*EVNTTIME($MSG,'%W-%G-%F %H:%T:%S',fld3)&gt;&lt;@disposition:*getDisposition(fld12)&gt;&lt;@msg_id:*PARMVAL(event_type)&gt;&lt;@vid:*PARMVAL(event_type)&gt;&lt;@event_id:*STRCAT(event_type,_,disposition)&gt;&lt;@event_cat:*getEventLegacyCategory(event_id)&gt;&lt;@event_cat_name:*getEventLegacyCategoryName(event_cat)&gt;"
content="IAU_EVENTTYPE=&lt;event_type&gt;^^^IAU_EVENTCATEGORY=&lt;category&gt;^^^IAU_COMPONENTTYPE=&lt;event_source&gt;^^^IAU_HOSTID=&lt;dhost&gt;^^^IAU_HOSTNWADDR=&lt;daddr&gt;^^^IAU_AGENTID=&lt;fld1&gt;^^^IAU_PROCESSID=&lt;process_id&gt;^^^IAU_SESSIONID=&lt;sessionid&gt;^^^IAU_SSOSESSIONID=&lt;sessionid1&gt;^^^IAU_APPLICATIONNAME=&lt;application&gt;^^^IAU_APPLICATIONDOMAINNAME=&lt;fld2&gt;^^^IAU_EVENTSTATUS=&lt;fld12&gt;^^^IAU_TSTZORIGINATING=&lt;fld3&gt;^^^IAU_THREADID=&lt;fld4&gt;^^^IAU_INITIATOR=&lt;username&gt;^^^IAU_USERID=&lt;uid&gt;^^^IAU_MESSAGETEXT=&lt;event_description&gt;^^^IAU_REMOTEIP=&lt;saddr&gt;^^^IAU_RESOURCE=&lt;fld5&gt;^^^IAU_DOMAINNAME=&lt;domain&gt;^^^IAU_SERVERNAME=&lt;hostname&gt;^^^IAU_INSTANCENAME=&lt;instance&gt;^^^IAU_AUTHORIZATIONPOLICYID=&lt;policy_id&gt;^^^IAU_AUTHENTICATIONPOLICYID=&lt;policy_id&gt;^^^IAU_RESOURCEHOST=&lt;shost&gt;^^^IAU_RESOURCEURI=&lt;url&gt;^^^IAU_ADDITIONALINFO=&lt;fld7&gt;" />

</DEVICEMESSAGES>