Netwitness Log Parser Cleaning Project

[JayShah]

The RSA NetWitness Log Parser Content team has cleaned all the log parsers to remove RSA EnVision (Legacy Product) footprint from the parsers.

These enhancements are part of a strategic initiative to clean all the parsers and remove the enVision footprint resulting in a more manageable, maintainable and flexible parser.

Modifications to the parser to achieve a cleaned parser:

1. Following tags from parser were removed:

• level
• parse   
• parsedefvalue
• tableid
• summary
• vid
• vidx
• devts
• enVision tag

Before:
<MESSAGE
level="6"
parse="1"
parsedefvalue="1"
tableid="90"
id1="addmember"
id2="addmember"
eventcategory="1701010000"
content="&lt;@ec_theme:Configuration&gt;&lt;@msg:*PARMVAL($MSG)&gt;&lt;@event_time:*EVNTTIME($HDR,'%B %F %N:%U:%O %W',hmonth,hdate,htime,hyear)&gt;&lt;@:*SYSVAL($MSGID,$ID1)&gt;&lt;@action:addmember&gt;addmember for &lt;username&gt; from &lt;daddr&gt; for group &lt;group&gt; exited with &lt;disposition&gt;"/>

After:
<MESSAGE
id1="addmember"
id2="addmember"
eventcategory="1701010000"
functions="&lt;@ec_theme:Configuration&gt;&lt;@msg:*PARMVAL($MSG)&gt;&lt;@event_time:*EVNTTIME($HDR,'%B %F %N:%U:%O %W',hmonth,hdate,htime,hyear)&gt;&lt;@:*SYSVAL($MSGID,$ID1)&gt;&lt;@action:addmember&gt;"
content="addmember for &lt;username&gt; from &lt;daddr&gt; for group &lt;group&gt; exited with &lt;disposition&gt;" />

2. Seperate out function from the content line and create two tags function and content for each message id:

Before:
content="&lt;@ec_theme:Configuration&gt;&lt;@msg:*PARMVAL($MSG)&gt;&lt;@event_time:*EVNTTIME($HDR,'%B %F %N:%U:%O %W',hmonth,hdate,htime,hyear)&gt;&lt;@:*SYSVAL($MSGID,$ID1)&gt;&lt;@action:addmember&gt;addmember for &lt;username&gt; from &lt;daddr&gt; for group &lt;group&gt; exited with &lt;disposition&gt;"/>

After:
functions="&lt;@ec_theme:Configuration&gt;&lt;@msg:*PARMVAL($MSG)&gt;&lt;@event_time:*EVNTTIME($HDR,'%B %F %N:%U:%O %W',hmonth,hdate,htime,hyear)&gt;&lt;@:*SYSVAL($MSGID,$ID1)&gt;&lt;@action:addmember&gt;"
content="addmember for &lt;username&gt; from &lt;daddr&gt; for group &lt;group&gt; exited with &lt;disposition&gt;" />

3.Removed the duplicate functions from the content line:
In certain log parsers, a few duplicate function were prevalent, which were removed.

Before:

<MESSAGE 
id1="000004"
id2="Access"
eventcategory="1001000000"
functions="&lt;@saddr:*HDR(hfld0)&gt;&lt;@event_type:VPN&gt;&lt;@event_time:*EVNTTIME($HDR,'%W%G%F %H:%U:%O',hfld31,hfld32,hfld33,time)&gt;&lt;@:*SYSVAL($MSGID,$ID1)&gt;&lt;@msg:*PARMVAL($MSG)&gt;&lt;@realm:*HDR(hfld1)&gt;&lt;@group:*HDR(hgroup)&gt;&lt;@username:*HDR(husername)&gt;&lt;@domain:*HDR(hdomain)&gt;&lt;@action:access blocked&gt;&lt;@username:*HDR(husername)&gt;"
content="Access blocked after DNS lookup. Check Web ACL settings - Host: &lt;hostip&gt;, Request: {&lt;web_method&gt; &lt;webpage&gt; &lt;fld1&gt; | &lt;url&gt;}" />

After:

<MESSAGE
id1="000004"
id2="Access"
eventcategory="1001000000"
functions="&lt;@saddr:*HDR(hfld0)&gt;&lt;@event_type:VPN&gt;&lt;@event_time:*EVNTTIME($HDR,'%W%G%F %H:%U:%O',hfld31,hfld32,hfld33,time)&gt;&lt;@:*SYSVAL($MSGID,$ID1)&gt;&lt;@msg:*PARMVAL($MSG)&gt;&lt;@realm:*HDR(hfld1)&gt;&lt;@group:*HDR(hgroup)&gt;&lt;@username:*HDR(husername)&gt;&lt;@domain:*HDR(hdomain)&gt;&lt;@action:access blocked&gt;&lt;@username:*HDR(husername)&gt;"
content="Access blocked after DNS lookup. Check Web ACL settings - Host: &lt;hostip&gt;, Request: {&lt;web_method&gt; &lt;webpage&gt; &lt;fld1&gt; | &lt;url&gt;}" />

4.Removes EE collisions: (RSA enVision (Legacy Product) Concept)
We removed EE_collisions which was an enVision concept

Before: 

<MESSAGE
id1="chpasswd"
id2="chpasswd"
eventcategory="1701020000"
functions="&lt;@ec_subject:Password&gt;&lt;@ec_theme:Configuration&gt;&lt;@msg:*PARMVAL($MSG)&gt;&lt;@event_time:*EVNTTIME($HDR,'%B %F %N:%U:%O %W',hmonth,hdate,htime,hyear)&gt;&lt;@:*SYSVAL($MSGID,$ID1)&gt;&lt;@action:chpasswd&gt;&lt;@fld61:*PARMVAL(disposition)&gt;"
content="chpasswd for &lt;username&gt; from &lt;daddr&gt; exited with &lt;disposition&gt;" />

After:

<MESSAGE
id1="chpasswd"
id2="chpasswd"
eventcategory="1701020000"
functions="&lt;@ec_subject:Password&gt;&lt;@ec_theme:Configuration&gt;&lt;@msg:*PARMVAL($MSG)&gt;&lt;@event_time:*EVNTTIME($HDR,'%B %F %N:%U:%O %W',hmonth,hdate,htime,hyear)&gt;&lt;@:*SYSVAL($MSGID,$ID1)&gt;&lt;@action:chpasswd&gt;&lt;@fld61:*PARMVAL(disposition)&gt;"
content="chpasswd for &lt;username&gt; from &lt;daddr&gt; exited with &lt;disposition&gt;" />

5. INI migration for future usage:
During this parser cleaning process we have also pulled needed tags from ini into the parser, this move is to remove dependency on INI file in the future

All the changes made to the parsers during this cleaning project will have no backward compatibility impact.

We have already posted a few such cleaned parsers to live during our pilot project (15 parsers were posted to live). To take advantage of these improvements, you will need to download the latest versions of the Log Parsers which will be released to NetWitness Live Portal by 1st September 2017.

Note: For customized parsers merge your customizations to the parser from live just like before, but make sure you get rid of the obsolete tags(mentioned above) and split up the functions and content tag. It will continue to work even otherwise, but you will not have a cleaned parser.