Custom Flat File Log Collection with NW-Endpoint 11.4

[JoshRandall]

22APR2020 - UPDATE: Naushad Kasu‌ has posted a video blog of this process and I have posted the template.xml and NweAgentPolicyDetails_x64.exe files from his blog here.

08APR2020 - UPDATE: adding a couple notes and example typespecs after some additional experimenting over the past week

• You may find the process easier to simply copy an existing 11.4 typespec in the /var/netwitness/source-server/content/collection/file directory on the Admin Server and modify it for the custom collection source you need
• example using IIS typespec:
  • comparison of the XML from the Log Collector/Log Decoder to the version I created on the Admin Server
    
    
    
• another example using a custom typespec to collect Endpoint v4.4 (A.K.A. legacy ECAT) server logs
  • two different typespecs to collect the exact same set of logs, we can see exactly how the values in the typespec affect the raw log that ultimately gets ingested by NetWitness
    
    
    

**END UPDATE**

The NetWitness 11.4 release included a number of features and enhancements for NetWitness Endpoint, one of which was the ability to collect flat file logs (https://community.rsa.com/docs/DOC-110149#Endpoint_Configuration), with the intent that this collection method would allow organizations to replace existing SFTP agents with the Endpoint Agent.

Flat file collection via the 11.4 Endpoint agent allows for a much easier management compared to the SFTP agent, in addition to the multitude of additional investigative and forensic benefits available with both the free version of the Endpoint agent and the advanced version (NetWitness Endpoint User Guide for NetWitness Platform 11.x - Table of Contents).

The 11.4 release included a number of OOTB, supported Flat File collection sources, with support for additional OOTB, as well as custom, sources planned for future releases.  However, because I am both impatient and willing to experiment in my lab where there are zero consequences if I break something, I decided to see whether I could port my existing, custom SFTP-based flat file collections to the new 11.4 Endpoint collection.

The process ended up being quite simple and easy.  Assuming you already have your Endpoint Server installed and configured, as well as custom flat file typespecs and parsers that you are using, all you need to do is:

1. install an 11.4+ endpoint agent onto the host(s) that have the flat file logs
2. ...then copy the custom typespec from the Log Decoder/Log Collector filesystem (/etc/netwitness/ng/logcollection/content/collection/file)
3. ...to the Node0/Admin Server filesystem (/var/netwitness/source-server/content/collection/file)
   1. ...if your typespec does not already include a <defaults.filePath> element in the XML, go ahead and add one (you can modify the path later in the UI)
   2. ...for example: 
      
4. ...after your typespec is copied (and modifed as necessary), restart the source-server on the Node0/Admin Server
   
5. ...now open the NetWitness UI and navigate to Admin/Endpoint Sources and create a new (or modify an existing) Agent File Logs policy (more details and instructions on that here: Endpoint Config: About Endpoint Sources)
   1. ...find your custom Flat File log source in the dropdown and add it to the Endpoint Policy
      
   2. ...modify the Log File Path, if necessary:
      
   3. ...then simply publish your newly modified policy
6. ...and once you have confirmed Collection via the Endpoint Agent, you can stop the SFTP agent on the log source (https://community.rsa.com/docs/DOC-101743#Replace)

And that's it.  Happy logging.

template.xml.zip

NweAgentPolicyDetails_x64.exe.zip

[NaushadKasu1]

Thank you for putting this together Josh!

[GergelyLitavszk]

Hi Josh!

Thanks for the article, this is really helpful!

I put it together in a test enviroment, I get the logs for Microsoft IIS however the parsing is not working properly. I got the latest parser from live.

Example of raw logs in Netwitness:

<13>1 - WIN-9S84OATDFF8 microsoftiis - - [lc@36807 lc.uuid="02C70B8A-DAE5-077C-5899-16CAD2C20667" lc.src="192.168.97.243" lc.ctime="1588069655446"] %IIS-4: 2020-04-28 10:21:51 192.168.97.243 GET /helloguys - 80 - 192.168.97.179 Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/81.0.4044.122+Safari/537.36 - 404 0 2 15

Do u have any idea why is that?

Thanks for your help in advance!

Best Regards,

Gergely Litavszki

[DaveGlover]

In order for IIS to properly work using this method a new IIS message definition needs to be built for your specific logging options.   The default IIS parser will not parse the messages properly.

Please reach out to me directly if you would like assistance in building this message definition 

Dave

[GergelyLitavszk]

Hi Dave,

Thanks for your help and the quick reply!

Is it possible for Josh to send out his already modified parser? If I see it right, in the article, it is working for him properly.

Thanks again!

Best Regards,

Gergely Litavszki

[JoshRandall]

Gergely Litavszki‌

The easiest way to fully parse everything

1. make sure you have all of the IIS logging fields enabled:
2. add this text as filename "v20_microsoftiismsg-custom.xml" to the "/etc/netwitness/ng/envision/etc/devices/microsoftiis" directory on your log decoder

   <?xml version="1.0" encoding="UTF-8"?>
   <DEVICEMESSAGES
         name="microsoftiis"
         displayname="Microsoft IIS"
         group="Web Logs"
         type="55">
   <VERSION
         xml="45"
         revision="129"
         checksum="c8e2b0c5fc3bd3dce9e76eacd8378460"
         device="2.0"/>
   <MESSAGE
         id1="IIS_LOG_Level:02"
         id2="IIS_LOG_Level"
         insertBefore="IIS_LOG_Level"
         eventcategory="1204010000"
         functions="&lt;@bytes:*CALC(sbytes,+,rbytes)&gt;&lt;@web_domain:*URL($DOMAIN,web_host)&gt;&lt;@tbdstr2:*PARMVAL(web_referer)&gt;&lt;@tbdstr3:*PARMVAL(web_query)&gt;&lt;@event_time:*EVNTTIME($MSG,'%W-%G-%F %N:%U:%O',fld20,fld21)&gt;"
         content="&lt;fld20&gt;&lt;fld21&gt;&lt;service&gt;&lt;dhost&gt;&lt;daddr&gt;&lt;web_method&gt;&lt;webpage&gt;&lt;web_query&gt;&lt;dport&gt;-&lt;saddr&gt;&lt;protocol&gt;&lt;user_agent&gt;-&lt;web_referer&gt;&lt;web_host&gt;&lt;resultcode&gt;&lt;result&gt;&lt;fld22&gt;&lt;sbytes&gt;&lt;rbytes&gt;&lt;duration&gt;"/>
   <MESSAGE
         id1="IIS_LOG_Level:03"
         id2="IIS_LOG_Level"
         insertBefore="IIS_LOG_Level"
         eventcategory="1204010000"
         functions="&lt;@bytes:*CALC(sbytes,+,rbytes)&gt;&lt;@web_domain:*URL($DOMAIN,web_host)&gt;&lt;@tbdstr2:*PARMVAL(web_referer)&gt;&lt;@tbdstr3:*PARMVAL(web_query)&gt;&lt;@event_time:*EVNTTIME($MSG,'%W-%G-%F %N:%U:%O',fld20,fld21)&gt;"
         content="&lt;fld20&gt;&lt;fld21&gt;&lt;service&gt;&lt;dhost&gt;&lt;daddr&gt;&lt;web_method&gt;&lt;webpage&gt;&lt;web_query&gt;&lt;dport&gt;-&lt;saddr&gt;&lt;protocol&gt;&lt;user_agent&gt;&lt;fld23&gt;&lt;web_referer&gt;&lt;web_host&gt;&lt;resultcode&gt;&lt;result&gt;&lt;fld22&gt;&lt;sbytes&gt;&lt;rbytes&gt;&lt;duration&gt;"/>
   </DEVICEMESSAGES>‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

...and make sure to reload the parsers on your log decoder (or restart the service...but just reloading the parsers is quicker).

The custom XML simply fixes the source and destination IP fields, which Dave Glover found are  backwards in the OOTB message id's that handle the log when all the IIS logging fields are selected.

[GergelyLitavszk]

Thank you both for your help, it works like a charm! I have only one more question if it's not a problem. It is more a cosmetic issue, when we collect logs there are some rows that are commented out on raw log. I would like to drop these messages, is there an easy way for this? Like modifying the file collection XML or something like that.

Example raw log:

#Software: Microsoft Internet Information Services 8.0

Thank you very much again!

BR,

Gergely

[JoshRandall]

Apologies for the very delayed response...I was just looking back over the comments and saw your question about filtering.

At the time I wrote the blog, there was no filtering capability.  That has since changed in the current release (my lab is at 11.5.1...but I'm not sure which release included this feature. I'll have to dig through the release notes to figure that out).

This looks to be a pretty straightforward capability - that main thing to consider when creating/testing your regex is that it will be looking at the original raw log (exactly as shown in your comment) and not what the log looks like after the NWE agent has picked it up.

So for your use case, the regex filter could look like any of these (really just depends on how specific/generic you want to get with the regex):

[LucaBernabei]

Hello,

thanks for the guide provided but i have a question about IIS collection using NWE Agent.

What if i have multiple log folders (like W3SVC* as you can see in the screenshot)?

I already tried using for example "C:\inetpub\logs\LogFiles\W3SVC*\*.log" but the collection doesn't work.

The SASFTP agent provide support for multiple folders, the NWE Agent can do the same?

For example part og the old agent config file where folders were set up:

dir0=C:\WINNT\system32\logfiles
dir0.filespec=*.log
dir0.dirspec=[Ww]3[Ss][Vv][Cc]*

dir1=C:\WINNT\system32\logfiles
dir1.filespec=*.log
dir1.dirspec=[Mm][Ss][Ff][Tt][Pp][Ss][Vv][Cc]*

dir2=C:\WINNT\system32\logfiles
dir2.filespec=*.log
dir2.dirspec=[Ss][Mm][Tt][Pp][Ss][Vv][Cc]*

dir3=C:\WINNT\system32\logfiles
dir3.filespec=*.log
dir3.dirspec=[Nn][Nn][Tt][Pp][Ss][Vv][Cc]*

Thanks.

Luca

[JoshRandall]

That path should work. We support multiple wildcards in the folder and filenames...

C:\inetpub\logs\LogFiles\W3SVC*\*.log

Which option do you have selected for "ON FIRST CONNECT" ?

* Collect new data only
* Collect historical and new data

[LucaBernabei]

Hello Josh,

i solved using this path:

C:\inetpub\logs\LogFiles\**\*.log

where "**" means all folders within the current folder.

Using the path "C:\inetpub\logs\LogFiles\W3SVC*\*.log" the collection doesn't work, with both option for "on first connect".

Luca

[Nijo]

@JoshRandall 

can I have the DNS flat file integration also in the same way