There has been a lot of great information published about Sysmon since Mark Russinovich's presentation at RSA Conference. Eric Partington posted a great blog showing how to use Sysmon data with RSA NetWitness for Logs: https://community.rsa.com/community/products/netwitness/blog/2017/02/28/log-sysmon-6-windows-event-collection. This prompted RSA’s IR Team to publish details on how to get the rich tracking information generated by RSA NetWitness Endpoint that they use everyday for their incident investigations into a SIEM Here.
The aim of this blog is to show you how to collect this tracking data from RSA NetWitness Endpoint with RSA NetWitness for Logs. The collection is done via the Log Collector using a custom ODBC typespec.
*** DISCLAIMER - this is a field developed Proof of Concept, shared with the Community. It is not endorsed by RSA Engineering. The database structure used by NWE may change at any time. No testing has been done to measure the impact on performance for a production NWE Server. This has been developed and tested using RSA NetWitness Endpoint v4.3.0.1 and RSA NetWitness for Logs v10.6.2.1. /DISCLAIMER ***
***DISCLAIMER 2 - for this Proof of Concept, we have disabled the requirement on the NWE SQL Server to Force Encryption. /DISCLAIMER 2 ***
The objective of this integration is to get the tracking data from NWE as it is being collected into NWL, so we can index it and use it for Investigations. Tracking data in NWE can only be viewed on a per machine basis - this integration allows us to get a global view of tracking data across all of our endpoints. Here's the high level summary of what we need to do (if you want to skip to the end, all files are attached as a zip):
- Create a new ODBC typespec definition (XML file) to query the NWE Database and get the data we want,
- Create a new Log Parser to map the results of the SQL query into metadata,
- Add the meta we are using to the table-map-custom.xml so it is persistent,
- Add the meta we want to index to the index-concentrator-custom.xml file,
- Configure a new ODBC DSN definition,
- Configure a new ODBC Event Collector,
- Configure a new Meta Group to show our data for investigations,
- Configure a new Column Group to show the data we want in Events view,
- Configure some Report Rules and Charts to visualise the data,
- Configure a new RSA NetWitness Endpoint Dashboard to keep track of our environment.
Here we go!
1. Create ODBC Definition
Thanks to Andreas Funk and his blog https://community.rsa.com/community/products/netwitness/blog/2016/10/18/integrating-a-mysql-community-database-with-netwitness-for-logs for giving us a primer on how to create a new ODBC connection. We need to create a new Filespec to tell the ODBC collector how to query the NWE database and get the data we want.
On the Log Collector (either the one on the Log Decoder, or a separate VLC - whichever you are going to use to collect these logs) the ODBC collection definitions are stored here:
/etc/netwitness/ng/logcollection/content/collection/odbc/
We need to add a new file for our NWE tracking data -
vi /etc/netwitness/ng/logcollection/content/collection/odbc/nwe_tracking.xml
Here is the query from Rui Ataide's blog, modified to work for NWL, included in our definition:
<?xml version="1.0" encoding="UTF-8"?>
<typespec>
<name>nwe_tracking</name>
<type>odbc</type>
<prettyName>NetWitness Endpoint Tracking</prettyName>
<version>2.0</version>
<author>Chris Thomas</author>
<description>Import NWE Tracking data</description>
<device>
<name>nwe_tracking</name>
</device>
<configuration>
</configuration>
<collection>
<odbc>
<query>
<tag>nwe_tracking</tag>
<outputDelimiter>||</outputDelimiter>
<interval>30</interval>
<dataQuery>
(SELECT
SE.PK_WinTrackingEvents,
SE.EventUTCTIme,
MA.MacAddress as src_mac,
MA.LocalIp as src_ip,
MA.MachineName,
LOWER(PA.Path),
LOWER(FN.FileName),
LOWER(PA.Path + FN.FileName) AS Source,
MO.HashSHA256,
LA.LaunchArguments AS SLA,
CASE
WHEN SE.BehaviorFileOpenPhysicalDrive = 1 THEN 'OpenPhysicalDrive'
WHEN SE.BehaviorFileReadDocument = 1 THEN 'ReadDocument'
WHEN SE.BehaviorFileWriteExecutable = 1 THEN 'WriteExecutable'
WHEN SE.BehaviorFileRenameToExecutable = 1 THEN 'RenameExecutable'
WHEN SE.BehaviorProcessCreateProcess = 1 THEN 'CreateProcess'
WHEN SE.BehaviorProcessCreateRemoteThread = 1 THEN 'CreateRemoteThread'
WHEN SE.BehaviorProcessOpenOSProcess = 1 THEN 'OpenOSProcess'
WHEN SE.BehaviorProcessOpenProcess = 1 THEN 'OpenProcess'
WHEN SE.BehaviorFileSelfDeleteExecutable = 1 THEN 'SelfDelete'
WHEN SE.BehaviorFileDeleteExecutable = 1 THEN 'DeleteExecutable'
WHEN SE.BehaviorRegistryModifyBadCertificateWarningSetting = 1 THEN 'ModifyBadCertificateWarningSetting'
WHEN SE.BehaviorRegistryModifyFirewallPolicy = 1 THEN 'ModifyFirewallPolicy'
WHEN SE.BehaviorRegistryModifyInternetZoneSettings = 1 THEN 'ModifyInternetZoneSettings'
WHEN SE.BehaviorRegistryModifyIntranetZoneBrowsingNotificationSetting = 1 THEN 'ModifyIntranetZoneBrowsingNotificationSetting'
WHEN SE.BehaviorRegistryModifyLUASetting = 1 THEN 'ModifyLUASetting'
WHEN SE.BehaviorRegistryModifyRegistryEditorSetting = 1 THEN 'ModifyRegistryEditorSetting'
WHEN SE.BehaviorRegistryModifyRunKey = 1 THEN 'ModifyRunKey '
WHEN SE.BehaviorRegistryModifySecurityCenterConfiguration = 1 THEN 'ModifySecurityCenterConfiguration'
WHEN SE.BehaviorRegistryModifyServicesImagePath = 1 THEN 'ModifyServicesImagePath'
WHEN SE.BehaviorRegistryModifyTaskManagerSetting = 1 THEN 'ModifyTaskManagerSetting'
WHEN SE.BehaviorRegistryModifyWindowsSystemPolicy = 1 THEN 'ModifyWindowsSystemPolicy'
WHEN SE.BehaviorRegistryModifyZoneCrossingWarningSetting = 1 THEN 'ModifyZoneCrossingWarningSetting'
END AS Action,
LOWER(SE.Path_Target),
LOWER(SE.FileName_Target),
LOWER(SE.Path_Target + SE.FileName_Target) AS Destination,
SE.LaunchArguments_Target AS TLA,
se.HashSHA256_Target
FROM
dbo.WinTrackingEvents_P1 AS SE WITH(NOLOCK)
INNER JOIN dbo.Machines AS MA WITH(NOLOCK) ON MA.PK_Machines = SE.FK_Machines
INNER JOIN dbo.MachineModulePaths AS MP WITH(NOLOCK) ON MP.PK_MachineModulePaths = SE.FK_MachineModulePaths
INNER JOIN dbo.Modules AS MO WITH(NOLOCK) ON MO.PK_Modules = MP.FK_Modules
INNER JOIN dbo.FileNames AS FN WITH(NOLOCK) ON FN.PK_FileNames = MP.FK_FileNames
INNER JOIN dbo.Paths AS PA WITH(NOLOCK) ON PA.PK_Paths = MP.FK_Paths
INNER JOIN dbo.LaunchArguments AS LA WITH(NOLOCK) ON LA.PK_LaunchArguments = SE.FK_LaunchArguments__SourceCommandLine
WHERE PK_WinTrackingEvents > '%TRACKING%'
UNION
SELECT
SE.PK_WinTrackingEvents,
SE.EventUTCTIme,
MA.MacAddress as src_mac,
MA.LocalIp as src_ip,
MA.MachineName,
LOWER(PA.Path),
LOWER(FN.FileName),
LOWER(PA.Path + FN.FileName) AS Source,
MO.HashSHA256,
LA.LaunchArguments AS SLA,
CASE
WHEN SE.BehaviorFileOpenPhysicalDrive = 1 THEN 'OpenPhysicalDrive'
WHEN SE.BehaviorFileReadDocument = 1 THEN 'ReadDocument'
WHEN SE.BehaviorFileWriteExecutable = 1 THEN 'WriteExecutable'
WHEN SE.BehaviorFileRenameToExecutable = 1 THEN 'RenameExecutable'
WHEN SE.BehaviorProcessCreateProcess = 1 THEN 'CreateProcess'
WHEN SE.BehaviorProcessCreateRemoteThread = 1 THEN 'CreateRemoteThread'
WHEN SE.BehaviorProcessOpenOSProcess = 1 THEN 'OpenOSProcess'
WHEN SE.BehaviorProcessOpenProcess = 1 THEN 'OpenProcess'
WHEN SE.BehaviorFileSelfDeleteExecutable = 1 THEN 'SelfDelete'
WHEN SE.BehaviorFileDeleteExecutable = 1 THEN 'DeleteExecutable'
WHEN SE.BehaviorRegistryModifyBadCertificateWarningSetting = 1 THEN 'ModifyBadCertificateWarningSetting'
WHEN SE.BehaviorRegistryModifyFirewallPolicy = 1 THEN 'ModifyFirewallPolicy'
WHEN SE.BehaviorRegistryModifyInternetZoneSettings = 1 THEN 'ModifyInternetZoneSettings'
WHEN SE.BehaviorRegistryModifyIntranetZoneBrowsingNotificationSetting = 1 THEN 'ModifyIntranetZoneBrowsingNotificationSetting'
WHEN SE.BehaviorRegistryModifyLUASetting = 1 THEN 'ModifyLUASetting'
WHEN SE.BehaviorRegistryModifyRegistryEditorSetting = 1 THEN 'ModifyRegistryEditorSetting'
WHEN SE.BehaviorRegistryModifyRunKey = 1 THEN 'ModifyRunKey '
WHEN SE.BehaviorRegistryModifySecurityCenterConfiguration = 1 THEN 'ModifySecurityCenterConfiguration'
WHEN SE.BehaviorRegistryModifyServicesImagePath = 1 THEN 'ModifyServicesImagePath'
WHEN SE.BehaviorRegistryModifyTaskManagerSetting = 1 THEN 'ModifyTaskManagerSetting'
WHEN SE.BehaviorRegistryModifyWindowsSystemPolicy = 1 THEN 'ModifyWindowsSystemPolicy'
WHEN SE.BehaviorRegistryModifyZoneCrossingWarningSetting = 1 THEN 'ModifyZoneCrossingWarningSetting'
END AS Action,
LOWER(SE.Path_Target),
LOWER(SE.FileName_Target),
LOWER(SE.Path_Target + SE.FileName_Target) AS Destination,
SE.LaunchArguments_Target AS TLA,
se.HashSHA256_Target
FROM
dbo.WinTrackingEvents_P0 AS SE WITH(NOLOCK)
INNER JOIN dbo.Machines AS MA WITH(NOLOCK) ON MA.PK_Machines = SE.FK_Machines
INNER JOIN dbo.MachineModulePaths AS MP WITH(NOLOCK) ON MP.PK_MachineModulePaths = SE.FK_MachineModulePaths
INNER JOIN dbo.Modules AS MO WITH(NOLOCK) ON MO.PK_Modules = MP.FK_Modules
INNER JOIN dbo.FileNames AS FN WITH(NOLOCK) ON FN.PK_FileNames = MP.FK_FileNames
INNER JOIN dbo.Paths AS PA WITH(NOLOCK) ON PA.PK_Paths = MP.FK_Paths
INNER JOIN dbo.LaunchArguments AS LA WITH(NOLOCK) ON LA.PK_LaunchArguments = SE.FK_LaunchArguments__SourceCommandLine
WHERE PK_WinTrackingEvents > '%TRACKING%' )
ORDER By SE.PK_WinTrackingEvents ASC
</dataQuery>
<trackingColumn>PK_WinTrackingEvents</trackingColumn>
<maxTrackingQuery> SELECT MAX(PK_WinTrackingEvents) FROM dbo.WinTrackingEvents_P0</maxTrackingQuery>
</query>
</odbc>
</collection>
</typespec>
This creates a log entry with a static format, that is delimited by a double pipe ||:
This makes it easy for us to create a new log parser.
2. Create a new Log Parser
For information on how to create a new log parser using the new Log Parser Tool, head over here: https://community.rsa.com/community/products/netwitness/blog/2017/03/08/rsa-netwitness-esi-tool-10-beta-2. We need to create a new directory where the Log Decoder parsers are kept, and add our ini and xml parser files
mkdir /etc/netwitness/ng/envision/etc/devices/nwe_tracking/
Here is the ini file that describes our parser: nwe_tracking.ini
DatabaseName=nwe_tracking
DisplayName=NetWitness Endpoint Tracking
DeviceGroup=
DeviceType=7104
And here is the Log Parser: v20_nwe_trackingmsg.xml - the meta keys to use were chosen to line up with where the data from sysmon gets mapped to, as shown here: https://community.rsa.com/community/products/netwitness/blog/2017/02/28/log-sysmon-6-windows-event-collection
<?xml version="1.0" encoding="UTF-8"?>
<DEVICEMESSAGES
name="nwe_tracking"
displayname="NetWitness Endpoint Tracking"
group=""
type="7104">
<VERSION
xml="1"
revision="1"
device="2.0"/>
<HEADER
id1="HDR1"
id2="HDR1"
messageid="STRCAT('NWEPMSG')"
content="%nwe_tracking:<trans_id>||<event_time>||<!payload:trans_id>"/>
<MESSAGE
id1="NWEPMSG"
id2="NWEPMSG"
eventcategory="1612000000" content="<trans_id>||<event_time>||<smacaddr>||<saddr>||<event_computer>||<directory>||<filename>||<parent_process>||<checksum>||<parent_params>||<category>||<directory>||<filename>||<process>||<params>||<checksum>"/>
</DEVICEMESSAGES>
There should be 2 files in the new directory:
[root@RSAANZSCSA nwe_tracking]# pwd
/etc/netwitness/ng/envision/etc/devices/nwe_tracking
[root@RSAANZSCSA nwe_tracking]# ls -l
total 8
-rw-r--r--. 1 root root 96 Mar 9 10:01 nwe_tracking.ini
-rw-r--r--. 1 root root 761 Mar 10 02:59 v20_nwe_trackingmsg.xml
[root@RSAANZSCSA nwe_tracking]#
3. Add meta to table-map-custom.xml
This step can be done using the Web GUI, but since we're already on the command line we'll do it there. It's always a good idea to make a back up copy of the file first!
cp /etc/netwitness/ng/envision/etc/table-map-custom.xml /etc/netwitness/ng/envision/etc/table-map-custom.xml.old
Then edit the table-map-custom.xml file:
vi /etc/netwitness/ng/envision/etc/table-map-custom.xml
We can add the meta we are using (that is not already set as persistent (flags="None") at the end of the file:
<!-- NWE Tracking Data
-->
<mapping envisionName="smacaddr" nwName="eth.src" flags="None" format="MAC" envisionDisplayName="SourceMacAddress" nullTokens="Unknown"/>
<mapping envisionName="checksum" nwName="checksum" flags="None"/>
<mapping envisionName="parent_params" nwName="parent.params" flags="None"/>
<mapping envisionName="process" nwName="process" flags="None"/>
<mapping envisionName="parent_process" nwName="parent.process" flags="None"/>
<mapping envisionName="params" nwName="params" flags="None"/>
<mapping envisionName="directory" nwName="directory" flags="None"/>
<mapping envisionName="category" nwName="category" flags="None"/>
Now that we've finished the modifications for the Log Collector and Log Decoder, restart those services so that the changes get loaded.
4. Add meta for indexing to index-concentrator-custom.xml
Again, you can do this in the GUI, but since we're on the command line already we'll do it there. Just make sure you switch to your Concentrator first! (I'm on a hybrid 
). Again - make a backup first
cp /etc/netwitness/ng/index-concentrator-custom.xml /etc/netwitness/ng/index-concentrator-custom.xml.old
Then edit the file
vi /etc/netwitness/ng/index-concentrator-custom.xml
Add the new meta to index at the end of the file - you may need to add more keys depending on your existing index settings:
<!-- NWE Tracking Data -->
<key description="Checksum" format="Text" level="IndexValues" name="checksum" valueMax="1000000" defaultAction="Open"/>
<key description="Parent Process" format="Text" level="IndexValues" name="parent.process" valueMax="1000000" defaultAction="Open"/>
<key description="Parent Process Parameters" format="Text" level="IndexValues" name="parent.params" valueMax="1000000" defaultAction="Open"/>
<key description="Process Parameters" format="Text" level="IndexValues" name="params" valueMax="1000000" defaultAction="Open"/>
<key description="Category" format="Text" level="IndexValues" name="category" valueMax="1000000" defaultAction="Open"/>
Restart the Concentrator service so that the changes get loaded.
5. Configure new ODBC DSN definition.
Now we can switch to the GUI for our configuration. Go to your Log Collector Config page, and create a new DSN.
Enter the details to connect to your NWE Database (do not use a template) and click Save:
6. Configure a new ODBC Event Collector
On the Log Collector Config page, create a new ODBC Event Category by selecting our new nwe_tracking source from the list:
Now add a new Event Source and enter the details for your NWE SQL Database:
Click Test Connection to see that it all works ...
If it's not turned on already, start ODBC collection method (and set it to auto-start). Now you should be collecting NWE Tracking events! run a query for device.type = 'nwe_tracking' to see:
The remaining steps go through ways to use the NWE Tracking data.
7. Configure a Meta Group to show the NWE Meta in Investigations
If you have a favourite Meta Group you use, just add these Meta keys to it. Otherwise, create a new Meta Group called NW Endpoint Tracking. Here's what I have in mine:
Note - I have all my Meta Keys set to open for testing purposes. Best Practice is to set did to open, and all other keys to closed. This gives better performance with large datasets by not sending 22 queries to the Concentrator at the same time.
Here's what you should be able to see:
By mapping the process name into the filename meta key, the data will trigger any feeds that are looking for matches on filename. The Investigation and Hunting feeds match this data:
8. Configure a new Column Group
The default view for reviewing logs in the Event Viewer is very simple:
We can change this view to show the meta extracted from our NWE Tracking logs. Create a new Column Group
Note that you can change the "Display Name" to something you like - this will be used for the column heading:
9. Configure Report Engine Rules and Charts
Use the new device.type = 'nwe_tracking' to create rules to use for reports and charts. Here's a rule to query on the Source Process (parent.process😞
results:
We can then use the rule as a basis for a chart:
10. Create a RSA NetWitness Endpoint Dashboard
One you create the charts that you want, you can create a new Dashboard to keep track of your environment. Simply create a new Dashboard, and add your charts as Dashlets using "Reports Realtime Chart"
All the files mentioned in this post are available for download in the zip below.
Happy Hunting!
Thanks to Rui Ataide & Eric Partington for their contributions to this integration.
2 Comments
