In Incident Response we often get called in to customer engagements after an incident has occurred (yeah, I know, that is why it is called Response). Unfortunately, not all organisations we work with have centralised log collection … or if they do, there are often still gaps in what systems / logs are collected like IIS, Remote Desktop, Terminal Services, and other Windows events. This means that we often have to manually retrieve log files from compromised systems in order to analyse them, and then we have to use native tools like Event Viewer to review and search the logs one by one. Clearly this is not very efficient, especially when you are hot on the trail of an adversary inside the network (hopefully before any destructive actions have been taken).
With recent updates to NetWitness for Logs, we now have a (more) straightforward way to process collected EVTX files. We can now process them and parse them into valuable meta data, and critically we can now use the event time from the raw event for performing queries and analysis rather than the default collection time. To do this, you will need the following:
- A windows host to run winlogbeat and process the EVTX files you’ve collected from your target systems,
- A running instance of logstash – if you don’t have an enterprise deployment, you can install it on the same host where you have winlogbeat installed,
- A running NetWitness Log Decoder,
- Custom Log Parser Rules for parsing the JSON data.
Grab the EVTX files you want to import and put them into a folder. We will use a PowerShell script to iterate through them all and process them through winlogbeat. To start with, however, we only want to process a couple of events to set up the initial configuration. Open your EVTX file in Event Viewer and export a couple of events into a new file. If you have a specific event you want to parse like a 4624 login event for example, make sure to include some of those in your export. Simply highlight the messages you want to export and Right Click or go to Actions and select Save Selected Events.
Follow the instructions to set up winlogbeat, and also review the FAQ answer on how to work with EVTX files. Also install Logstash following the instructions if you have not done so already.
Here's a bunch of links to documents that will help!
- Logstash Integration Guide for RSA NetWitness Platform 11.x - Table of Contents
- Logstash:Install Logstash
- Logstash:Install and Configure the NetWitness Codec
- Logstash:Configure Logstash Output Plugins
- <Elastic> Winlogbeat Overview | Winlogbeat Reference [7.10] | Elastic
- <Elastic> Not sure how to read from .evtx files | Winlogbeat Reference [7.10] | Elastic
There are two configuration settings that we need to set up to feed the data into NetWitness:
- The config file for winlogbeat to ingest the EVTX files, and
- The config file for logstash to tell it to send the events to NetWitness.
Winlogbeat Configuration
Here is the config file I created for winlogbeat to process the EVTX file and output to logstash – it is pretty much default settings winlogbeat-evtx.yml:
winlogbeat.event_logs:
- name: ${EVTX_FILE}
no_more_events: stop
winlogbeat.shutdown_timeout: 30s
winlogbeat.registry_file: evtx-regsitry.yml
# ------------------------------ Logstash Output -------------------------------
output.logstash:
hosts: ["localhost:5044"]Stick this file in your winlogbeat folder. Take note of the setting for winlogbeat.registry_file - this file will keep bookmark entries for each file to track the events and prevent you from re-ingesting the same events over and over again.
During testing and initial setup, you'll probably want to delete the file between runs so you can use the same file and events. The default location for the file is C:\Program Files\winlogbeat\data.
Logstash Configuration
Next step is to tell logstash what to do when it gets these events – we want it to send the events to a NetWitness Log Decoder. Here is my logstash config file nw_evtx.conf:
input {
beats {
port => 5044
}
}
output {
tcp {
id => "nw-output-tcp"
host => "192.168.2.83" #IP or Hostname of destination Log Decoder or VLC
port => 514 #Port for syslog listener
codec => netwitness
}
}Put this file in the logstash config folder (or wherever you want to store your config files). Now we have the configuration we need, start up logstash to receive events using a PowerShell terminal:
PS C:\logstash-7.10.0> .\bin\logstash -f .\config\nw_evtx.conf
If it all works properly, it should look something like this:
Now run a few events through to see that the pipeline is working and events are making it to NetWitness:
'C:\Program Files\winlogbeat\winlogbeat.exe' -e -c 'C:\Program Files\winlogbeat\winlogbeat-evtx.yml' -E EVTX_FILE=your_test_file.evtx
Check the events in NetWitness - since we do not have a parser defined yet, they will show up with the Device Type of 'unknown' but there will still be some basic parsing thanks to the default dynamic Log Parser Rules. The Collector ID will be the hostname of the system where you are running winlogbeat:
NetWitness Configuration
Now that we have events successfully being sent to NetWitness from logstash, we need to tell NetWitness how to process them. To do this we need to create JSON Mappings in the Log Parser Rules configuration, and also map the right parser to the event source.
Go to the event view for one of the unknown events. Here you should be able to toggle between the Raw Text view and Render JSON view. Raw text view:
Rendered JSON view:
Go to the Raw Text view and copy the Raw Log text into your clipboard - we're going to use it to create the JSON Mapping. Go to the Log Parser Rules page, which can now be found under the Config "sliders" icon in the top right corner of your window:
Click the button to Add Parser, and give it a good name - I called mine windowsrdp (so had to put windowsrdp1 in the feild for the screenshot):
Now paste your raw log text into the box for sample JSON Message. If you get an error (the box turns red!) trim the start of the text up until the opening parenthesis {. You should now be able to Render the JSON:
Click the button to Auto-Discover Mappings - this will pull out all of the key value pairs from the message and allow you to choose which ones you want to map into meta. This process is pretty simple - simply choose which meta to map the JSON value to from the drop down box (or type it in):
I've included my JSON mappings at the end of the post so you can see which meta keys I chose. Unfortunately there is no way (yet) to easily export or import mappings - this feature is currently still in beta after all!
One field that can be tricky is the event timestamp - make sure you map the correct timestamp as the event.time, and make sure you use the right Value Format. This is critical so that we can later sort the events by their actual Event Time rather than Collection Time. In this example, the true event time is shown at the end of the text as @timestamp (the other times are the timestamp from winlogbeat and the collection time).
The correct Value Format to use for this timestamp is InternetEventTime (InternetTime). Choosing the wrong format will result in parsing errors.
When you've finished mapping the fields you want from that message, press the button to Remove Unmapped fields and hit save but do not deploy it yet. Now go back to Event view, find another message to parse, copy the raw text, paste it into the JSON builder and map any additional fields. Do this a few times for a few different messages with different event codes. When satisfied, save the parser and then deploy it to your log decoder(s).
Since the JSON mapping and dynamic parser rules do not use Header IDs like regular log parsers to identify events, NetWitness can't automatically find the right parser to match these events. We need to do a manual parser mapping: Go to Admin / Log Decoder / Config / Parser Mappings and add a mapping for your parser to work against the logs from your logstash server ... the Admin menu is now under the tools icon:
Run the same EVTX file through that you used earlier and check that the logs are identified correctly and fields are mapped.
Remember - You will need to delete the bookmark file to get winlogbeat to read the same file and events again!
You should have all fields mapped as per your JSON mappings:
If you are still seeing some meta registered under the word meta key, your parser is not running properly. Try restarting the Log Decoder service, or issuing a Parser Reload command using the Explore view, and review the steps above.
In the NetWitness Navigate view, you can only query and view events based on their Collection Time. This means that no matter how far back your events go in your EVTX, they will all show based on their Collection Time which is stamped by the Log Decoder when it receives the event. This is clearly not ideal when importing archived events! The new Events view however does allow you to sort events based on their event time (which is what we want for investigation into older events) – you just have to set that as your preference in your profile:
If you get to the Events view with a query from Navigate (or anywhere else I think) it will start out with using Collection Time. As soon as you drill or pivot with a new query it will switch the view to Event Time. This probably means your screen will go blank with no results! Check your query timeframe and make sure it matches the timeframe for your events and not their collection time (like “last 24 hours” or “last 2 days” etc).
If you want to automatically process a whole folder of EVTX files, here is a PowerShell script that I use to iterate through all the files in a folder process_evtx.ps1:
Get-ChildItem -Path "C:\Analysis\events\" |
ForEach-Object {
& 'C:\Program Files\winlogbeat\winlogbeat.exe' -e -c 'C:\Program Files\winlogbeat\winlogbeat-evtx.yml' -E EVTX_FILE=$($_.FullName)
Happy Hunting!
And here is my JSON Mappings as promised:
<Capture name="message" key="/message" meta="msg"/>
<Capture name="name" key="/host/name" meta="alias.host"/>
<Capture name="Param1" key="/winlog/user_data/Param1" meta="user.dst"/>
<Capture name="Param3" key="/winlog/user_data/Param3" meta="ip.src"/>
<Capture name="provider_name" key="/winlog/provider_name" meta="event.source"/>
<Capture name="event_id" key="/winlog/event_id" meta="reference.id"/>
<Capture name="path" key="/log/file/path" meta="sourcefile"/>
<Capture name="@timestamp" key="/@timestamp" type="InternetEventTime"/>
<Capture name="Param2" key="/winlog/user_data/Param2" meta="domain"/>
<Capture name="TargetUserName" key="/winlog/event_data/TargetUserName" meta="user.dst"/>
<Capture name="SubjectUserName" key="/winlog/event_data/SubjectUserName" meta="user.src"/>
<Capture name="LogonType" key="/winlog/event_data/LogonType" meta="logon.type"/>
<Capture name="TargetDomainName" key="/winlog/event_data/TargetDomainName" meta="domain"/>
<Capture name="LogonProcessName" key="/winlog/event_data/LogonProcessName" meta="process"/>
<Capture name="IpAddress" key="/winlog/event_data/IpAddress" meta="ip.src"/>
<Capture name="task" key="/winlog/task" meta="category"/>
<Capture name="ParentProcessName" key="/winlog/event_data/ParentProcessName" meta="param.src"/>
<Capture name="NewProcessName" key="/winlog/event_data/NewProcessName" meta="param.dst"/>
<Capture name="computer_name" key="/winlog/computer_name" meta="alias.host"/>
<Capture name="channel" key="/winlog/channel" meta="msg.id"/>
<Capture name="param1" key="/winlog/event_data/param1" meta="event.type"/>
<Capture name="SubjectDomainName" key="/winlog/event_data/SubjectDomainName" meta="domain.src"/>
<Capture name="action" key="/event/action" meta="action"/>
<Capture name="outcome" key="/event/outcome" meta="ec.outcome"/>
<Capture name="keywords" key="/winlog/keywords/" meta="event.type"/>
