LUA - Common Event Format (CEF) Script Template

[IanRedden]

The CEF helper script attempts to fill the gap of CEF extensions that the system parser does not currently parse. For a complete list of what we do parse, please refer to Supported CEF Meta Keys - RSA Security Analytics Documentation .

This CEF helper template was written to be highly configurable as well as adhere to revision 16 of the Common Event Format (CEF) standards document. Its aim is to be able to parse meta data into any meta key in RSA Netwitness from any security appliance with the least amount of programming.

This is performed by several routines within the code that associates “csX” and “cnX” within a CEF extension with their appropriate “csXLabel” and “cnXLabel”. If no “csLabel” exists, it will use “csX” or “cnX” as the key name (where X is a numeric value). For example, a CEF message containing “cs3=name cs3Label=Ian” will set the key name to “name” and the value to “Ian” whereas a CEF message containing “cs3=Ian” will set the key name to “cs3” and the value to “Ian”. However, using scripts configuration, you can translate cs3 to any meta key name you want (i.e. username). This is configured in the t_keys_to_use variable.  More information on that below!

As CEF messages can also contain carriage returns (\n), if a carriage return is found, it will parse each line into a separate meta value of the same key name. For example, a CEF message containing “cs2=user.names cs2Label=iredden\nepartington” will result in 2 meta values (user.names) containing “iredden” and “epartington”.

The template can be downloaded at the bottom of this article.  It is configured for a FireEye HX appliance but can be easily modified for anything!

Configuration

The main configuration of the script is in 2 variables.  These variables are t_keys_to_use and b_debug.

t_keys_to_use – Is a LUA key/value table containing which keys to parse.

b_debug – By default, this variable is set to true.  This means that no meta will be created.  Instead, output will be provided for debugging to logs.

You also need to configure the cefhelper:setKeys() section of the script.  It needs to contain all the same keys from the t_keys_to_use table.  For example:

cefhelper:setKeys({
    nwlanguagekey.create("vx.threatscore"),
    nwlanguagekey.create("vx.detection"),
    nwlanguagekey.create("virusname"),
   ...
    nwlanguagekey.create("alias.host"),
    nwlanguagekey.create("vx.filedesc")
})

You can access logs on a Log Decoder via the REST API:

http://de.co.der.ip:50102/logs?msg=pull&force-content-type=text/plain&expiry=600&count=50 

Example - Accellion File Transfer (FTA)

The CEF system parser from RSA Live parses numerous CEF extension keys.  However, there are a few meta keys that we need a helper for to be able to parse CEF logs from Accellion FTA:

CEF Extension	Description	System Parser	NetWitness Key Name
msg	Message	Yes	msg
src	Source	Yes	src
deviceFacility	Device Facility	Yes	deviceFacility
fname	File Name	No	filename
fsize	File Size	No	n/a
location	Location	No	directory
type	Transfer Type	No	action
seconds	Transfer Time - # Seconds	No	n/a
suser	Source User	Yes	username

Helper Configuration:

local t_keys_to_use = {
        ["fname"] = "filename",
        ["location"] = "directory",

        ["type"] = "action"
}

cefhelper:setKeys({
   nwlanguagekey.create("filename"),
   nwlanguagekey.create("directory"),
   nwlanguagekey.create("action")
})

local b_debug = false

Example CEF Message:

CEF:0|Accellion|FTA|FTA_9_12_80|203|Download|1|msg=(172.16.20.45) test.txt downloaded by ian.redden@rsa.local (23 bytes, 3.5 sec) (Type: ssl_download, Location: ) src=172.16.20.45 deviceFacility=local5 fname=test.txt fsize=23 cs2Label=location cs2= cs1Label=type cs1=ssl_download cn1Label=seconds cn1=3.5 suser=ian.redden@rsa.local

Example - Payload Security's VxStream

The CEF system parser from RSA Live parses numerous CEF extension keys.  However, there are a few meta keys that we need a helper for to be able to parse CEF logs from Accellion FTA:

CEF Extension	Description	System Parser	NetWitness Key Name
Threat Score	Threat Score	No	vx.threatscore
AV Detection Rate	Detection Rate	No	vx.detection
Malware Family	Malware Family Name	No	virusname
fileHash	File Hash Value	No	vx.filehash
fname	File Name	No	vx.fname
fsize	File Size	No	vx.fsize
fileType	File Type	No	vx.filetype
flexString1	Comments	No	vx.comments
Client	Client	No	vx.client
Indicators	Source User	No	vx.indicators
Contacted Domains	Contacted Domains	No	alias.host
Contacted Hosts	Contacted Hosts	No	alias.host
Compromised Hosts	Compromised Hosts	No	alias.host
File Description	File Description	No	vx.filedesc

Helper Configuration:

local t_keys_to_use = {
   ["Threat Score"] = "vx.threatscore",
   ["AV Detection Rate"] = "vx.detection",
   ["Malware Family"] = "virusname",
   ["fileHash"] = "vx.filehash",
   ["fname"] = "vx.fname",
   ["fsize"] = "vx.fsize",
   ["fileType"] = "vx.filetype",
   ["flexString1"] = "vx.comments",
   ["Client"] = "vx.client",
   ["Indicators"] = "vx.indicators",
   ["Contacted Domains"] = "alias.host",
   ["Contacted Hosts"] = "alias.host",
   ["Compromised Hosts"] = "alias.host",
   ["File Description"] = "vx.filedesc"
}

cefhelper:setKeys({
   nwlanguagekey.create("vx.threatscore"),
   nwlanguagekey.create("vx.detection"),
   nwlanguagekey.create("virusname"),
   nwlanguagekey.create("vx.filehash"),
   nwlanguagekey.create("vx.fname"),
   nwlanguagekey.create("vx.fsize"),
   nwlanguagekey.create("vx.filetype"),
   nwlanguagekey.create("vx.comments"),
   nwlanguagekey.create("vx.client"),
   nwlanguagekey.create("vx.indicators"),
   nwlanguagekey.create("alias.host"),
   nwlanguagekey.create("vx.filedesc")
})

local b_debug = false

Example CEF Message:

Mar 29 15:42:59 192.168.1.100 CEF:0|Payload Security|VxStream|6.20|Sample Analysis Result - Malicious|Sample Analysis Result - Malicious|100|end=03/29/2017 21:28:55 cn1=100 cn1Label=Threat Score cn2=88 cn2Label=AV Detection Rate cs1=Trojan.Generic cs1Label=Malware Family cs2=100 cs2Label=EnvironmentID cs3=Windows 7 32 bit cs3Label=Environment Description fileHash=8271d841b9971f04d6a48804d06ecd7185d71ed8546988b1697fbe01741a8572 fname=8271d841b9971f04d6a48804d06ecd7185d71ed8546988b1697fbe01741a8572 fsize=357888 fileType=exe request=https://www.hybrid-analysis.com/sample/8271d841b9971f04d6a48804d06ecd7185d71ed8546988b1697fbe01741a8572/?environmentId\=100 msg=Malicious flexString1= flexString1Label=Uploader Comment cs4=zpr5huq4bgmutfnf.tor2web.org \nzpr5huq4bgmutfnf.onion.to \ncrl2.alphassl.com \nipinfo.io \nkosdfnure75.op1gifsd05mllk.com \ngfdkotriam.fo4j4wnq51hepa.com cs4Label=Contacted Domains cs5=84.200.69.80:53 \n34.196.176.140:80 \n185.100.85.150:443 \n192.36.27.5:443 cs5Label=Contacted Hosts cs6=84.200.69.80 \n185.100.85.150 \n192.36.27.5 cs6Label=Compromised Hosts cs8=2812134 \n2015576 \n2812134 \n2020116 \n2020716 \n2016810 \n2016810 cs8Label=ET Alerts cs9=api-12:1:4 \napi-75:1:10 \nsuricata-2:2:10 \nnetwork-21:2:10 \napi-10:1:6 \ntarget-4:2:10 \nnetwork-0:0:1 \nnetwork-2:0:5 \ntarget-25:0:3 \nnetwork-22:1:10 \nregistry-25:1:3 \nstatic-3:1:10 \ntarget-58:2:10 \nstatic-6:1:10 \navtest-3:2:10 \nsuricata-1:1:10 \napi-25:1:7 \nnetwork-27:2:10 \nregistry-27:1:10 \nnetwork-14:2:9 \napi-16:0:1 \nregistry-36:1:8 \nstring-14:1:3 \napi-51:1:5 \nregistry-35:1:5 \napi-37:0:10 \napi-76:1:10 \nnetwork-24:2:6 \nstatic-17:1:10 \nregistry-1:1:10 \nnetwork-5:1:7 \ntarget-14:2:8 \nhandle-0:1:7 \nstatic-0:1:10 \ntarget-3:0:5 \napi-27:1:10 \napi-77:1:10 \navtest-0:2:8 \nnetwork-1:0:1 \napi-55:0:7 \napi-9:1:4 \napi-42:2:7 \nstatic-60:1:10 \napi-39:0:8 \nregistry-61:1:10 \nstring-13:1:7 \nnetwork-15:2:5 \napi-26:1:10 \navtest-5:2:5 \nmutant-0:0:3 \nregistry-55:0:10 \napi-11:0:2 \napi-6:1:4 \nstatic-1:1:1 cs9Label=Indicators priority=9 cs10=PE32 executable (GUI) Intel 80386 (stripped to external PDB) \n for MS Windows cs10Label=File Description

Example – FireEye HX

FireEye HX is FireEye’s endpoint protection appliance.  There are a few meta keys that we need to parse from FireEye HX:

CEF Extension	Description	NetWitness Key Name
IOC Name	IOC Name / Malware Family	risk.warning
Target OS	Target Operating System	OS
dhost	Destination Hostname	alias.host
dst	Destination IP	ip.dst
dntdom	Destination Domain	ad.domain.dst

Helper Configuration:

local t_keys_to_use = {
        ["IOC Name"] = "risk.warning",
        ["Target OS"] = "OS”,
        ["dhost"] = "alias.host",
        ["dst"] = "ip.dst",
        ["dntdom"] = "ad.domain.dst"
}

cefhelper:setKeys({
   nwlanguagekey.create("risk.warning"),
   nwlanguagekey.create("OS"),

   nwlanguagekey.create("alias.host"),
   nwlanguagekey.create("ip.dst"),
   nwlanguagekey.create("ad.domain.dst")
})

local b_debug = false

Example CEF Message:

CEF:0|fireeye|hx|3.1.3|IOC Hit Found|IOC Hit Found|10|rt=Sep 29 2016 02:39:54 UTC dvchost=lab.rsa.local categoryDeviceGroup=/IDS categoryDeviceType=Forensic Investigation categoryObject=/Host cs1Label=Host Agent Cert Hash cs1=YUGh0fvlBG5ewBBahhbEZH dst=172.16.10.50 dmac=aa-bb-7a-fa-75-d8 dhost=victim-a3c696c8 dntdom=WORKGROUP deviceCustomDate1Label=Agent Last Audit deviceCustomDate1=Sep 29 2016 02:39:49 UTC cs2Label=FireEye Agent Version cs2=21.33.0 cs5Label=Target GMT Offset cs5=PT0H cs6Label=Target OS cs6=Windows 7 Professional 7601 Service Pack 1 externalId=1049717 start=Sep 29 2016 02:39:28 UTC categoryOutcome=/Success categorySignificance=/Compromise categoryBehavior=/Execute categoryTechnique=Exploit act=Detection IOC Hit msg=Host victim-a3c696c8 IOC compromise alert categoryTupleDescription=A Detection IOC found a compromise indication. cs4Label=IOC Name cs4=SANDSTORM (FAMILY)

DISCLAIMER: As always, this script is provided as is.  If you have any questions, feel free to reach out to me at ian.redden@rsa.com.

Happy Hunting!

Update 5/26/2017 - Updated CEF template fixing several bugs.

Update 6/14/2017 - Added sample for VxStream

ceftemplate.lua.zip