I have been working with a few customers to add custom CEF log sources to SA and got into using Lua to parse logs instead of customizing the cef.xml parser or other default parsers. VxStream logs came my way via a side project from the developers of the sandbox software from Payload Security.
https://www.payload-security.com/products/vxstream-sandbox
If you are looking for an alternative sandbox this one looks pretty interesting with a huge number of behaviour detections to flag and create reports. I haven't focused on getting the files from either packets or malware to VxStream Sandbox yet but we were assured that there is an API that can be leveraged to post files to the sandbox.
Back to CEF logs... how would be onboard these CEF formatted logs to RSA NetWitness logs without customizing the default cef.xml parser ? CEF by default will parse the items in the first part of the message where the | values are. Once you get past that, the cn* and cs* will need to be extracted manually with Lua.
Here is a sample log from Payload Security. (default extractions in bold):
Aug 18 10:26:15 aaa.xxx.t-yyyy.de CEF:0|Payload Security|VxStream|5.00|Sample Analysis Result - Malicious|Sample Analysis Result - Malicious|100|end=08/18/2016 15:22:05 cn1=100 cn1Label=Threat Score cn2=62 cn2Label=AV Detection Rate cs1=Trojan.GenericKD cs1Label=Malware Family cs2=4 cs2Label=EnvironmentID cs3=W7 32 bit Kernelmode cs3Label=Environment Description fileHash=8d79bba763f5cbe4b778ddae6de1c97a9aca7049763466ffc289cf1306c71932 fname=Multi_Process.bin fsize=2474496 fileType=PE32 executable (GUI) Intel 80386, for MS Windows request=https://www.hybrid-analysis.com/sample/8d79bba763f5cbe4b778ddae6de1c97a9aca7049763466ffc289cf1306c71932/?environmentId\=4 msg=Malicious flexString1= flexString1Label=Uploader Comment cs4=degreat247.no-ip.biz \ndegreat248.no-ip.org cs4Label=Contacted Domains cs5=69.62.234.85:7070 cs5Label=Contacted Hosts cs6= cs6Label=Compromised Hosts cs8=2013743 \n2013743 \n2013743 \n2013743 \n2013743 \n2013743 \n2013743 \n2013743 \n2013743 \n2013743 cs8Label=ET Alerts priority=9
From this we are going to extract meta from the CEF format where the default data isn't extracted (cs or cn fields):
Fields extracted:
· Device IP -> IP of the VxStream Sandbox sending the logs
· Medium -> 32 is RSA NW internal for logs (packets is 1)
· Device.type -> payload_security_vxstream (Payload Security|VxStream)
· Event.time.str -> analysis start time (08/18/2016 15:22:05)
· Alias.host -> domain name of the VxStream service/appliance (aaa.xxx.t-yyyy.de)
· Product-> VxStream (VxStream)
· Version -> version of the VxStream service/appliance (5.00)
· Event.type -> from the CEF message (Sample Analysis Result – Malicious)
· Event.desc -> from the CEF message (Sample Analysis Result – Malicious)
· Severity -> from the CEF message (100)
· Checksum -> filehash (8d79bba763f5cbe4b778ddae6de1c97a9aca7049763466ffc289cf1306c719320)
· Filename -> fname (Multi_Process.bin)
· Extension -> from the filename (.bin)
· Filename.size -> fsize (2474496)
· url -> request (https://www.hybrid-analysis.com/sample/8d79bba763f5cbe4b778ddae6de1c97a9aca7049763466ffc289cf1306c71932/?environmentId\=4 msg=Malicious flexString1= flexString1Label=Uploader Comment)
· virusname -> cs1 (Trojan.GenericKD)
· risk.num.sand -> cn1 (75)
o Above 90 – very sure
o 75 – pretty sure
· Event.type -> Sample Analysis Result – Malicious ( Malicious, Suspicious, No Threat, Unknown)– matches with RSA Sandbox malware meta
To Do (requires more Lua foo)
· alias.host -> cs4 (cs4=degreat247.no-ip.biz \ndegreat248.no-ip.org cs4Label=Contacted Domains
· ip.dst -> cs6 (cs6=52.58.99.202 \n38.229.70.4 \n217.197.83.197 \n93.184.220.29 \n52.85.184.221 cs6Label=Compromised Hosts)
To get these fields to be indexed you need to add the following changes to the index:
Log Decoder
<!-- checksum malware hash -->
<mapping envisionName="checksum" nwName="checksum" flags="None"/>
Concentrator
Index-concentrator-custom.xml <!-- checksum meta for vxstream logs--> <key description="Checksum" format="Text" level="IndexValues" name="checksum" valueMax="250000" defaultAction="Open"/>
Restart the services to bring those keys online
You might want to create meta profile to help you locate the logs and set the metagroup for you automatically.
[UPDATE] Adding the Lua parser to the Log Decoder
- Admin > Services > Log Decoder > Config > Parsers Tab
- Click upload
- Locate the Lua parser
- Click upload
- watch for green bar for success
- close the window
- go back to the Config > General tab
- wait for the parsers to reload (screen may stay blank for a bit while they reload)
now the parser will show up top right in the Parsers section
Here is how the parser looks when installed:
Here is how the meta looks when extracted from the logs (parser shows up in the upper right section , not the usual bottom right 😞
Metaprofile (malware sandbox)
Device.type = payload_security_vxstream
Meta group = malware sandbox
Metagroup (malware sandbox)
Device.type
Device.class
Filename
Filetype
Checksum
Version
Event.type
Event.desc
Did
Event.source
Virusname
Alias.host (tbd)
Ip.dst (tbd)
Risk.num.sand
You might also want to create app rules on your log decoders to flag for events of interest (sandbox detect malicious file but no AV signature for it, high confidence detection in malicious file):
Application Rules
name=nwfl_malicious_file_no_av_detection rule="device.type='payload_security_vxstream' && event.type = 'malicious file' && virusname !exists" alert=alert order=50 type=application
name=nwfl_malicious_file_high_confidence rule="device.type='payload_security_vxstream' && event.type = 'malicious file' && risk.num.sand = 90-u" alert=alert order=49 type=application
