Interesting question from an internal resource about how to parse CSV files that contain information pulled from Cisco Umbrella S3 Buckets to a local filestore and how to get them into RSA NetWitness Logs. As a learning process I have documented the steps that were used to get this working, hopefully you find it useful.
Thanks to Dave Glover for the help with the parser framework and yXdv4bqA9XVNsaGmPKK98IFsBDtWJau7Yl3sVxI7k90= for the original question.
At high level we will assume that the logs are pulled down by an external script to a location that either has the SFTP agent installed or can be placed in the correct directory on the log collector for the parsing pipeline to take over in NetWitness.
Review the typespec framework as we will create a new one for the file collection method.
https://community.rsa.com/docs/DOC-54570
Create the typespec file for file collection
<?xml version="1.0" encoding="UTF-8"?>
<typespec>
<name>cisco_umbrella</name>
<type>file</type>
<prettyName>cisco_umbrella</prettyName>
<version>1.0</version>
<author>eric_partington</author>
<description>FileCollection specification for eventsource type "Cisco Umbrella" using file handler type "cisco_umbrella"</description>
<device>
<name>cisco_umbrella</name>
<parser>cisco_umbrella</parser>
</device>
<configuration>
</configuration>
<collection>
<file>
<parserId>file.cisco_umbrella</parserId>
<processorType>generic</processorType>
<dataStartLine>1</dataStartLine>
<fieldDelim>,</fieldDelim>
<idField></idField>
<lineDelim>\n</lineDelim>
<transformPrefixTag>cisco_umbrella_logs</transformPrefixTag>
<transformReplaceFieldDelim>0</transformReplaceFieldDelim>
<transformPrefixFilename>0</transformPrefixFilename>
<transformMultipleDelimiterAsOne>0</transformMultipleDelimiterAsOne>
<transformReplacementFieldDelim></transformReplacementFieldDelim>
</file>
</collection>
</typespec>
Upload the typespec file to the log collector
/etc/netwitness/ng/logcollection/content/collection/file
set the name as cisco_umbrella.xml
make sure permissions are set right (same as the other files in the directory)
Restart the log collector service
Create Event Source
In the log collector UI Set up collection
Log collector > Config > Event Sources > File > New Event Category
call it cisco_umbrella
Create a new source
File Directory is cisco_umbrella (this is the directory inside the uploads directory where files will be placed for this collection)
Address - this set the device.ip of this collection so set it to the IP of the cloud system potentially so you have a good record in device.ip of where logs 'came from' originally
Review filesystem for uploads directory created for this collection
/var/netwitness/logcollector/upload/
Start file collection mechanism on log collector
Upload test file
review that the collection mechanism works
place the csv in this directory for collection to take place
/var/netwitness/logcollector/upload/cisco_umbrella/cisco_umbrella
Review Logs
Jan 8 20:53:49 nw11ldecoder NwLogCollector[62493]: [FileCollection] [info] [file:WrkGrp[1]:64008] [getWork:651] [cisco_umbrella.cisco_umbrella] [idle] Work Unit Given to Work Manager: /var/netwitness/logcollector/upload/cisco_umbrella/cisco_umbrella/work/2017-12-13-00-10-dcd8.csv
Jan 8 20:53:49 nw11ldecoder NwLogCollector[62493]: [FileCollection] [info] [file:WrkUnit[2]:64010] [postWork:1073] [cisco_umbrella.cisco_umbrella] [processing] [generic:2017-12-13-00-10-dcd8.csv] [processing success] File processed successfully: /var/netwitness/logcollector/upload/cisco_umbrella/cisco_umbrella/work/2017-12-13-00-10-dcd8.csv
Jan 8 20:53:49 nw11ldecoder NwLogCollector[62493]: [FileCollection] [info] [file:WrkUnit[2]:64010] [postWork:1104] [cisco_umbrella.cisco_umbrella] [processing] [generic:2017-12-13-00-10-dcd8.csv] [processing success] File deleted: /var/netwitness/logcollector/upload/cisco_umbrella/cisco_umbrella/work/2017-12-13-00-10-dcd8.csv
View investigator
Default Parsing with no parser
New features added in NW11.0 are now included in the log decoders which are the lua parsers below...these help in the best effort parsing of messages to try to locate useful information from logs and parse them out.
With no parser created pulls out some information from these logs by default (new feature in NW11.0)
ip.addr, alias.host, filename, sld and tld.
Create Log Parser
Category for the logs will be Web Logs
Define the header with the LPT1.0 tool (or notepad++)
this is the raw log data as the decoder sees it
%cisco_umbrella-4: "2017-12-13 00:08:01","DC
Define the additional fields to parse out according to the columns in the CSV and the need for the data.
<?xml version="1.0" encoding="UTF-8"?>
<DEVICEMESSAGES
name="cisco_umbrella"
displayname="Cisco Umbrella:custom"
group="Web Logs"
type="7104">
<VERSION
xml="1"
revision="1"
device="2.0"/>
<TAGVALMAP/>
<HEADER
id1="HDR1"
id2="HDR1"
content="%cisco_umbrella_<messageid>-4:<!payload>"/>
<MESSAGE
id1="logs"
id2="logs"
eventcategory="1612000000" content=""<event_time_string>","<rulename>","<saddr>","<stransaddr>","<daddr>","<content_type>","<action>","<url>","<web_referer>","<user_agent>","<resultcode>","<fld1>","<fld2>","<fld3>","<uid>","<fld4>","<fld5>","<fld6>","<fld7>","<fld8>","<fld9>","<group>""/>
</DEVICEMESSAGES>
Verified in the LPT1.0 tool
save the file as cisco_umbrellamsg.xml
create the cisco_umbrella.ini for his device as well
DatabaseName=cisco_umbrella
DisplayName=cisco_umbrella
DeviceGroup=Web Logs
DeviceType=7104
save it in this directory structure for easy upload to the log decoder
/etc/devices/cisco_umbrella/place the xml and ini in this directory
zip archive the structure and rename the etc.zip as cisco_umbrella.envision
Upload the parser
Log Decoder > Config > Parsers
upload
check the filesystem to make sure the permissions are set right on the folder and files (same as the other files in the directory structure
/etc/netwitness/ng/envision/etc/devices/cisco_umbrella/
Reload the Parsers
Log Decoder > Explore menu
Decoder > Parsers - right click - select properties
reload - submit
Review the logs to ensure the parser was loaded with no errors
Cat /var/log/messages | grep –i cisco_umbrellaShould show up in the log decoder parsers list when enabled correctly
Upload new file to test collection and new parsing
place in same folder location as before
Review Parsing
parsing looks good!
review the data that was parsed and what is visible on the decoder/concentrator ( keys like referer and url may not be indexed by default depending on what other customizations have been done those may need to be added to table-map.-custom.xml and index-concentrator-custom.xml)
