Extracting filenames from Ironport Proxy Logs

[KEVINDIENST]

#lua‌ #parsing‌ #proxy‌ #ironport Christopher Ahearn‌

I have a question regarding how feasible it is to extract from an ironport proxy log the value after the last / in a URL?

example:

httpX://www.badsite.com/attacker/firefox_updater.exe

I want to extract everything after that last slash. I tweaked a regex and it works, but since app rules just do true/false for regex, not parsing, I assume I may have to use a Lua parser to extract that information. 

Sample Regex: (?:[^\/][\d\w\.\-]+)$(?<=(?:.))

Link to Lua on patterns: lua-users wiki: Patterns Tutorial 

1. Yes I know there is a ton of stuff after that / that will be completely garbage, not related to any actual filename. 

2. We already index url meta from the ironports, yes the index is massive (a few billion unique entries a day) but we have to be able to search against that content, even then however using url contains ".exe" over a long period of time isn't fast for our analysts. 

If there is another method to do this I'm open to that, if it isn't really feasible, how is the community doing this type of searching today? 

Use Case: I found out that machine A got malware from httpX://www.badsite.com/attacker/firefox_updater.exe and want to see if the attacker may be hosting this same malicious file across other throw away sites. 

Thanks!

[Anonymous]

Kevin,

This is quite feasible. You could do a meta callback against the url meta key and break that meta value out into hostname, directory, filename and even query.

I can share an example shortly of what I'm referring to.

Chris

Sent from my mobile device

[KEVINDIENST]

Yes please, if you have an example I'd love to see it.

[Anonymous]

Kevin,

I'm currently battling some network demons at the moment so I'm not able to upload an example but will do so asap.

However, I did write a blog post on RSA Link that may help out.

https://community.rsa.com/community/products/netwitness/blog/2016/06/07/who-referred-you-getting-more-out-of-http-referer-meta

In this parser, I'm doing a meta callback against the 'referer' meta key to extract some juicy bits.

I'm thinking the same could be done against your 'url' meta key as well. We'd probably want to change the meta keys we write into to the default ones like alias.host, directory, filename, extension, and query respectively.

I had commented out those areas I wasn't interested in for 'referers', but you may be interested in for the proxy log url data.

Hope to have network connectivity back shortly to get you some examples.

Chris

Sent from my mobile device

[Anonymous]

Another example of a previous version of the parser.  

****************************** BEGIN HERE ******************************

local urlpath = nw.createParser("URL_PATH2", "Extract directory, filename, and extension from proxy device logs")

--[[
DESCRIPTION

Extract path data from webpage

VERSION

2015-11-05 - Initial development


AUTHOR

christopher.ahearn@rsa.com


DEPENDENCIES

This parser requires 'nwll'.

You can download from Live (Search, Lua Parsers). Extract the contents of the
nwll.zip file and upload via Administration/Log Decoder, Config, Parsers or place
in /etc/netwitness/ng/parsers directory on the Log Decoder.


NOTES

None

--]]

-- Since we are using an external module, we declare it here.
-- This must be in the parsers directory
--local nwll = require('nwll')

-- commented out the require('nwll') because some log decoders are not set up with the nwll extra functions file. I just copied it here.

-- These are the meta keys that we will write meta into
urlpath:setKeys({
nwlanguagekey.create("directory"),
nwlanguagekey.create("filename"),
nwlanguagekey.create("extension"),
})

function extractPathElements(fullpath)
if not fullpath or #fullpath == 0 then
do
return
end
end
local foundPosition, lastPosition, dir, file, ext
local stringFind, stringSub = string.find, string.sub
foundPosition = stringFind(fullpath, "/")
if foundPosition then
while foundPosition do
lastPosition = foundPosition
local stringFind = stringFind
foundPosition = stringFind(fullpath, "/", lastPosition + 1)
end
else
foundPosition = stringFind(fullpath, "\\")
while foundPosition do
lastPosition = foundPosition
local stringFind = stringFind
foundPosition = stringFind(fullpath, "\\", lastPosition + 1)
end
end
if lastPosition then
local stringSub = stringSub
dir = stringSub(fullpath, 1, lastPosition)
file = stringSub(fullpath, lastPosition + 1, -1)
else
file = fullpath
end
lastPosition = nil
local foundPosition = stringFind(file, "%.", 1)
while foundPosition do
lastPosition = foundPosition
local stringFind = stringFind
foundPosition = stringFind(file, "%.", lastPosition + 1)
end
if lastPosition then
ext = stringSub(file, lastPosition + 1, -1)
end
return dir, file, ext
end

-- This is our function. What we want to do when we match a token...or in this case, the
-- URL meta callback.
function urlpath:urlPath(index, path)
local somePATH = path


-- apply the nwll.extractUrlElements function from the nwll module
local directory, filename, extension = extractPathElements(somePATH)

if directory then
nw.createMeta(self.keys["directory"], directory)
--nw.logInfo("DIRECTORY: " .. directory)
end
if filename then
nw.createMeta(self.keys["filename"], filename)
--nw.logInfo("FILENAME: " .. filename)
end
if extension then
nw.createMeta(self.keys["extension"], extension)
--nw.logInfo("EXTENSION: " .. extension)
end
end

urlpath:setCallbacks({
[nwlanguagekey.create("web.page")] = urlpath.urlPath, -- this is the meta callback key
--[nwlanguagekey.create("url")] = urlpath.urlPath -- we can also use url if that is what is necessary
})

******************************* END HERE *******************************

In this one, i was doing a meta callback of web.page, but had one that also did the same thing against 'url'.  I think between the two, there should be enough to get going.  

Chris