This website uses cookies. By clicking Accept, you consent to the use of cookies. Click Here to learn more about how we use cookies.
Accept
Reject
Register Now
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
NetWitness Discussions

LUA Parser to convert meta separated by commas into single meta values

DavidWaugh1
Employee
Employee

‎2016-01-25 04:20 AM

If you are using the CEF parser, some event sources may return meta in the form of comma separated lists.

 

For example in the following log message:

 

Jan 25 08:43:09 rsareNsa CEF: 0|RSA|Security Analytics Malware|10.5.1.2.8514.5.0|Suspicious Event|Detected suspicious network event|2|nextgen=100.0 event.type=NEXTGEN event.id=34954 country.dst.code=US city.dst=Dublin org.dst=Amazon.com payload=31779809 packets=32690 country.dst=Ireland,Internal time=Mon Jan 25 08:37:29 UTC 2016 threat.source=snort rule,netwitness filetype=x86 pe,windows_executable,windows executable latdec.dst=53.3331 eth.src=00:50:56:03:01:C8 ip.proto=6 tcp.flags=24 ip.src=192.168.123.250 tcp.dstport=80 threat.category=spectrum,suspicious,malware,shellcode-detect,data leakage,attempted-user,informational eth.dst=00:50:56:03:08:2B lifetime=41 did=rsadecoder alert.id=nw05130,nw20045,nw25130,nw30035,nw30040,nw32505,nw32765,nw110060,\rO,.U,3y,4y,cA,jQ,xR,ˆ.,¥L,Ã6,Ä6 sessionid=46822006 medium=1 size=33557045 ad.username.src=Administrator rpackets=97 action=get ad.domain.src=WAUGH eth.src.vendor=VMware, Inc. rpayload=97 content=spectrum.analyze,application/octet-stream,spectrum.consume extension=exe eth.dst.vendor=VMware, Inc. rid=329515862 directory=/offlineupdate/avg_v26141/ risk.suspicious=abnormal exe,escalation multiple informational,packer armadillo,fake antivirus malware indicators eth.type=2048 ip.dst=54.229.143.120 service=80 filename=Metascan_Offline_Updater_avg_v26141_1453659907.exe streams=2 risk.info=flags_ack,flags_psh,FILE-MULTIMEDIA Microsoft Windows DirectX directshow wav file overflow attempt,INDICATOR-SHELLCODE x86 NOOP,FILE-MULTIMEDIA MultiMedia Jukebox playlist file handling heap overflow attempt,high risk filetypes,http1.1 without referer header,FILE-MULTIMEDIA MultiMedia Soft Components AdjMmsEng.dll PLS file processing buffer overflow attempt,FILE-OTHER Adobe Acrobat EMF conversion heap buffer overflow attempt,outbound_traffic,large outbound data transfer,FILE-OTHER Interactive Data eSignal stack buffer overflow attempt,FILE-MULTIMEDIA CyberLink PowerDVD playlist file handling stack overflow attempt,FILE-MULTIMEDIA RealNetworks RealPlayer mpeg width integer memory under

 

The values of content, alert.id, risk.info,risk.suspicious threat.source are all comma separated lists.

 

This makes investigation on these values harder, as you would have to use an expression such as risk.info contains <some value> in order to match this meta across events.

 

The following LUA parser takes care of this for you as it splits comma separated lists of meta, into individual meta values.

 

Currently the parser only works on events where the device.type is netwitnessspectrum or rsa_security_analytics_esa.

 

It also works on the meta keys:

  • filename
  • threat.source
  • threat.category
  • content.type
  • risk.info
  • risk.suspicious
  • action
  • alert.id
  • filetype

 

The parser though can be easily modified to handle other device types or metakeys.

array2meta.lua.zip
13 REPLIES 13

AlexeyFedorov
Frequent Contributor
Frequent Contributor

‎2016-02-24 03:11 AM

It will be very situable for me and I assume for many other users of this community if you make same for MS Exchnage user.dst metakey where array maybe separate by semicolon and metakey can have only one value (not array).

0 Likes

DavidWaugh1
Employee
Employee
In response to AlexeyFedorov

‎2016-02-24 04:02 AM

Good  Morning Alex,

 

Can you paste a log sample (just anonymize any private info) and I'll put something together.

0 Likes

AlexeyFedorov
Frequent Contributor
Frequent Contributor
In response to DavidWaugh1

‎2016-02-24 04:24 AM

Hello David,

 

I'm not able to send you raw logs today, but I want to say: device.type = 'msexchnage' and user.dst can be 'xxx@xxx.xx' or 'xxx@xxx.xx;yyy@yyy.yy;zzz@zzz.zz' (2 or more emails separated by semicolon). You LUA parser correctly work with any array like 'xxx@xxx.xx;yyy@yyy.yy;zzz@zzz.zz' if change a separate symbol. But I don't want to see source array 'xxx@xxx.xx;yyy@yyy.yy;zzz@zzz.zz' in  user.dst metakey. If I change type of metakey 'to' to transient - I don't see single user.dst like 'xxx@xxx.xx' only result of work your parser.

0 Likes

DavidWaugh1
Employee
Employee

‎2016-02-24 04:47 AM

Ah okay. Is the request here to write a parser that will work on all user.dst fields (whatever the device type).

 

Is the plan something like

- Set user.dst to transient

 

- Look at user.dst field

- if user.dst is a comma separated list, then write out seperate user.dst

-if user.dst is an LDAP list then write it out as firstname.lastname

- else just write out user.dst

 

Am I understanding you correctly?

0 Likes

AlexeyFedorov
Frequent Contributor
Frequent Contributor

‎2016-02-24 05:00 AM

Not all correct, correct is:

 

Is the plan something like

- Set user.dst to transient

 

- Look at user.dst field

- if user.dst is a semicolon separated list, then write out seperate user.dst

- else just write out user.dst

No LDAP parsing and work with only user.dst field.

PS: Your current array parser don't write out result if it not be in array.

0 Likes

DavidWaugh1
Employee
Employee
In response to AlexeyFedorov

‎2016-02-24 06:38 AM

Hi this will do what you want:

 

In the LUA Parser replace the existing function split with this new function:

function split(mystring,metakey)

  local pattern = ';'

  local maxsplit = -1

  local s = mystring

  local t = {}

  local patsz = #pattern

  local commaexists=nil

  local loopcount=0

 

 

  --print("LUA DEBUG:split: " .. s)

  while maxsplit ~= 0 do

    local curpos = 1

    local found = string.find(s, pattern)

 

 

    if found == nil and loopcount==0 then

      t[mystring]=1

    end

 

 

    if found ~= nil then

  commaexists=true

      --table.insert(t, string.sub(s, curpos, found - 1))

  t[string.sub(s, curpos, found - 1)]=1

      curpos = found + patsz

      s = string.sub(s, curpos)

    else

      -- Comma not found in string so stop

  --table.insert(t, string.sub(s, curpos))

  if commaexists then

  t[string.sub(s, curpos)]=1

  end

      break

    end

    loopcount=loopcount+1

    maxsplit = maxsplit - 1

    if maxsplit == 0 then

      --table.insert(t, string.sub(s, curpos - patsz - 1))

  t[string.sub(s, curpos - patsz - 1)]=1

    end

  end

  for k,_ in pairs(t) do

    nw.createMeta(metakey, k)

  end

end

DavidWaugh1
Employee
Employee

‎2016-02-24 06:43 AM

A good way of trying out LUA is the Lua: demo webpage.

 

This is the function that I pasted into the test at the top:

function split(mystring,metakey)

  local pattern = ';'

  local maxsplit = -1

  local s = mystring

  local t = {}

  local patsz = #pattern

  local commaexists=nil

  local loopcount=0

 

 

  print("LUA DEBUG:split: " .. s)

  while maxsplit ~= 0 do

    local curpos = 1

    local found = string.find(s, pattern)

 

 

    if found == nil and loopcount==0 then

       t[mystring]=1

    end

 

 

    if found ~= nil then

   commaexists=true

      --table.insert(t, string.sub(s, curpos, found - 1))

   t[string.sub(s, curpos, found - 1)]=1

      curpos = found + patsz

      s = string.sub(s, curpos)

    else

      -- Comma not found in string so stop

   --table.insert(t, string.sub(s, curpos))

   if commaexists then

  t[string.sub(s, curpos)]=1

   end

      break

    end

    loopcount=loopcount+1

    maxsplit = maxsplit - 1

    if maxsplit == 0 then

      --table.insert(t, string.sub(s, curpos - patsz - 1))

   t[string.sub(s, curpos - patsz - 1)]=1

    end

  end

  for k,_ in pairs(t) do

    print(metakey ..":".. k)

  end

end

 

 

split("abc1","user.dst")

 

In the output box I then got:

 

LUA DEBUG:split: abc1

user.dst:abc1

AlexeyFedorov
Frequent Contributor
Frequent Contributor

‎2016-02-24 06:58 AM

Thanks you so much. I will test it in my environment (I suppose tomorrow) and then I post here full your parser adapting to split user.dst for MS Exchange.

0 Likes

DavidWaugh1
Employee
Employee

‎2016-02-24 07:31 AM

Hi,

You should not set user.dst to transient.

If you look at your logs your will see the following

 

Feb 24 12:29:49 LOGDECCOL1 NwLogDecoder[38147]: [Parse] [warning] Maximum meta callback depth reached

Feb 24 12:29:49 LOGDECCOL1 NwLogDecoder[38147]: [Parse] [warning] Maximum meta callback depth reached

Feb 24 12:29:49 LOGDECCOL1 NwLogDecoder[38147]: [Parse] [warning] Maximum meta callback depth reached

Feb 24 12:29:49 LOGDECCOL1 NwLogDecoder[38147]: [Parse] [warning] Maximum meta callback depth reached

Feb 24 12:29:49 LOGDECCOL1 NwLogDecoder[38147]: [Parse] [warning] Maximum meta callback depth reached

 

This means an infinite loop is being set up.

 

What you need to do is change the log XML parser so that it writes user.dst to another key and then set this key to transient and use this as the input key on your LUA parser.

 

The following script is a quick way to rename one key in a parser to another:

 

more ~/replace-keys.sh

#1 first key

#2 second key

#3 parser name

cp $3 $3.backup.$(date -d "today" +"%Y%m%d%H%M")

sed -i "s/\&lt\;$1\&gt\;/\&lt\;$2\&gt\;/g" $3

 

The following example shows me replacing the fld1 metakey with rn in the windows parser

 

~/replace-keys.sh fld1 rn v20_winevent_nicmsg.xml

 

Once you have modified a parser make sure that you

  1. Unsubscribe from Live
  2. Remove the parser from the Deployment group

Otherwise it will be overwritten.

 

Another lazy way of doing this is to set the immutable flag on the parser. (eg chattr +i v20_winevent_nicmsg.xml)

0 Likes
© 2022 RSA Security LLC or its affiliates. All rights reserved.