Using Regular Expressions within Investigator

[RSAAdmin]

Hello,

I recently blogged about my experiences around detecting domains in alias.host field created by Domain Generating algorithms within investigator by using some regex.  I'm not sure if the community allows link outs but I have screenshots and such there that would be a bit cumbersome to recreate here.  Also I wanted to get the word out about the home version of Investigator for people toying with info sec at home to the 3 or so people that stumble upon my blog a month.  

I'm interested in any more information around the syntax for regex within investigator. I view it as powerful as the regex abilities of an IDS signature, which I guess informer alerts would qualify as.

https://scottfromsecurity.com/blog/2012/10/28/rsa-netwitness-investigator-regular-expressions

If there is an issue around posting links, just let me know.  I didn't really see a "community rules" sticky.

[RSAAdmin]

Regex content searching or queries within Investigator follows the Perl regex conventions.

In addition, when using it as part of a query, you should place quotes around your expression and escape any quotes in your expression using a backslash.

[RSAAdmin]

Thanks Scott,

  Do you know of any documentation around use of Regex within Investigator? 

It was at Netwitness Analyst I and II training I attended in Reston, VA where I was told by the instructor that Investigator uses BOOST regex engine as opposed to perl's interpreter, so I was just going by that.  Looking at the NW Investigator User Guide 9.7,  I see the below..

NetWitness uses the Boost Perl regular expression engine. All regular expressions must be formatted in the

appropriate syntax. More information about the Boost Perl regular expression library and syntax can be found at

the Boost Homepage (pg 85)

I'm not familiar enough with the difference to know if it even really matters for the non-complicated Regex search strings, but any clarification you can provide is welcome. I've just been using trial and error to until the regex is accepted by the custom drill at this point as well as using this link for syntax reference.

http://www.boost.org/doc/libs/1_34_0/libs/regex/doc/syntax_perl.html

[RSAAdmin]

AFAIK, it's completely compatible with perl syntax, but I admit I'm not a perl regex expert by any stretch.  The current version of boost we use is 1.48, so this link would be better:

http://www.boost.org/doc/libs/1_48_0/libs/regex/doc/html/boost_regex/syntax/perl_syntax.html

Scott

[RSAAdmin]

Scott,

Is there a way to export regular expression patterns, so other users in my organization can easily import them to their worksations and start using them? 

[RSAAdmin]

Unfortunately, there isn't a way via the UI to import new searches.  However, you can copy your expressions out of the settings file under your profile and paste them into another.  The Investigator settings file is an XML file and can be found under your profile Roaming\NetWitness\NwInvestigator9.settings

Copy the full <Queries>...</Queries> node into another user's settings file.  Make sure Investigator is not running before editing the file.

HTH,

Scott

[Anonymous]

This syntax does not appear to work when I put a comma between {}, I get the following error:

This is what I used to get this this error:

query regex \/[0-9a-zA-Z]{7,}\/\?[0-9a-zA-Z]{5,};

The below query patterns I am trying to match:

Running my regex through an online regex tester this works fine.  I have checked my syntax against Perl, Boost Perl, Ruby, and python.  All of them accept this command; why does investigator fail to understand this?

[RSAAdmin]

You need to quote your value.  Quoting strings is mandatory within the query language.  If you use a quote within the regex, you will need to escape it \'.

Also, I recommend upgrading your Investigator client to 9.8.5.19, which has a fix to prevent mangling the regex when it's submitted, assuming it's properly quoted.

[Anonymous]

So the above Regex should look like:

query regex "\/[0-9a-zA-Z]{7,}\/\?[0-9a-zA-Z]{5,};"

Even if I use quotes, I will need to upgrade to 9.8.5.19 for the quotes to even work?

[RSAAdmin]

Yes, you need to quote it, just like that.

It might or might not work with a version earlier than 9.8.5.19.  In particular, it didn't like parenthesis before 9.8.5.19 and possibly a few other characters.

9.8.5.19 should resolve any regex issues you have though.

If you want to make sure the expression you sent is received on the concentrator/broker correctly, you will need to look at the audit log on the service to make sure the query is formatted correctly.  If it isn't, then you know Investigator mangled it.