I find myself spending more time on regex then using the platform

[RSAAdmin]

Seems I spend more time troubleshooting regexs instead of investigating events....for all the money spent on the tool its pretty primitive and time consuming to narrow the data down to what you want.

I figured simple things like ip.dst != 10.0.0.0/8 would simply work....or src.org !='org name" would work....

How about a better regex guide....I really dont have the time to figure out what works and what doesn't....

Scott,

[Anonymous]

Hi

May I ask whether the likes of the sample below is legal in SA REGEX?

alias.host regex "sam\d\dv\d\d\d\d\d\d","sam\d\dp\d\d\d\d\d","ph\d\dc\d\dn\d\d","ph\d\dc\d\dn\d\d","ph\d\dc\d\dn\d\d"

• samXXv666666 - virtual servers
  (e.g. sam03v000452)
• samXXp55555 - Physical Servers
    (e.g. sam01p000012)
• phXXcYYnZZ - Physical  Server
    (e.g. ph01c02n24)
• phXXcYYnZZ - Physical Servers
  (e.g. ph01c01n16)

Thank you.

[RSAAdmin]

I haven't tested this, but I believe you need to double-escape the backslashes \\.

The parameter passing interface to NextGen services uses the backslash to escape certain characters, so if you double escape them, it should get passed to regex correctly (as a single \).

Let me know if that fixes the issue.

[Anonymous]

Hi Scott,

Thanks for the reply, I did some test and the following sample yielded the one below

(from investigator)

alias.host regex "sam\\d\\dv\\d\\d\\d\\d\\d\\d","sam\\d\\dp\\d\\d\\d\\d\\d","ph\\d\\dc\\d\\dn\\d\\d","de\\d\\dc\\d\\dn\\d\\d"

(when i look at the rest sdk stat queries it got the following)

flags=2305 id2=2484766586040 id1=1336423529586 where="(alias.host regex \"sam\\\\d\\\\dv\\\\d\\\\d\\\\d\\\\d\\\\d\\\\d\",\"sam\\\\d\\\\dp\\\\d\\\\d\\\\d\\\\d\\\\d\",\"ph\\\\d\\\\dc\\\\d\\\\dn\\\\d\\\\d\",\"de\\\\d\\\\dc\\\\d\\\\dn\\\\d\\\\d\") && time=\"2014-04-02 00:00:00\"-\"2014-04-03 00:00:59\"" fieldName=did expiry=3900000 threshold=100000 size=20

I guess it auto adds the extra "\"

[RSAAdmin]

After looking at your use case in closer detail, I think it would serve you better to create a custom feed to tag your server infrastructure better.  I imagine you already have a spreadsheet of some kind that identifies your KNOWN hostnames.  This could serve as the basis of your feed.

Create a new feed called known infrastructure.

Column 1 would be all your hostnames.  It is a non-ip feed. The meta callback would be alias.host for column 1.

Column 2 would be filled top to bottom with the feed.name of Known Infrastructure

Column 3 would be feed.category listed as physical or virtual or desktop, or whatever

Column 4 would be feed.description where you could further label what they do- mail, dns, domain controller, backup server, etc.

Once it is deployed, you should create a rule to look for new hosts that were not identified by your feed.  (History shows this will be a bunch).  That rule would be alias.host begins sam,ph where feed.name != "known infrastructure"

Once you find new hosts, get them added to your feed.  Then wash, rinse, repeat. 

And once you have your infrastructure properly labeled and tracked in SA, you can implement great use cases.  such as:

Are there any unusual UA strings connecting to my mail servers?

Are windows desktops using the correct PDC or are they failing over to the BDC?

Is my VPN pool over-utilizing my virtual servers?

And the list could go on and on.

[RSAAdmin]

So I just ran the following query on a concentrator:

client regex 'mozilla/\d\.\d'

which (on Investigator 9.8) turned into this query in the audit log:

id1=1 id2=318957729 size=20 flags=sessions,sort-total,order-descending threshold=100000 fieldName=client where="time=\"2014-Mar-06 23:23:00\"-\"2014-Mar-07 23:22:59\" && (client regex mozilla/\\\\d\\\\.\\\\d)"

And my results are correct:

id1=2249947 id2=3924797 count=8011 value=mozilla/5.0 type=client flags=0 group=0

id1=2515501 id2=3923474 count=5041 value=mozilla/5.0 (windows nt 6.1; wow64; rv:27.0) gecko/20100101 firefox/27.0 type=client flags=0 group=0

id1=2628520 id2=3922757 count=665 value=mozilla/5.0 (windows nt 6.1; wow64; trident/7.0; rv:11.0) like gecko type=client flags=0 group=0

id1=2250546 id2=3924889 count=347 value=mozilla/4.0 type=client flags=0 group=0

id1=2628152 id2=3924797 count=306 value=mozilla/5.0 (compatible; msie 10.0; windows nt 6.1; trident/7.0) type=client flags=0 group=0

id1=2734419 id2=3921778 count=271 value=mozilla/5.0 (linux; android 4.4.2; nexus 7 build/kot49h) applewebkit/537.36 (khtml, like gecko) version/4.0 chrome/30.0.0.0 safari/537.36 type=client flags=0 group=0

id1=3922309 id2=3924889 count=245 value=mozilla/4.0 (compatible; msie 7.0; windows nt 5.2; trident/4.0; .net clr 1.1.4322; .net clr 2.0.50727; .net clr 3.0.4506.2152; .net clr 3.5.30729) type=client flags=0 group=0

id1=2517374 id2=2618595 count=181 value=mozilla/5.0 (macintosh; intel mac os x 10_9_2) applewebkit/537.36 (khtml, like gecko) chrome/33.0.1750.146 safari/537.36 type=client flags=0 group=0

id1=2720497 id2=2733918 count=87 value=mozilla/5.0 (windows nt 6.1; trident/7.0; rv:11.0) like gecko type=client flags=0 group=0

id1=2517356 id2=2576611 count=64 value=mozilla/5.0 (macintosh; intel mac os x 10.9; rv:24.0) gecko/20100101 thunderbird/24.3.0 lightning/2.6.4 type=client flags=0 group=0

id1=2519422 id2=3922826 count=64 value=mozilla/5.0 (compatible; msie 10.0; windows nt 6.1; wow64; trident/7.0) type=client flags=0 group=0

id1=2550850 id2=3923095 count=64 value=mozilla/4.0 (compatible; msie 7.0; windows nt 6.1; win64; x64; trident/7.0; .net clr 2.0.50727; slcc2; .net clr 3.5.30729; .net clr 3.0.30729; media center pc 6.0; .net4.0c; .net4.0e; infopath.3; tablet pc 2.0; microsoft outlook 14.0.7113; ms-office; msoff type=client flags=0 group=0

id1=2550094 id2=3922932 count=16 value=mozilla/4.0 (compatible; msie 7.0; windows nt 6.1; wow64; trident/7.0; slcc2; .net clr 2.0.50727; .net clr 3.5.30729; .net clr 3.0.30729; media center pc 6.0; .net4.0c; .net4.0e) type=client flags=0 group=0

id1=3922637 id2=3924782 count=10 value=mozilla/5.0 (windows nt 6.1; wow64) applewebkit/537.36 (khtml, like gecko) chrome/33.0.1750.146 safari/537.36 type=client flags=0 group=0

id1=2550052 id2=3922801 count=10 value=mozilla/4.0 (compatible; msie 7.0; windows nt 6.1; win64; x64; trident/7.0; .net clr 2.0.50727; slcc2; .net clr 3.5.30729; .net clr 3.0.30729; media center pc 6.0; .net4.0c; .net4.0e; infopath.3; tablet pc 2.0) type=client flags=0 group=0

id1=2734897 id2=3920385 count=4 value=mozilla/5.0 (windows nt 6.1; win64; x64; trident/7.0; rv:11.0) like gecko type=client flags=0 group=0

id1=3920400 id2=3920403 count=4 value=mozilla/4.0 (compatible; msie 7.0; windows nt 6.1; win64; x64; trident/7.0; .net clr 2.0.50727; slcc2; .net clr 3.5.30729; .net clr 3.0.30729; media center pc 6.0; .net4.0c; .net4.0e; infopath.3; tablet pc 2.0; ms-office; msoffice 14) type=client flags=0 group=0

id1=3923854 id2=3923856 count=2 value=mozilla/5.0 (windows nt 6.1; wow64; rv:24.0) gecko/20100101 thunderbird/24.3.0 lightning/2.6.4 type=client flags=0 group=0

id1=2720496 id2=2720504 count=2 value=mozilla/5.0 (windows nt 6.1; trident/7.0; rv:11.0) like gecko/20100101 firefox/12.0 type=client flags=0 group=0

id1=2628350 id2=2628353 count=2 value=mozilla/4.0 (compatible; msie 7.0; windows nt 6.1; trident/7.0; slcc2; .net clr 2.0.50727; .net clr 3.5.30729; .net clr 3.0.30729; media center pc 6.0; .net4.0c; .net4.0e; bri/2) type=client flags=0 group=0

[Anonymous]

Thank you for that. 

[Anonymous]

Thank you very much for the clarification.

[Anonymous]

We've been seeing a lot of malware lately that uses long hex strings in filenames, directories, and sometimes in POSTs- ie:

GET /c83e2e10c9721fcbdadbefb5184397c7/pace-traditional.php

or

directory = /c83e2e10c9721fcbdadbefb5184397c7/

or

filename = 6d5ce4ac1c678e97a3cb45db2e0e2681.php

So I think that regex is the perfect (and maybe the only) way to search for these strings.

So far, after the better part of 2 days, I've got this regex, which works perfectly in several "regex testers", but does not produce the correct results in SA 10.3.2 

directory regex '([A-Fa-f0-9]{20,})'

or

filename regex '([A-Fa-f0-9]{20,})'

[RSAAdmin]

Try removing the parenthesis, as they have a special meaning in the query language and might be tripping up the results.

Also, keep in mind that directory and filename likely have multiple values per session.  All it takes is a single meta in a session to hit in order to return that session.  This means you will likely see values in the result set which *do not* match your regex, but if you view all the meta for each session, you will see at least one meta value that matches for each session.

[RSAAdmin]

Okay, I've been testing this against a 10.3 Concentrator using Investigator 9.8 and I can definitely say that the regex is working correctly on the Concentrator.

However, Investigator appears to be modifying the regex when parenthesis are in the expression to the point that what you want to submit is not what is received by the Concentrator.  When that happens, the query correctly returns no results.  Investigator is placing backslashes around the parens and this causes the regex to become invalid.

However, when I use the REST python tool to submit this query:

/sdk values

id1=32355915 id2=33228508 size=20 flags=2305 threshold=100000 fieldName=alias.host where="time=\"2014-Apr-03 20:46:00\"-\"2014-Apr-04 20:45:59\" && (alias.host regex '([a-zA-Z0-9]{20,})')"

I get correct results back.  Every session I've received has at least one alias.host with a string of at least 20 alphanumerics in a row.

Sample result:

id1=853016  id2=853236  count=3  format=65  value=comcastresidentialservices.tt.omtrdc.net  type=alias.host  flags=0  group=0

Since the parens are not necessary, if you use Investigator, I would just remove them and the regex should work fine.  At least that particular regex.  I have not tested SA and it may not have this issue.

The python app is here:

You can also just use curl or NwConsole.