ESA rule broken at 11.3

[BohdanR]

We recently upgraded from NetWitness 10.6.6 to 11.3. Several rules got disabled during the upgrade and they no longer work.
I suppose it is mainly because directory meta changed type from string to string[], so that it became an array.

Simplified version of the rule, which compares directory in the event with directory in enrichment and it fires alert if there is match on the directory (or its subdirectory, that is why there is .startsWith):

@RSAAlert(oneInSeconds=0)
@UsesEnrichment(name='folders_test')
SELECT * FROM Event (
(device_type IN ( 'emcisilon' ) AND directory IS NOT NULL AND
EXISTS (SELECT * FROM folders_test WHERE(Event.directory.toLowerCase().startsWith(directory)) )
) );

Enrichment RSI1_folders_test is in-memory table and contains several folder paths, named as directory of type string.

The rule worked fine at Netwitness 10.6.6.
The rule is no longer possible to be deployed to ESA with 11.3, because directory doesn't have method toLowerCase now (it is not a string anymore).

I tried converting directory to string using cast(Event.directory,string), which works when it is directly in event filters (except there is [ character at start and ] at end of the string), but it doesn't seem to work properly when used together with the enrichment.

The bellow rule fires alert on all events of type emcisilon, no matter if startsWith on the directory matches or not (EXISTS part seems to result always to TRUE for some unclear reason):

@RSAAlert(oneInSeconds=0)
@UsesEnrichment(name='folders_test')

SELECT * FROM Event (
(device_type IN ( 'emcisilon' ) AND directory IS NOT NULL AND
EXISTS (SELECT * FROM folders_test WHERE( cast(Event.directory,string).toLowerCase().startsWith(directory,1) ) )
) ) ;

I tried various things, but didn't manage to get this rule working with 11.3.

Can you please advice how to fix the rule to work in Netwitness 11.3 ?

[LeeKirkpatrick]

Hey Bohdan,

Dealing with vectors can be a little complicated within the ESA, but the change from string to string[] is definitely a good idea as a single session may have multiple directory metadata registered, and without this being a string[], you would not see all directory metadata. What I would recommend in this instance is to cast the directory vector to a string, split the directory metadata into individual elements and return as an array while removing the unwanted characters ([]), and iterate through the directory metadata, checking to see if they contain a value in your window:

SELECT * FROM Event(
 cast(directory, string).split(', |\\[|\\]', 0).anyOf(i => i.toLowerCase() LIKE '%' || (SELECT directory FROM folders_test) || '%' ))‍‍‍‍‍‍

NOTE: I haven't had a chance to test this so there may be potential tweaking required.

Cheers,

Lee

[BohdanR]

Hi Lee,

Thank you very much! It seems this works 🙂

Wouldn't it be more effective to use .toArray() method instead of splitting the string by regexp? I tried it now and it seems to work too:

directory.toArray().anyOf(i => cast(i,string).toLowerCase() LIKE  (SELECT directory FROM folders_test) || '%' )‍‍‍

[LeeKirkpatrick]

Awesome! I hadn't had a chance to test the .toArray() but thank you for letting me know that it works as well, I would agree that is a far more elegant solution.

Cheers,

Lee

[BohdanR]

Would you be able to convert this query to iterations too?

EXISTS (SELECT * FROM folders_test WHERE ( Event.directory.toLowerCase().startsWith(directory)  AND  isNotOneOfIgnoreCase(Event.username,{ username }) ) )‍

In this case, enrichment has 2 columns - directory and username. It means that only specified user (owner) can access that directory. I am not sure how to extend the query you suggested to achieve that.

[LeeKirkpatrick]

I would suggest trying something like the below. 

SELECT * FROM Event.win:length(1), folders_test
WHERE
cast(Event.directory, string).split(', |\\[|\\]', 0).anyOf(i => i.toLowerCase() LIKE folders_test.directory || '%') 
AND 
cast(Event.username, string).split(', |\\[|\\]', 0).anyOf(i => i.toLowerCase() != folders_test.username);‍‍‍‍‍

Cheers,

Lee

[BohdanR]

Hi Lee, 

Thank you, I will try that.
Why there is that ".win:length(1)" behind Event? Is it creating a sliding window for one element? Does it need to be there?

[LeeKirkpatrick]

The reason that a data window (in this case .win:length(1)) must be declared, is that a data window specifies which events are considered for the join (i.e. last event, last 10 events, all events, last 1 second of events etc.). It is a requirement from Esper for unbounded streams, which the Event stream is - otherwise it would not know how much data to use for the join. So we are just saying, compare the last event received from the Event stream against our window each time an event arrives that matches our condition.

[BohdanR]

Hmm, I tried it like that, but it fails to deploy. I get the following message in the log:

Failed to validate filter expression 'device_type="emcisilon" and event_t...(244 chars)': Property named 'device_type' is not valid in any stream

It seems that event properties (metas) are not accessible if Event is used this way 😞

[LeeKirkpatrick]

Could you paste the full rule here?