ESA - correlating 3 events into 1 alert with input from one of the events?

[VladimirPrevin]

esa question - 2 events by reference id in 3 mins (1 log type) (one of these has a checksum), then by checksum from this in another type.

hello, 

ESA question -

•    ( log type 1) have 2 events - 2 events for this log device with different contents. both arriving within say 0-10s (one of these has a checksum,order of arrival of 2 events for log type 1 is not fixed when they arrive within 0s), want to match these by checksum to second log type.
•    ( log type 2) arriving up to 30m later. (also has the checksum meta

want to:

•  produce an ESA alert with all 3 events
• match the first two events (log type 1) by reference.id, and (keeping in mind the order of arrival for log type type 1 is not fixed)
• the second (log type 2)by the checksum from one of the first events (definitely later than the two say 10-40m)
• ideally: check some of the meta in the 3 events to decide if to alert at all.   (both of these type are low volume - say 10k messages/day)(extra bonus points for checking the array of strings meta for count of those or specific ones) 

Doable with Rule builder or EPL only? issues with time windows (two time windows,the second one a bit long)? issues with looking up the second event type by checksum from first?

Can anyone help with the syntax (use abstract device names and meta key names dt1,dt2, dt1e1,dt2e2,dt2e3, mkStr1

bit more info:

device type 1 (dt1):

 - dt1nm - network meta messages 

 - dt1fm - file meta messages (have some checksum meta , and let's some and some numeric and string meta keys)

 - linked by reference_id

 - these are emmited and arrive within 30s (actually usually within the same second if not max 5 i suppose) 

 - unfortunately the order these arrive is is not fixed when within the same second [sometimes event 0 seems to be dt1nm, sometimes dt1fm)

 - ideally want to factor in some of the meta (want to if on a few attributes say ip_src \ ip_dst and some of the numeric and string+ string array meta keys)

device type 2 (dt2)

 - dt2fm - file meta and the also the checksum meta but also some a few number, array of strings and string meta keys. 

ps already registered for the 3 RSA university free EPL/ESA modules (but not sure if the syntax changed for 10.6.2+)

but thought I'd ask if this is possible via the UI and or any issues with the time window? or any other caveats

https://community.rsa.com/community/training/netwitness
https://community.rsa.com/videos/26295

[DanBremer]

I use this to link two events together/ Both events must occur within 10 seconds.

The SELECT statement limits the events to look at, and provides the time window...

The 'MATCH RECOGNIZE' defines the match between the events.

You check both ways (the 'PATTERN' part of the statement), because the decoders and esa's are multi-threaded, so the second event can arrive before the first because it gets processed faster (if the events are really close together time-wise)...

These events are linked together by connection_id, operation_id, obj_val ('PARTITION BY' part of the statement). Both events must match all three of these fields in order to be processed.

Adding a third event would be just another statement defining what to look for, and another 'E#'...

------------------------------------------------------------------------------------------------

module Module_5577352545ce2d8be05700f8;

@Name('Module_5577352545ce2d8be05700f8_Alert')
@Description('Alert-OracleED')
@RSAAlert(oneInSeconds=0)

SELECT * FROM Event(
/* Statement: Bind */
(device_type IN ( 'oracleed' ) AND ( 'BIND' = ANY( action ) ) AND user_dept IN ( 'department' ))
OR
/* Statement: Result3 */
(device_type IN ( 'oracleed' ) AND ( 'RESULT1' = ANY( action ) ) AND result IN ( '0' ) AND user_dept IN ( 'department' ))

).win:time(10 Seconds)
MATCH_RECOGNIZE (
PARTITION BY connection_id, operation_id, obj_val
MEASURES E1 as e1_data , E2 as e2_data
PATTERN (E1+ E2 | E2 + E1)
DEFINE
E1 as (E1.device_type IN ( 'oracleed' ) AND ( 'BIND' = ANY( E1.action ) ) AND E1.user_dept IN ( 'department' )),
E2 as (E2.device_type IN ( 'oracleed' ) AND ( 'RESULT1' = ANY( E2.action ) ) AND E2.result IN ( '0' ) AND user_dept IN ( 'department' ))
);

[VladimirPrevin]

thanks Dan

>connection_id, operation_id, obj_val
>Both events must match all three of these fields in order to be processed.

unfortunately It doesn't seem that I can easily extend your example by adding another property in to the select event bit and match recognize bit  and extending the time window .

if I understood correctly, in your example all 3 events are linked by the same 3 properties that must match. What I'm trying to is slightly different. 2 sets of events linked by different keys and with different time windows. 

e1 - dt1nm
e2 - dt1fm
connected by reference.id
(small time window)

e3 - dt2fm is connected to dt1fm by checksum
(large time window from dt1 events - up to 40m)

thoughts?

p.s the statement to link events e1 and e2 (dt1nm,dt1fm) looks fairly similar and works ok for us. it's just adding the extra linkage and time window...

[DanBremer]

That sounds alot like this example:

A create user, login as that user, and delete user in a 300 second window...

EPL #3:

https://sadocs.emc.com/0_en-us/090_10.4_User_Guide/50_Alert/40_DefineESARules/20_DefineAdvESARule/SampleAdvESARule

SELECT * FROM Event(ec_subject='User'
and ec_activity in ('Create','Logon','Delete')
and ec_theme in ('UserGroup', 'Authentication')
and ec_outcome='Success'
).win:time(300 seconds)
match_recognize (measures C as c, L as l, D as d
pattern (C L D)
define
C as C.ec_activity = 'Create',
L as L.ec_activity = 'Logon' AND L.user_dst = C.user_src,
D as D.ec_activity = 'Delete' AND D.user_src = C.user_src
);

[VladimirPrevin]

thanks Dan. probably worth consulting the 10.6 not 10.4 rules(thought the Esper verion/syntax changed a bit?) but the two examples you gave should be combinable into what we need Sample Advanced EPL Rules - RSA Security Analytics Documentation  

[VladimirPrevin]

neither of these seem to work ... 

both are syntactically correct apparently but first one gets disabled on deployment, and second one matches nothing. tried prefixing things with metaname is not null AND - doesn't make a difference. not the faintest idea whether any of it is correct or what the issue is . 

@Name('Module_4309c840_55ee_451e_bb9f_e8a334ad7a91_Alert')
@Description('')
@RSAAlert(oneInSeconds=0)

SELECT * FROM Event(
/* Statement: f */
(device_type.toLowerCase() IN ( 'dt1' ) AND event_desc.toLowerCase() IN ( 'filesubtype' ))
OR
/* Statement: n */
(device_type.toLowerCase() IN ( 'dt1' ) AND event_type.toLowerCase() IN ( 'networksubtype' ))
OR
/* Statement: s */
(device_type.toLowerCase() IN ( 'dt2' ) AND device_ip.toLowerCase() IN ( 'a.n.i.p' ))

).win:time(60 Minutes)
MATCH_RECOGNIZE (

MEASURES E1 as e1_data , E2 as e2_data , E3 as e3_data
PATTERN (E1*E2 + E2*E1 + E3) /* (basically we want E1 then E2 then E3, or E2 E1 E3) I don't know if it means this */
DEFINE
E1 as (E1.device_type .toLowerCase() IN ( 'dt1' ) AND E1.event_desc .toLowerCase() IN ( 'filesubtype' )),
E2 as (E2.device_type .toLowerCase() IN ( 'dt1' ) AND E2.event_type .toLowerCase() IN ( 'networksubtype' ) AND E1.reference_id=E2.reference_id),
E3 as (E3.device_type .toLowerCase() IN ( 'dt2' ) AND E3.device_ip .toLowerCase() IN ( 'a.n.i.p' ) AND E3.checksum=E1.checksum)
);

@Name('Module_4309c840_55ee_451e_bb9f_e8a334ad7a91_Alert')
@Description('')
@RSAAlert(oneInSeconds=0)

SELECT * FROM Event(
/* Statement: f */
(device_type.toLowerCase() IN ( 'dt1' ) AND event_desc.toLowerCase() IN ( 'filesubtype' ))
OR
/* Statement: n */
(device_type.toLowerCase() IN ( 'dt1' ) AND event_type.toLowerCase() IN ( 'networksubtype' ))
OR
/* Statement: s */
(device_type.toLowerCase() IN ( 'dt2' ) AND device_ip.toLowerCase() IN ( 'a.n.i.p' ))

).win:time(60 Minutes)
MATCH_RECOGNIZE (

MEASURES E1 as e1_data , E2 as e2_data , E3 as e3_data
PATTERN (E1 E2 E3 | E2 E1 E3 ) /* (basically we want E1 then E2 then E3, or E2 E1 E3) I don't know if it means this */
DEFINE
E1 as (E1.device_type.toLowerCase() IN ( 'dt1' ) AND E1.event_desc.toLowerCase() IN ( 'filesubtype' )),
E2 as (E2.device_type.toLowerCase() IN ( 'dt1' ) AND E2.event_type.toLowerCase() IN ( 'networksubtype' ) AND E1.reference_id=E2.reference_id),
E3 as (E3.device_type.toLowerCase() IN ( 'dt2' ) AND E3.device_ip.toLowerCase() IN ( 'a.n.i.p' ) AND E3.checksum=E1.checksum)
);