ESA alert is flooding when using sliding time window

[XavierFerrier1]

Hello Community,

I have the following ESA rule to detect portscan :

@RSAAlert(oneInSeconds=0)

SELECT * FROM Event

(

device_type='snort'

AND ip_dstport=137

)

.std:groupwin(ip_src)

.std:unique(ip_dst)

.win:time(60 sec)

GROUP BY ip_src HAVING count(ip_dst) > 3;

But when I do a test (a portscan, from the same source, on 4 different IP and on the port 137) , I have lots of ESA alerts : something like 10 alerts. These alerts match the same ip_src and the same ip_dst

However, if I choose to use a batch time windows (win:time_batch(60 sec) , I'm not flooded by ESA alerts. I have only one alert which corresponds to my test.

Do you know if this is normal ? And would you be aware of any way to avoid this flood when using sliding time windows ?

I would prefer use sliding time windows to avoid to miss some alerts if events trigger between 2 batch time windows...

Thanks for your help ! 🙂

[DavidWaugh1 · Accepted solution]

Hi this is normal. I would recommend putting some suppression on your rule to limit the amount of alerts it will generate. For example

https://rsaportal.force.com/customer/articles/How_To/Example-Advanced-ESA-Rule-which-shows-suppression-on-multiple-variables-in-RSA-Security-Analytics

Create an Advanced Event Stream Analysis rule with the following content:

module Module_564eeceae4b06807f34ebd2d; 
@RSAPersist @Name('Module_564eeceae4b06807f34ebd2d_Alert') 
@Description('') @RSAAlert(oneInSeconds=0)  
SELECT  ip_src, ip_dstport, device_type, ip_dst FROM Event( 
/* Statement: CheckPoint */ (device_type IN ( 'checkpointfw1' ) 
AND ip_src is not null AND ip_dstport is not null AND ip_dst is not null)  )
.std:groupwin(ip_src,ip_dstport)
.win:time(3600 seconds)
.std:firstunique(ip_src,ip_dstport) retain-intersection

The rule does the following:

1. Group events into distinct ip_src and ip_dstport combinations.
2. Has a sliding window of 1 hour.
3. Only sends the first unique ip_src and ip_dst port combination.

The retain-intersection command ensures that only events matching ALL three of these criteria are forwarded.

[XavierFerrier1]

Hello David,

Thanks for this feedback

I tried your solution and it looks like ok. I'm still doing some tests.

You say that this is a normal behaviour, could you explain me why?

Thanks !

[DavidWaugh1]

Lets take a look at your original rule and play some events through it:

@RSAAlert(oneInSeconds=0)

SELECT * FROM Event

(

device_type='snort'

AND ip_dstport=137

)

.std:groupwin(ip_src)

.std:unique(ip_dst)

.win:time(60 sec)

GROUP BY ip_src HAVING count(ip_dst) > 3;

Okay,

lets us have (time in seconds)

time=1 event1={device.type=snort,ip.dstport=137,ip.src=1.2.3.4,ip.dst=2.3.4.5}

time=2 event2={device.type=snort,ip.dstport=137,ip.src=1.2.3.4, ip.dst=2.3.4.5}

time =3 event3={device.type=snort,ip.dstport=137,ip.src=1.2.3.4, ip.dst=2.3.4.5}

so in general we at time n seconds, we get eventn.

(for simplicity assume one every second)

For a sliding window,

we get an alert at time =4,5,6,7,8 ...and every second onwards

For a batch window of size 60  seconds we get an alert at time =4,5,6 .. 60

Then at time 61, the batch window will get destroyed and all events dropped, so in effect we start again.

so will get alerts at 64,64,66....up to 120 and the process repeats.

Hope that clears it up.

[XavierFerrier1]

Yes perfect thanks for your explanation !

[NikolayKlender]

Hi!

First of all I should note that this rule may cause high memory usage, beacuse if you have lots of ip_src (for example you have snort which listen internet trafic) than .std:groupwin(ip_src) will hold all this information and after some time will cauase out of memory (please look at Chapter 13. EPL Reference: Views   unfortunately RSA use such aproach in default rules without specail hint usage like this:

@Hint('reclaim_group_aged=10,reclaim_group_freq=30')

If you want alert supression use "output first every 1 hour" instruction. So please try following rule

@RSAAlert

SELECT window(*) FROM Event

(

device_type='snort'

AND ip_dstport=137

).win:time(60 sec)

GROUP BY ip_src

HAVING count(ip_dst) > 3

output first every 30 min;

Also I advice you to debug Esper rules with help of EsperTech Esper EPL Online

(do not paste @RSAAlert annotation becuase it is made by RSA and esper knows nothing about it)

[NikolayKlender]

David your rule may cause out of memory!

Beacuse events from Event stream will go to .std:groupwin(ip_src,ip_dstport) so all coombination of ip_src,ip_dstport will be in this group window!

In rules where std:groupwin is combined with time window on high amount of distinct values by wich events are grouped (ip_src,ip_dstport in your example) special hind must be used!

Chapter 13. EPL Reference: Views ​

The @Hint("reclaim_group_aged=age_in_seconds") hint instructs the engine to discard grouped data window state that has not been updated for age_in_seconds seconds. The optional @Hint("reclaim_group_freq=sweep_frequency_in_seconds") can be specified in addition to control the frequency at which the engine sweeps data window state. If the hint is not specified, the frequency defaults to the same value as age_in_seconds. Use the hints when your group criteria returns a changing or unlimited number of values. By default and without hints the view does not reclaim or remove data windows for group criteria values.

[XavierFerrier1]

David Waugh I made more tests with retain syntax and I had flood again...

Nikolay Klender​ Thanks for your comment !  I found the same solution using the pdf ESA_Rule_Development available on the RSA Knowledge Base https://rsaportal.force.com/customer/servlet/fileField?retURL=%2Fcustomer%2Farticles%2FHow_To%2FHelpful-information-for-developing-ESA-rules-in-RSA-Security-Analytics%3Fpopup%3Dfalse%26navBack%3DH4sIAAAAAAAAAIuuVipWslLyzssvz0lNSU_1yM9NVdJRygaKFSSmp4Z...

Anyway thanks for the advices ! 🙂

@RSA Link Moderator : please change Corret Answer flag to Nikolay comment

[XavierFerrier1]

Nikolay Klender​ : Does the argument select window(*) is mandatory ? What is the differrence with select * ?

If I don't use a HAVING  COUNT in the alert, is it usefull ?

[NikolayKlender]

window(*) will select all events which are in sliding window win:time(60 sec) so in alerts page you will see all events corresponding to alerts. Please made two types of rules with * and window(*) and you will see difference.

having - is operator which can be used to filter aggregated functions results, because you cannot use something like "where count(ip_dst)>3". I think it is better to google some sql examples to understood "having" usage because esper epl is very similar to sql.