Trend Analysis with the Netwitness Suite

[DavideVeneziano]

The Netwitness Suite provides out-of-the-box a number of tools to analyze your data. But there is a capability hidden under the hood which if implemented correctly may be precious to identify additional suspicious patterns: the development of a baseline to perform a trend analysis.

This approach can help whenever a significant change in the rate  of a given value could imply a security issue. Of course not all the threats can be identified in this way!

To perform any statistical analysis, numbers are an obvious requirement and these have to be derived from the collected events first. The attached (unofficial) model, inspired by the new 10.6 Event source Automatic Monitoring functionality, offers a solid way to count the number of occurrences without requiring to buffer all the events in memory for a long timeframe. 

For each value of a given meta key, the number of occurrences are counted every minute and then aggregated every five minutes, hour and day to minimize the impact on ESA performance. Then, for each hour (and for each day), a baseline is created.

In case there is a significant deviation in the rate of any meta value, an alert is generated.

The duration of the learning phase, the entity of the deviation and the duration of the baseline are all configurable parameters. 

As an implementation best practice, do not use meta keys with too many unique values (e.g. ip.src) since would generate too many false positives. Start focusing on those with a few but significant unique values, like as:

• Browsers - uncommon client may be associated with malicious codes
• Country source/destination - can help identifying attacks or potential data exfiltration
• TLDs - uncommon TLDs can be an indicator of something strange happening

All the details regarding the model, how it works, how to implement it and all the technical details can be found in the attached presentation together with the full EPL code.

For a different but complementary approach, I'd suggest reading this excellent post by Nikolay Klender‌: https://community.rsa.com/thread/187264 

Please note this is not RSA official/supported content so use it at your own risk!

ESA_baseline_model_v2.1.txt.zip

ESA_baseline_model_v2.2.txt.zip

[DavideVeneziano]

Hi, for those asking how this model can be used to generate and apply a different baseline between working days and the weekend, I'm attaching to the post the file ESA_baseline_model_v2.2.txt. 

Since having the same baseline model for multiple days is not that easy to implement in EPL, we are creating a model for each day of the week (e.g. comparing Monday 8am-9am timeframe with the baseline created across previous mondays during the same hour).

[MihaMesojedec]

Davide have you consider using Memory Pool in ESA for baseline use case?

https://community.rsa.com/docs/DOC-54932 

Thanks,

Miha

[DavideVeneziano]

Good point Miha. From my understanding and for this specific use case the Memory Pool would not completely solve the issue of the large timeframe and may not perform in an ideal way. For sure a combination between the multi-phase approached I've used in the rule and Memory Pool could be powerful.

Thanks