RSA NetWitness - ESA Filtering

[NaushadKasu1]

Notes below from the video presentation...

Traditionally, ESA aggregation has been an "all or nothing" approach. That is, we had no way to provide ESA just the necessary events that are relevant to the ESA Rules deployed on that ESA service. RSA NetWitness 11.3+ brings us a new feature that allows us to tag data necessary for ESA on Log  & Network (Packet) Decoders and configure the ESA to aggregate ONLY these sessions. 

Let's take a question & answer approach to understanding how ESA filtering is setup… 

Why would we use ESA filtering? 
 

Due to the sheer volume and advanced logic in place on ESA, at times ESA gets behind and does not alert in a real-time manner. This is because traditionally as mentioned above, ESA is capable of aggregating 100-125K EPS (Events Per Second). To mitigate this load and allow for the overall health and scaling of the ESA service, we set filters, giving it an opportunity to work real-time and continue to remain healthy and reliable. 

How is ESA filtering implemented? 
 

Application Rules are set to write 'esa_<log|network>" to the 'customer.esa' meta key (reference image below). Note that on all Log Decoders, we have 'esa_log' written to the 'customer.esa' meta key with the conditions necessary to tag relevant data. To minimize management overhead, we can allow many device.type(s) that are low volume and do not need granular filtering allowed in the top-most rule. Further below in the Application Rules chain, we tag larger volume datasets e.g. ciscoiportwsa, and we set conditions to granular tag traffic with the respective 'esa_log' tag.  

How does ESA know to only pull those events tagged into 'customer.esa' meta key? 

In ESA Correlation Service -> View -> Explore -> correlation -> stream, there is a 'filter' field for which we add NWDB-syntax to instruct ESA to pull specific events. For ESA, we have a filter as shown below e.g. select * where customer.esa = 'esa_log' 

How do I validate ESA is getting data related to my ESA rule? 

 
To validate your ESA rule will see the proper set of data, you can put the conditions into Investigate view and look for the 'customer.esa' key. If it contains 'esa_log', then those event(s) will make their way to ESA.  

Let's go through an example: 

1. Analyst wants to write an ESA Rule as follows: 
   1. SELECT * FROM Event( 
       device_type = 'ciscoiportwsa' 
2.     AND result_code = 'games' 
   ); 
    
3. Go into Investigate and query the data set using NWDB syntax 
   1. Query changed to NWDB-syntax as shown below with a timeframe of 24 hours (arbitrary time chosen just to capture more events in the event we have not seen those specific events we care about in a shorter-time). That is, if you query too short of a timeframe, the data may in fact be configured to tagged by Application Rules but the SIEM has not events in your timeframe. Thus, 24 hours is a good timeframe to start with. 
       
   device.type = 'ciscoiportwsa' && result.code = 'games' 

1. If 'customer.esa' meta key has 'esa_log' values, then ESA will be getting data. If that is not the case, update the Application Rule set to accommodate your requirement.  

1. See below from our example that 'customer.esa' key does in fact have 'esa_log' value. Therefore, I can conclude my new or modified ESA rule will see the data required for my condition to trigger an alert. 
    

[drewjc]

Great video Naushad! Thanks for posting as I wasn't aware of this pre-filtering optimization. Do you find customers/clients who implement this filtering add another ESA server to their environment so they can also maintain one that gets the full event stream?

[NaushadKasu1]

I haven't worked with customers doing this today, however, it is definitely a viable approach. That is, using one ESA with targeted data aggregation (filtered data) with advanced analytical use cases deployed. Then using another that receives all data with standard rule set.