Traditionally, ESA aggregation has been an "all or nothing" approach. That is, we had no way to provide ESA just the necessary events that are relevant to the ESA Rules deployed on that ESA service. RSA NetWitness 11.3+ brings us a new feature that allows us to tag data necessary for ESA on Log & Network (Packet) Decoders and configure the ESA to aggregate ONLY these sessions.
Let's take a question & answer approach to understanding how ESA filtering is setup…
Why would we use ESA filtering?
Due to the sheer volume and advanced logic in place on ESA, at times ESA gets behind and does not alert in a real-time manner. This is because traditionally as mentioned above, ESA is capable of aggregating 100-125K EPS (Events Per Second). To mitigate this load and allow for the overall health and scaling of the ESA service, we set filters, giving it an opportunity to work real-time and continue to remain healthy and reliable.
How is ESA filtering implemented?
Application Rules are set to write 'esa_<log|network>" to the 'customer.esa' meta key (reference image below). Note that on all Log Decoders, we have 'esa_log' written to the 'customer.esa' meta key with the conditions necessary to tag relevant data. To minimize management overhead, we can allow manydevice.type(s) that are low volume and do not need granular filtering allowed in the top-most rule. Further below in the Application Rules chain, we tag larger volume datasets e.g.ciscoiportwsa, and we set conditions to granular tag traffic with the respective 'esa_log' tag.
How does ESA know to only pull those events tagged into 'customer.esa' meta key?
InESA CorrelationService ->View->Explore->correlation->stream, there is a 'filter' field for which we add NWDB-syntax to instruct ESA to pull specific events. For ESA, we have a filter as shown below e.g.select * wherecustomer.esa= 'esa_log'
How do I validate ESA is getting data related to my ESA rule?
To validate your ESA rule will see the proper set of data, you can put the conditions into Investigate view and look for the 'customer.esa' key. If it contains 'esa_log', then those event(s) will make their way to ESA.
Let's go through an example:
Analyst wants to write an ESA Rule as follows:
SELECT * FROM Event(
ANDresult_code = 'games' );
Go into Investigate and query the data set using NWDB syntax
Query changed to NWDB-syntax as shown below with a timeframe of 24 hours (arbitrary time chosen just to capture more events in the event we have not seen those specific events we care about in a shorter-time). That is, if you query too short of a timeframe, the data may in fact be configured to tagged by Application Rules but the SIEM has not events in your timeframe. Thus, 24 hours is a good timeframe to start with.
If 'customer.esa' meta key has 'esa_log' values, then ESA will be getting data. If that is not the case, update the Application Rule set to accommodate your requirement.
See below from our example that 'customer.esa' key does in fact have 'esa_log' value. Therefore, I can conclude my new or modified ESA rule will see the data required for my condition to trigger an alert.