I've seen and heard a fair bit of discussion recently about whether it's possible to create custom matchCondition and groupBy fields within the new 11.x Respond Server. "We have the capability within 10.x," the question goes, "but can we do this in 11.x?"
The answer is "Yes," but the process is slightly different, hence the reason for this blog post.
First, I think it will be useful to lay some groundwork and establish a common understanding of the incident creation process within Respond.
When the Respond Server consumes alerts off the message bus, those original, raw alerts can have many different meta fields. The Respond Server needs to create a common schema for these alerts so that it knows where and how to store each piece of incoming data. To do this, the Respond Server relies on a group of scripts to extract, normalize, and group meta.
With this common schema formed, the Respond Server can then begin to aggregate these alerts into incidents. The initial aggregation process relies on matchCondition values within the Incident Rule. For example, the OOTB User Behavior incident aggregation rule:
After aggregating incoming alerts based on these matchCondition values, the Respond Server then attempts to group them into separate Incidents (or suppress them) according to the groupBy values:
The common use case that we will be discussing here is in response to the need for aggregation and grouping using non-default options. For instance, if we want to group incidents according to email subject, or threat description, or any other arbitrary or custom metakey, how to add those so that they appear as options within the UI, AND so that the aggregation and grouping works?
****Before getting into any of the details, I strongly recommend that you try these procedures on a lab or test system first, both to familiarize yourself with the process and to ensure it works, before making any changes to a production system.****
Now then, on to the good stuff.
First thing we need to do is identify the locations and names of the specific files that we will be modifying. This is one area where the process in 11.x is slightly different compared to 10.x, as these files are in a different location.
To modify the available groupBy and matchCondition fields, we need these two files on the Node0 Server (aka Head Server; aka Admin Server):
- /var/netwitness/respond-server/scripts/normalize_alerts.js
- /var/netwitness/respond-server/data/aggregation_rule_schema.json
AND, depending on the source(s) of the alert(s), we will ALSO need to modify (at least) one of the following:
- /var/netwitness/respond-server/scripts/normalize_core_alerts.js
- for alert sources:
- ESA
- Reporting Engine
- NetWitness Investigate
- for alert sources:
- /var/netwitness/respond-server/scripts/normalize_ecat_alerts.js
- for alert source NetWitness Endpoint (aka ECAT)
- /var/netwitness/respond-server/scripts/normalize_ma_alerts.js
- for alert source Malware Analysis
- /var/netwitness/respond-server/scripts/normalize_wtd_alerts.js
- for alert source Web Threat Detection
Once we know the source of the incoming alerts, we will then need to identify the key(s) and/or value(s) within those raw alerts that we want to match and group against. At this point, you will most likely need to examine the raw alert within the Respond Server.
Browse to Respond --> Alerts and select your specific alert. The Raw Alert will be visible in the window on the right (or, if you clicked on the hyperlink Alert Name, it will be on the left in the newly-opened window), allowing you to scroll through the raw data and identify the key or value specific to your use case.
For my test case, I generated alerts from the ESA with different "event_type" values:
...which meant I first needed to modify the "/var/netwitness/respond-server/scripts/normalize_core_alerts.js" file and add the "event_type" key.
Within "normalize_core_alerts.js" there is a "generateEventInfo" function, which is where we can define additional keys to be normalized by the Respond Server, and where I added my "event_type" key.
****NOTE: it is VERY important that you pay close attention to the formatting and syntax within this file when you add a new key, especially where trailing commas are needed/not needed.****
Next I modified the "/var/netwitness/respond-server/scripts/normalize_alerts.js" file and added a new line for my "event_type" key to the "normalizeAlert" function.
****Again, it is very important that you pay close attention to the formatting and syntax when you add your keys to this function block.****
Then I modified the "/var/netwitness/respond-server/data/aggregation_rule_schema.json" file and added a new schema for the "event_type" key.
****And yet again, pay very close attention to formatting and syntax when modifying this file, especially where commas are needed/not needed.****
****I recommend saving copies of each of these modified files, as they get over-written during the upgrade process.****
And finally, restart the Respond Server service, either from within the UI:
...or via command line from the Node0 Server:
# systemctl restart rsa-nw-respond-server
Give it a minute or two for the service to fully restart, refresh your browser, and you can now select your custom matchCondition and groupBy keys from the drop down menus:
...and view the fruits of your labor:
**Bonus note for anyone still reading: if you're like me and it bugs you if something is not in alphabetical order, you can adjust where your custom keys appear within the dropdown menus by inserting your custom schema within the "aggregation_rule_schema.json" file in a different location.
In the example shown above, I added my custom schema to the very end of the file, which is why my key appeared at the very bottom of each dropdown menu.
But if I place my custom schema in alphabetical order within the JSON file, it will appear within the dropdown menu in its new location:
Happy customizing, everybody!
17 Comments
