This website uses cookies. By clicking Accept, you consent to the use of cookies. Click Here to learn more about how we use cookies.
Accept
Reject

NetWitness Community

  • Home
  • Products
    • NetWitness Platform
      • Advisories
      • Documentation
        • Platform Documentation
        • Known Issues
        • Security Fixes
        • Hardware Documentation
        • Threat Content
        • Unified Data Model
        • Videos
      • Downloads
      • Integrations
      • Knowledge Base
    • NetWitness Cloud SIEM
      • Advisories
      • Documentation
      • Knowledge Base
    • NetWitness Detect AI
      • Advisories
      • Documentation
      • Knowledge Base
    • NetWitness Investigator
    • NetWitness Orchestrator
      • Advisories
      • Documentation
      • Knowledge Base
      • Legacy NetWitness Orchestrator
        • Advisories
        • Documentation
  • Community
    • Blog
    • Discussions
    • Events
    • Idea Exchange
  • Support
    • Case Portal
      • Create New Case
      • View My Cases
      • View My Team's Cases
    • Community Support
      • Getting Started
      • News & Announcements
      • Community Support Forum
      • Community Support Articles
    • Product Life Cycle
    • Support Information
    • General Security Advisories
  • Training
    • Blog
    • Certification Program
    • Course Catalog
    • New Product Readiness
    • On-Demand Subscriptions
    • Student Resources
    • Upcoming Events
  • Technology Partners
  • Trust Center
Sign InRegister Now
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
NetWitness Community Blog
Subscribe to the official NetWitness Community blog for information about new product features, industry insights, best practices, and more.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Announcement Banner

Scheduled Maintenance for the Case Portal during May 20-22

View Details
  • NetWitness Community
  • Blog
  • Creating Custom Match Condition and Group By Fields for Respond in 11.x

Creating Custom Match Condition and Group By Fields for Respond in 11.x

JoshRandall
Valued Contributor JoshRandall Valued Contributor
Valued Contributor
Options
  • Subscribe to RSS Feed
  • Mark as New
  • Mark as Read
  • Bookmark
  • Subscribe
  • Email to a Friend
  • Printer Friendly Page
  • Report Inappropriate Content
‎2018-06-21 03:51 PM

I've seen and heard a fair bit of discussion recently about whether it's possible to create custom matchCondition and groupBy fields within the new 11.x Respond Server.  "We have the capability within 10.x," the question goes, "but can we do this in 11.x?"

 

The answer is "Yes," but the process is slightly different, hence the reason for this blog post.

 

First, I think it will be useful to lay some groundwork and establish a common understanding of the incident creation process within Respond.

 

When the Respond Server consumes alerts off the message bus, those original, raw alerts can have many different meta fields.  The Respond Server needs to create a common schema for these alerts so that it knows where and how to store each piece of incoming data.  To do this, the Respond Server relies on a group of scripts to extract, normalize, and group meta.

 

With this common schema formed, the Respond Server can then begin to aggregate these alerts into incidents.  The initial aggregation process relies on matchCondition values within the Incident Rule.  For example, the OOTB User Behavior incident aggregation rule:

 

pastedImage_2.png

After aggregating incoming alerts based on these matchCondition values, the Respond Server then attempts to group them into separate Incidents (or suppress them) according to the groupBy values:

 

pastedImage_4.png

The common use case that we will be discussing here is in response to the need for aggregation and grouping using non-default options.  For instance, if we want to group incidents according to email subject, or threat description, or any other arbitrary or custom metakey, how to add those so that they appear as options within the UI, AND so that the aggregation and grouping works?

 

****Before getting into any of the details, I strongly recommend that you try these procedures on a lab or test system first, both to familiarize yourself with the process and to ensure it works, before making any changes to a production system.****

 

Now then, on to the good stuff.

 

First thing we need to do is identify the locations and names of the specific files that we will be modifying.  This is one area where the process in 11.x is slightly different compared to 10.x, as these files are in a different location.

 

To modify the available groupBy and matchCondition fields, we need these two files on the Node0 Server (aka Head Server; aka Admin Server):

  • /var/netwitness/respond-server/scripts/normalize_alerts.js
  • /var/netwitness/respond-server/data/aggregation_rule_schema.json

 

AND, depending on the source(s) of the alert(s), we will ALSO need to modify (at least) one of the following:

  • /var/netwitness/respond-server/scripts/normalize_core_alerts.js
    • for alert sources:
      • ESA
      • Reporting Engine
      • NetWitness Investigate
  • /var/netwitness/respond-server/scripts/normalize_ecat_alerts.js
    • for alert source NetWitness Endpoint (aka ECAT)
  • /var/netwitness/respond-server/scripts/normalize_ma_alerts.js
    • for alert source Malware Analysis
  • /var/netwitness/respond-server/scripts/normalize_wtd_alerts.js
    • for alert source Web Threat Detection

 

Once we know the source of the incoming alerts, we will then need to identify the key(s) and/or value(s) within those raw alerts that we want to match and group against.  At this point, you will most likely need to examine the raw alert within the Respond Server.

 

Browse to Respond --> Alerts and select your specific alert.  The Raw Alert will be visible in the window on the right (or, if you clicked on the hyperlink Alert Name, it will be on the left in the newly-opened window), allowing you to scroll through the raw data and identify the key or value specific to your use case.

pastedImage_8.png

 

For my test case, I generated alerts from the ESA with different "event_type" values:

pastedImage_14.png

 

...which meant I first needed to modify the "/var/netwitness/respond-server/scripts/normalize_core_alerts.js" file and add the "event_type" key.

 

Within "normalize_core_alerts.js" there is a "generateEventInfo" function, which is where we can define additional keys to be normalized by the Respond Server, and where I added my "event_type" key.

****NOTE: it is VERY important that you pay close attention to the formatting and syntax within this file when you add a new key, especially where trailing commas are needed/not needed.****

pastedImage_17.png

 

Next I modified the "/var/netwitness/respond-server/scripts/normalize_alerts.js" file and added a new line for my "event_type" key to the "normalizeAlert" function.

****Again, it is very important that you pay close attention to the formatting and syntax when you add your keys to this function block.****

pastedImage_19.png

 

Then I modified the "/var/netwitness/respond-server/data/aggregation_rule_schema.json" file and added a new schema for the "event_type" key.

****And yet again, pay very close attention to formatting and syntax when modifying this file, especially where commas are needed/not needed.****

pastedImage_21.png

 

****I recommend saving copies of each of these modified files, as they get over-written during the upgrade process.****

 

And finally, restart the Respond Server service, either from within the UI:

pastedImage_23.png

 

...or via command line from the Node0 Server:

# systemctl restart rsa-nw-respond-server

 

Give it a minute or two for the service to fully restart, refresh your browser, and you can now select your custom matchCondition and groupBy keys from the drop down menus:

pastedImage_25.png

pastedImage_27.png

 

...and view the fruits of your labor:

pastedImage_29.png

 

**Bonus note for anyone still reading: if you're like me and it bugs you if something is not in alphabetical order, you can adjust where your custom keys appear within the dropdown menus by inserting your custom schema within the "aggregation_rule_schema.json" file in a different location.

 

In the example shown above, I added my custom schema to the very end of the file, which is why my key appeared at the very bottom of each dropdown menu.

 

But if I place my custom schema in alphabetical order within the JSON file, it will appear within the dropdown menu in its new location:

pastedImage_31.png

 

pastedImage_33.png

 

pastedImage_35.png

 

Happy customizing, everybody!

  • Aggregation Rules
  • Custom
  • group by
  • incident fields
  • match condition
  • NetWitness
  • normalize alerts
  • NW
  • NWP
  • respond
  • respond-server
  • RSA NetWitness
  • RSA NetWitness Platform
8 Likes
Share
17 Comments

You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.

  • Comment
Latest Articles
  • Ransomware Email Attacks: Beware of BazarLoader
  • Detecting Impacket with Netwitness Endpoint
  • Exotic Lily: Global Activity Analysis
  • Threat Research Data Hygiene Exercise: Retirement of Threat Research Intelligence Content and Report...
  • Netwitness Orchestrator Dashboarding Overview
  • Highlights from Recent Releases - Here's What's New in NetWitness Platform 11.7 and 11.7.1
  • NetWitness News Bytes: Improved Broker Query Experience
  • NetWitness News Bytes: Meta Only Event Reconstruction
  • NetWitness News - Press Releases
  • Endpoint Bundle Tuning
Labels
  • Announcements 52
  • Events 2
  • Features 9
  • Integrations 6
  • Resources 56
  • Tutorials 21
  • Use Cases 20
  • Videos 116
Powered by Khoros
  • Blog
  • Events
  • Discussions
  • Idea Exchange
  • Knowledge Base
  • Case Portal
  • Community Support
  • Product Life Cycle
  • Support Information
  • About the Community
  • Terms & Conditions
  • Privacy Statement
  • Acceptable Use Policy
  • Employee Login
© 2022 RSA Security LLC or its affiliates. All rights reserved.