Endpoint Alert Tuning
Overview
This blog post serves to help answer the most common question I receive in regards to Endpoint... "How do I tune the false positives?" This is not a guide on how to write new rules but the information here will certainly help you with that task. We will focus on modifying existing rules to be more high-fidelity instead of resorting to turning them off.
This is a visual for a standard, and very basic NetWitness deployment. Some key differences here are that hybrids exist. A hyrid appliance is a single server that contains both the decoder and concentrator services instead of two seperate appliances. In the case of Endpoint..the hybrid would also have the Endpoint service running as well.
This is how the Endpoint Bundle alerts work at a very high level.
Application vs ESA Rules
Application Rules
This section serves as a reference as to what application rules are and what they do. Application rules utilize the same query syntax that is used in the Investigate module. For example device.type='windows' is valid as both a query in Investigate and as an application rule. These rules provide META based alerts.
The application rules configuration page looks like this:
This is the breakdown behind an application rule name.
This is the edit page for an application rule.
Again, this is a META based alert. No alert will fire into the Respond module from an application rule. Since the rule generates a meta value (the title of the rule) in a specific meta key (Ex. ioc), you can carve data to look for matches to the rule. This is beneficial to automating data carves for hunting along or for simplifying alerting upstream with the ESA. As an example using the screenshots above; if we were to have an event match the 'transfer file using bits' we would be able to query ioc = 'transfer file using bits'and display all events that matched the application rule's criteria.
ESA Rules
ESA rules are used for event correlation. It is a very powerful service that allows customers to cross reference different data types (logs, endpoint (process, console, etc.), packets). I recommend referencing the following documentation before or while writing ESA rules:
The ESA is required in order to send alerts into the Respond module.
By default, the ESA Rules tab will bring you to this page.
You can create a new rule using the '+' icon.
I recommend learning and using 'Advanced EPL' as there is more flexibility in whatever it is you want to do. Tuning most ESA rules from Live effectively will require use of the Advanced EPL rule editor.
Tuning the Endpoint Bundle
tl;dr Endpoint Bundle rules are tuned via their associated application rules.
How it works
When the Endpoint Bundle is downloaded from Live; rules are pushed to both the endpoint hybrid's log decoder and the ESA appliance. The log decoder receives the application rules and the ESA receives all associated correlation rules. You aren't able to see all the correlation rules however. You will see a single rule called 'Endpoint Risk Scoring Bundle' that can not be modified. These rules check for the existence of meta values in pre-defined meta keys. The meta is generated by the application rule downstream when an event matches it's criteria. By altering the criteria of the application rule, we can determine what triggers an alert in the ESA and Respond.
If you modify an Endpoint Bundle application rule. DO NOT modify the meta key in which the alert is fired. If you do, you will have to disable the associated correlation rule on the ESA and write a new rule to look for the meta you specified. Otherwise the alerts for that rule will not fire at all.
How do I know what to tune?
Any alert that is generating a significant amount of false positives should be tuned. This does not necessarily mean any alert that consistently fires in a high volume. For example, if I have 1000 mimikatz alerts per day. Does it make sense to just tune the alert? Or does it make sense to find out why mimikatz is firing in such high frequency? A high volume of alerts can also be a sign of poor security practice and that should be taken into account when determining what to tune and how.
Where do I tune?
In 9 out of 10 engagements, the first thing I am told is that the Endpoint Bundle is not tunable; which is true to some extent. I'll walk through that below.
When viewing deployed rules on ESA, your screen will look like this (ensure you select the correct deployment in case you have more than one)
The Endpoint Bundle itself is not editable. If you double click the rule in the above screen, nothing will happen. If you click on the bundle within the rule library you will receive this error:
Rules from Live are not currently modifiable. Some rules will allow for basic tuning but for the most part the tuning needs to come from elsewhere. The nice thing is that tuning isn't required on the ESA for the Endpoint Bundle. The ESA rule(s) are simply looking for the existence of their appliacation rule counterparts in the form of meta. For example, the ESA rule for 'transfer file using bits' simply looks for ioc = 'transfer file using bits'.
A common example is the 'Runkey Persistence' alert. This rule is part of the Endpoint Bundle and is generally one of the first that requires tuning. In the example below, OneDriveSetup.exe (a default application on Windows) is the root cause for the majority of 'Runkey Persistence' alerts. By reviewing the alert we will be able to identify values that could be used for whitelisting.
By opening the alert we can see more details for the event that triggered the alert. Some useful values here are the path, filename, launch argument, user, hash, and registry key. From here you can investigate the event and determine if the rule is firing due to a bad practice occurring in the environment, if the event is part of a legitimate threat, or if the event is simply noise that happens to trigger the rule. The decision to whitelist should not be taken lightly. Prior to whitelisting a file/process/or action I ask myself the following questions:
- Why is the rule firing?
- Is the file/process triggering the rule still in use or is it a legacy scheduled job that has been forgotten?
- Is the rule firing due to bad practice that should be reviewed as part of a security policy update?
- Can I tune the rule without whitelisting?
- What're the security risks for whitelisting?
For the example rule at hand, I will tune the rule using the hash value for OneDriveSetup.exe as I know this safe file for my environment (a list of company approved software by hash value would make a great custom feed). Pick the meta key that will best suit the tuning you need to do. If I am tuning the actual detection tactic logic, I may opt to use a meta key such as param.src or param.dst to account for variations in the launch argument. If I am whitelisting a file as known good, I prefer to use the hash value as a bad actor could easily rename a malicious file with a 'good' name. By keeping track of known-good hash values we are able to alert on any new and/or unknown hashes. The hash for the OneDriveSetup.exe file is highlighted in blue below. In this example, I will take the hash value and will use it with checksum.src meta key.
You'll need to head over to the 'app rules' section of the configuration view on the log decoder service on the Endpoint Hybrid appliance. In this view, you can use CTRL-F or the table filter to find the rule(s) that require tuning.
My modifications to the rule being tuned are shown in red below. With the Runkey Persistence rule there is not much logic tuning that can take place as it simply looks for a runkey. If I were to run the same logic as a query in Investigate, the returned results would provide me runkey events. By appending the && (checksum.src != 'hash value') to the end of the query, I can easily filter out the runkey events for all files that have a hash value that match (in this case OneDriveSetup.exe). The same concept applies to application rules. Instead of a carve returning data, we will have a meta based alert that the ESA can leverage to efficiently generate an alert in the Respond module. I placed my whitelisting query in ( ) so that I could easily utilize or statements for any more identified hashes to whitelist going forward. It is important to understand that by whitelisting, you accept the risk of the whitelisted entity no longer being used for alerting purposes in the specific rule being tuned. Whitelisting is not always the best option (our services teams are great resources to help with tuning and content creation). After the rule editing is complete, you can save and apply the changes. There is nothing more you need to do. It is worth noting that the use of custom feeds containing known good filenames, hashes, ip addresses, domains, etc. could be used to help tune as well. Leveraging a feed would simplify the criteria syntax when higher amounts of tuning are required.
Threshold Breached Alerts
What does it mean?
The 'Threshold Breached' alerts and incidents are the result of the Endpoint Bundle. In the Incident Rules configuration page (see the 'Adjusting the Threshold' section below), there is a risk score threshold. Once a host's risk score is above that threshold, it will generate a threshold alert and incident. The easiest way to maintain these is to reset the risk score for a host once the alert the warranted the higher risk score has been investigated. The same concept applies to files (see the Files tab). The risk score can be reset by selecting a host in the hosts tab or a file in the files tab and clicking 'reset risk score'.
Adjusting the Threshold
The risk scores for hosts and files can be adjusted in the Incident Rules view. You can raise or lower the thresholds as needed for your organization.
1 Comment
