Checking Event Source Status via API/Script or Report

[Anonymous]

Is it possible to check the status of the event sources under 'Admin > Event Sources' via the API or remote script?

I have a daily check to make sure event sources are being sent to NW and parsed correctly and act on any source that hasn't reported logs to NW for a time and I'd like to speed up the time instead of manually checking each individual event source.

If that's not possible, is there an alternative like using the reporting engine to generate a report that would achieve the same outcome?

[JoshRandall]

I worked on an ESA rule with one of my customers aiming to accomplish a nearly identical goal - tracking log sources and identifying if/when the volume of ingested logs rises or falls dramatically. I based the majority of the ESA rule on the User Login Baseline syntax, leveraging the built-in EPL stats module to calculate standard deviations and then allow for alerting when those standard deviations deviate (plus or minus) from the calculated average.

I've packaged the rule into a Live Resource Bundle, so you can deploy the ZIP file through the Live interface within the UI, and then modify the parameters to fit your needs:

A couple points to consider when choosing your time ranges:

• the Time Period Length + Time Period Type control the window in which your logs will be evaluated for deviations from the average, which means....
  • the longer this window is, the more accurate your average will be and the less susceptible the rule will be to alert on short dips/spikes in event counts
  • the rule can only fire alerts at end of this window, so if there's a short period when ingest drops to 0 but....
    • it occurs in the middle of this window
    • the average at the end of this window is within your configured standard deviation threshold
    • --> you won't get any alert
• the longer you set your time widows, the more resources (CPU and memory) this rule will take on your ESA
• the more device types you monitor with this rule, the more resource it will take on your ESA

The rule will output alerts like

{"severity":5,"eventSourceId":"unknown","respondEnabled":true,"moduleType":"ESA_ADVANCED","engineUri":"test-sa-managed","moduleName":"Log Device Baseline","suppressMessageBus":false,"transientAlert":false,"notificationReasons":[],"actualEventsCount":1,"instanceId":"650a9caccf52a2c14f42dff1008dc737a2554ed092a5ef90c4649d4ba4b79c02","statement":"Log Device Baseline -- windows:192.168.20.20","id":"f2ca62cd-4e5f-49dd-8e29-e3a7a5ea6a25","time":"Mar 1, 2021 12:56:25 PM UTC","moduleId":"603965f1e4b06457803479ce","events":[
    {
        "device_ip": "192.168.20.20",
        "datapoints": 13,
        "last_average": 11705.923076923076,
        "last_population_stddev": 1163.316046847144,
        "device_type": "windows",
        "last_time_period_events": 7952
    }
],"suppressNotification":false}
{"severity":5,"eventSourceId":"unknown","respondEnabled":true,"moduleType":"ESA_ADVANCED","engineUri":"test-sa-managed","moduleName":"Log Device Baseline","suppressMessageBus":false,"transientAlert":false,"notificationReasons":[],"actualEventsCount":1,"instanceId":"391a7e7b256b34e2b90823914c78f8fab35c138208cba6548a9886af169d82eb","statement":"Log Device Baseline -- vmware_esx_esxi:192.168.2.4","id":"11545161-925a-4e89-9786-14271c8de482","time":"Mar 1, 2021 08:56:24 AM UTC","moduleId":"603965f1e4b06457803479ce","events":[
    {
        "device_ip": "192.168.2.4",
        "datapoints": 9,
        "last_average": 179,
        "last_population_stddev": 2.8284271247461903,
        "device_type": "vmware_esx_esxi",
        "last_time_period_events": 171
    }
],"suppressNotification":false }

[Anonymous]

That looks great @JoshRandall thanks for sharing. I'll give it a shot and see how I go.

[Anonymous]

Out of curiosity, is there a way to have this not apply to particular event sources?

For example, both our Active Directory DCs and Windows endpoints are using NWE to log ship their event logs to NW. Since they all show up as event source 'windows', this would start alerting on every workstation in the fleet. Something I'd rather not do since that would make the alerts really noisy.

I've added the DCs into a group under Admin > Eventsources > Manage, could that be utilised to filter our the Windows 10 workstations from the 'windows' event type?

[JoshRandall]

---

@Anonymous wrote:

Out of curiosity, is there a way to have this not apply to particular event sources?

---

Using the provided bundle exactly as-is, no. I would recommended setting your desired options in the existing rule (to get all the dependent calculations), then copy the rule's syntax into an advanced rule and add any additional filters to the ON Event(....) section...

...for the exact hosts or IPs you do want to include in the rule.

e.g.: 

on Event(
	medium = 32
	AND device_type IS NOT NULL
	AND device_ip IS NOT NULL
	AND device_ip IN ('10.10.10.20', '10.10.10.21', 'etc...')
	AND device_type.toLowerCase() IN ......

[Anonymous]

That makes sense, thanks.

Could I use a list within the context hub as an additional filter?

I'm thinking it would be easier to manage a list rather than modifying the ESA rule everytime a source is added or removed.

[JoshRandall]

Absolutely, that's one of the main benefits of CH Lists.

Even better, adding/removing values from the CH List will automatically reload the ESA rule for you - no need to redeploy the entire ESA deployment.

[Anonymous]

That sounds perfect. Thanks.