NetWitness Endpoint Meta Primer

[JoshRandall]

The NetWitness Endpoint meta schema, while fully accessible, has remained a mostly opaque and little understood topic. Exactly what metadata gets created, where from, and how to modify/customize the schema is not something that we have provided any easy methods or tools to assist with. This blog aims to rectify those issues as much as possible.

First, to understand where and how NetWitness Endpoint metadata is created, we need to delve into nw-shell to view the default schema. We can this accomplish like so (endpointMeta.txt from v11.6 is attached to this blog):

# nw-shell
>> connect --service endpoint-server
>> cd endpoint/meta/get-default
/// to display meta schema only
>> invoke
/// to write meta schema to file
>> invoke outputFile /path/to/file.json

There is a lot of information in this output, so let's walk through an example of what the contents mean. 

....snip....
    {
      "metaKeyPairsCategory" : "MACHINE",
      "keyPairs" : [
        {
          "endpointJpath" : "users/domainUserOU",
          "metaName" : "dn",
          "type" : "text",
          "enabled" : true
        },
        {
          "endpointJpath" : "agentVersion",
          "metaName" : "version",
          "type" : "text",
          "enabled" : true
        }
....snip....

First, we have the Category in which all this metadata will appear --> MACHINE. There are a few caveats and special cases regarding this field, but in general the value here corresponds to the category metakey, and controls the metadata that will be created for that particular type of nwendpoint event (machine, in this case).

Next we have the contents of the keyPairs array, which is the real meat and bones of the schema. Each of these fields controls exactly how the metadata gets created (going from the bottom up...):

• enabled --> controls whether this particular meta get created
  • boolean
• type --> the metakey datatype (options are the NW metakey format types)
  • Int8, UInt8, Int16, UInt16, Int32, UInt32, Int64, UInt64, UInt128, Float32, Float64, TimeT, Binary, Text, IPv4, IPv6, MAC)
• metaName --> the NetWitness table-maps' envisionName value
• endpointJpath --> the JSON path in the endpoint events from where the metavalues originate

Now, you may be asking, "But Mr. Mongo, what are these endpoint events of which you speak? Endpoint events don't have any raw data...mostly... Where...? How...are we supposed to find these JSON paths?"

To which I would respond, "That is a very good question. But before I show you..."

All kidding aside, the config that controls this will end up causing potentially significant issues with your endpoint server if it is not managed properly. This will take all the incoming event tracking and scan data from all endpoint agents and write them all to disk at a location of your choosing on the endpoint server's local filesystem.

This can place a significant load on the endpoint server's resources. If your endpoint server is at or near max capacity (CPU, RAM, disk space), then I strongly recommend you not enable this setting.

Furthermore, if you enable this setting and leave it enabled until resources are exhausted, there is a very strong chance you will crash your endpoint server and potentially lose data.

You have been warned.

Now, in the interests of helping you to configure this in a responsible manner, I have put together a script (exportEndpointJSONs.sh - attached to this blog) that will enable the setting for a limited time - 30 minutes - and then disable it.

**This script must be run on an Endpoint Log Hybrid.**

After you run the script, it may be a good idea to initiate a scan on one (or a couple) of your endpoints to ensure you get a thorough and as-complete-as-possible export of JSON files and the associated event metadata that you might be able to extract from them.

**Some of these timestamp directories may contain additional/other JSONs, especially for completed scans, so be sure to check for those**

Once we have a set of JSON files, we can dig into the actual mapping and figure out where the meta is coming from–actually, scratch that.

Apologies if that’s something you’re interested in, but after putting together 16 pages in a rambling attempt to explain the intricacies of the endpoint meta mapping, I decided it would probably be more useful and worthwhile to just skip to the point, while doing my best to highlight the caveats and gotchas that I found. (If you really want to read through my bumbling process to figure this stuff out, I’ve attached the original draft of this blog in a word doc.)

Ok! We have a sample of exported endpoint JSON events, and we want to add some custom metas. Some rules of thumb, caveats, and gotchas to be aware of:

1. The JSON file name tells us the metaKeyPairsCategory in which we need to add our custom metas; e.g.: values coming from the certificate.json need to be added to either the CERTIFICATE or EVENT_SOURCE_CERTIFICATE category

2. The metaKeyPairsCategory tells us the nwendpoint category that will have our custom metas; e.g. the metas in category = ‘network event’ sessions are defined in the "metaKeyPairsCategory": "NETWORK" object

3. nwendpoint event categories (process event, network event, file event, etc.) can have metas sourced from any of the "metaKeyPairsCategory": "EVENT*" objects.

4. endpointJpath values coming from the event.json file will start inside the top-level windowsEvents, macEvents, and linuxEvents arrays; e.g.: the full JSON path in the screenshot below is windowsEvents[].sourceProcess.path but the corresponding endpointJpath value is sourceProcess/path:

   

5. endpointJpath values that correspond to JSON array will create multiple metavalues for the the mapped metakey; e.g.: each of the fields in the pe.features[] and signature.features[] arrays creates a metavalue in the context.src metakey:

   

6. metaName values must use the envisionName values found in the table-map.xml and table-map-custom.xml files; e.g.: a "metaName": "event_description" mapping would create metavalues in the event.desc meta key:

   

7. The interaction between the MACHINE_IDENTITY and EVENT_MACHINE metaKeyPairsCategory objects and their metas requires some extra tweaks and considerations:

   1. The MACHINE_IDENTITY mappings will always create all their metas in nwendpoint category = ‘machine’ events (e.g.: scheduled and ad-hoc scans)

   2. MACHINE_IDENTITY mappings create their metas in all other nwendpoint events if and only if the MACHINE_IDENTITY mapping’s metaName has a common, or shared, value with an EVENT_MACHINE mapping’s metaName

   3. For example…

      1. Custom meta entries for MACHINE_IDENTITY and EVENT_MACHINE mappings will create custom meta in all nwendpoint events if EVENT_MACHINE has any mapping with that same metaName

         

      2. Custom meta entries for MACHINE_IDENTITY mappings will not create custom metas in all nwendpoint events if EVENT_MACHINE has no mapping with that same metaName

         
8. And remember – as is usually the case in NetWitness, custom entries/selections/configurations will override the default, OOTB settings

With all that in mind, now we can go ahead and add some custom endpoint meta mappings. Like I normally do, I’ve created another script (customEndpointMeta.sh - attached to this blog) to help you add your own custom metas to supplement or modify the OOTB mappings.

This script will help create, modify, and apply custom endpoint mappings. It takes no arguments and will prompt for all necessary inputs.  It must be run on your NODE0 server.

None of the commands or interactions with NetWitness in this script will cause or require any service interruptions or restarts.

The only input validation this script enforces is on the metaKeyPairsCategory selection. Everything else (endpointJpath, metaName, type) is up to you verify for accuracy. I recommend double-checking for incorrect entries and/or typos to avoid unnecessary issues.

New metas created from this script will take approx. 2-5 minutes to be available in your NetWitness instance, depending on incoming endpoint events and the speed/performance of your system.

Lastly, the script will help merge new mappings with any existing mappings, so you don't have to recreate every single mapping every time you run the sciript.

If you run into any issues or problems, please comment on this blog with as much information (plus screenshots) as you can.