When should one create a new indexed key?

[TomiReiman]

In RSA NetWitness, when does it make sense to create a new index key instead of using an existing one to store more heterogeneous data?

My example use case is a key named 'hash'. If hashes from logs and other sources can be either MD5, SHA-1, SHA-256, etc., would it make more sense to create separate keys for each of them instead of using a single key called 'hash'? First of all, these hashes have different lengths so putting them all under a single key would make the size of the index bigger. Secondly, if I can extract feeds per hash type from external systems, wouldn't it make sense to only match MD5 hashes to MD5 hashes, SHA-256 hashes to SHA-256 hashes, and so on?

I plan on keeping the hash keys closed by default in the investigation view, so would there be any real and impactful downside of creating all these keys instead of just using a single key for all hashes?

[NaushadKasu1]

Hello Tomi,

It's a balance between data organization and query/system performance...

When it comes to NW Indexing, uniqueness matters -- in that, if you have a variety of unique values (as you would expect with hashes), the high number of unique values in any given "index slice", could be overrun if your valueMax setting for that key is not set appropriately for your given environment.

I can certainly elaborate on the topic above as I'm not sure how familiar you are with the way indexing works in NetWitness and its relationship of unique values stored per slice for a given key in that slice. Please let me know if you need me to dive deeper on this topic.

Given the information you have provided, it would make sense to make separate keys for each subset of data i.e.  hash.md5, hash.sha256 etc.. to avoid index overruns and/or lowering the index save frequency to accommodate the high number of unique values stored in those keys. Particularly, as you mentioned that you are doing matching against some feed content, it could certainly help keep down the query times as you use this data for reporting, charts etc...

[NaushadKasu1]

Tomi,

I should also mention that in NW 11.1, there is a new feature implemented called Meta Entities. This feature allows you to set a new "pointer" meta key in the index-concentrator-custom.xml that will "point" to other keys for the purpose of queries. So if you do move forward with separating your keys into hash.md5, hash.sha256 etc... In 11.1, you can/will be able to setup a new pointer meta key, let's call it "hash.all" and have it point to hash.md5, hash.sha256 etc.. so that any queries against hash.all would run a query against all of the sub-keys (entities) you have pointed to in the index-concentrator-custom.xml to make it easier to write queries.