Automating Meta Key Updates with UDM

[JoshRandall]

While the release of the Unified Data Model (UDM) has given us a unified meta key foundation on which to build moving forward (awesome!), it has also opened an administrative can of worms (not so awesome...).

With these new and/or modified meta keys comes the challenge of combing through your NetWitness architecture to find all the places that the discontinued meta exist, identifying the discontinued keys that you want to change, and then actually changing them. We can’t automate this entire process yet, but we can still automate some to make our lives easier.

One of the primary places that meta keys live within NetWitness is the custom XML file that allows for tuning and adding to the default out-of-the-box meta. In the UI, these files are accessible at Admin (or Administration) → Services → <serviceName> → Config → Files:

And on disk at /etc/netwitness/ng/index-<serviceName>-custom.xml, (Log Decoders have an additional custom XML at /etc/netwitness/ng/envision/etc/table-map-custom.xml):

We could search through and update these files manually for every discontinued meta key...but frankly, that would be an enormous headache and a waste of time, which is why I put together this script to do it instead.

Before running the script, go to the UDM page on RSA Link (https://community.rsa.com/community/products/netwitness/rsa-content/udm) and check out the table of Discontinued Meta (https://community.rsa.com/community/products/netwitness/rsa-content/udm#deprecated). Copy the contents of this table (with or without the header – the script will omit that line if you do include it) into a text file. No modification of this copied table is necessary – again, the script will take of that for us.

Any discontinued meta keys from this table that do not have a specific 1-to-1 replacement meta key, such as orig_ip or any of the risk.* keys, will also be omitted when the script runs.

Next, copy this text file and the script to the filesystem of the appliance that you want to run it on (Log/Decoder, Log/Concentrator, Log/Hybrid, Archiver, or Broker), and make the script executable.

The script will require two arguments – the name of the text file that you copied the Discontinued Meta table into, and the name of the custom XML that you want to modify:

# python <scriptName.py> <text_file_with_copied_table.txt> <target_custom_file.xml>

For example:

# python replaceDiscontinuedMeta.py filename.txt index-concentrator-custom.xml

# python replaceDiscontinuedMeta.py filename.txt index-archiver-custom.xml

# python replaceDiscontinuedMeta.py filename.txt table-map-custom.xml

The script will ask whether to perform a dry run replacement or to do it for real. If run as a dry-run, you will get an output of all the discontinued meta keys that were identified within the target custom XML, as well as the new meta key that replaces it in the UDM.

If you do not choose the dry-run option, the script will give you the option to view each discontinued meta key and the corresponding new meta key and accept or deny its replacement, or to simply replace everything without any further prompts.

If the actual replacement(s) are accepted, the script will backup the original custom XML before making any changes.

Once complete, I recommend that you compare the new and original files using your diff tool or utility of choice to verify that everything proceeded without error. And as a reminder, you will need to restart the service for these changes to take effect.

Happy UDM'ing!

replaceDiscontinuedMeta.py.zip

[someone]

RSA once again decided to go through a massive change because they thought it would be better for customers, without being customers themselves.

It seems that this "new feature" has to be done manually by the customer, which is the same as if a customer wanted to change their META keys. 

Is there anything forcing me to change them? What would stop working If I didn't? In that case, what would "deprecated" really mean for customers?

[JoshRandall]

Hi Marinos,

  The UDM was developed to bring various meta in line with each other, the goal being to reduce the overall complexity of the meta framework.  This was done in large part because of feedback from our customers about the need to have a more standardized meta model.  In fact, many of my customers have been asking me about the UDM's release since they first heard about it during our roadmap discussions last year.  I should also point out that RSA/EMC/Dell uses NetWitness in our CIRC, so we are a customer of our own product and are affected by this the same as anybody else.

  The UDM page (RSA NetWitness Unified Data Model) will help answer many of your questions, but probably the most relevant blurb from this link is under the Discontinued Concepts section:

For backward compatibility of existing analytical content and analysts used to these keys, we will continue to use these keys. The discontinued keys will be removed from the default Content, Index-concentrator.xml and Table-map.xml files in future releases. 

Eventually, the discontinued meta keys will be replaced within the default XMLs.  Our OOTB content that relies on the old meta will be updated when that happens.

[SaketBajoria]

Hi Josh, 

Thanks for putting this together. I just wanted to clarify, you dont need to add the new keys to the Custom files, they have already been added to the latest OOTB config files. Also, the deprecated section shouldnt impact any backward compatibility for existing OOTB and/or Custom content that you have. All the changes that we make in OOTB content will be backward compatible as well. We just wanted to clearly document the list of keys that will be deprecated in the future. All new content should be created with this in mind. 

The UDM table documents every Meta Concept available in NetWitness today, and everybody (RSA. Customers, PS, CS, etc) should be using this to create content so that everybody is aligned to the same data model. Please send any further feedback to nws.udm@rsa.com

[SaketBajoria]

Hi Marinos, 

Thanks for the post. You dont need to change anything in your environment if you dont need/want to. This is just the overall guideline that we would like everybody to follow going forward. please let me know your concerns would be happy to setup some time to discuss this further. 

Thanks!

[someone]

Thanks Josh, 

Aligning METAs with eachother was possible for any customer to do manually if they wanted, it would have taken some effort though. With that said, this is not something new. 

I know that Dell's CIRC is using Netwitness but is not the same to compare a regular customer with RSA that have the RSA resources (PS,CS, CE) Also you shouldn't compare SOCs that consist of 50 people with another with 5. 

The section that you pasted is exactly what confused me. From the one hand it's saying that the keys will be left there for compatibility but then saying they will be removed.

I guess that if we just copied them from index-concentrator.xml to the custom one and same for table-map.xml then we should be completely unaffected. If that's the case then the wording in the documentation could have been much better with 1-2 examples. 

[someone]

Thanks Saket, 

I think I'm good for now. I understood that we just need to move the keys that RSA will deprecate to index-concentrator-custom.xml and table-map-custom.xml and we would be fine.

From the (lack of) documentation it seemed as some changes would be code changes and not just in the XMLs, but from your feedback it seems that this is not the case which is good.