2014-01-22 04:07 AM
Hi
I would like to ask whether anybody had tried to anonymize certain meta-key or logs in SA when it is received by the Collector and stored in the Decoder.
e.g. a weblog having the ip address <x.x.x.x> when it is stored instead of its original value.
Thank you very much for your time.
Regards,
Khalil
2014-03-17 06:59 AM
I've been looking into this for clients in Europe, and while I haven't tried it all in detail, I think the following items will give you a workable solution:
Do keep in mind the original log message cannot be modified, and will either be stored as received, or it will be filtered/truncated. (Filter removes log and meta, and truncate remove raw log and keeps meta).
For the meta, we have more options, and we can follow two approaches:
1) For meta for which there is an exhaustive list (internal email addresses, internal ip addresses, hostnames usernames), we can setup feeds that will add a custom token for each value. The list would need to be prepared (based on CMDB dumps, Active Directory exports, etc) and be set up as a recurring feed. This would require some scripting but you are effectively preprocessing tokenization.
2) For meta for which there is no exhaustive list (internet ip addresses, other email addresses, etc) the only approach is to use a custom parser that will obfuscate the value. (I still need to write this, but I imagine a meta callback parser in Lua that runs an MD5 or CRC32 on a meta value would not be too difficult).
Don't forget to set the original meta to "Transient" in your table-map-custom.xml, or it will store the original data as well, defeating the point of the whole exercise.
Also there is an interesting security features in the Concentrator called the 'sdk.roles', if you go to Explorer (/sdk/config, I think) you can set this to value '2' and this will enable an additional security model giving you RBAC access to individual meta keys. (It will cause a slight performance hit of course)
Michel.
(updated 2014-03-17 to reference correct table-map-custom.xlm)
2014-03-16 12:32 PM
possible need to raise enhancement request?
2014-03-17 06:59 AM
I've been looking into this for clients in Europe, and while I haven't tried it all in detail, I think the following items will give you a workable solution:
Do keep in mind the original log message cannot be modified, and will either be stored as received, or it will be filtered/truncated. (Filter removes log and meta, and truncate remove raw log and keeps meta).
For the meta, we have more options, and we can follow two approaches:
1) For meta for which there is an exhaustive list (internal email addresses, internal ip addresses, hostnames usernames), we can setup feeds that will add a custom token for each value. The list would need to be prepared (based on CMDB dumps, Active Directory exports, etc) and be set up as a recurring feed. This would require some scripting but you are effectively preprocessing tokenization.
2) For meta for which there is no exhaustive list (internet ip addresses, other email addresses, etc) the only approach is to use a custom parser that will obfuscate the value. (I still need to write this, but I imagine a meta callback parser in Lua that runs an MD5 or CRC32 on a meta value would not be too difficult).
Don't forget to set the original meta to "Transient" in your table-map-custom.xml, or it will store the original data as well, defeating the point of the whole exercise.
Also there is an interesting security features in the Concentrator called the 'sdk.roles', if you go to Explorer (/sdk/config, I think) you can set this to value '2' and this will enable an additional security model giving you RBAC access to individual meta keys. (It will cause a slight performance hit of course)
Michel.
(updated 2014-03-17 to reference correct table-map-custom.xlm)
2014-03-17 10:04 AM
Thanks Michael, how to do this?
Don't forget to set the original meta to "Transient" in your index-decoder-custom.xml, or it will store the original data as well, defeating the point of the whole exercise.
2014-03-17 10:11 AM
I'm assuming you are on 10.3SP2, the process was a little different before this. Also I specified the wrong file, it's not index-decoder-custom.xml, but it's table-map-custom.xml. This only works on log decoders, not packet decoders.
Go on your log decoder to /etc/netwitness/ng/envision/etc and copy the file table-map.xml to table-map-custom.xml
Then you can modify this file and your changes will not be overwritten when the table-map.xml is supplied with the next update. You can modify this with 'vi' or another editor and change the "flags" field from "None" to "Transient". This will make sure the meta is not written to disk. You can also make other modification such as turning Transient meta into normal meta.
2014-03-17 10:15 AM
2014-03-17 10:35 AM
thanks. let me check out.
what does all the options means?
None|File|Duration|Transient
It there any same concept for decoder?
2014-03-17 10:38 AM
I don't know what all the flags mean. Just that Transient means it's available for application rules and feeds, but it is not written to the metadb.
The packet decoder unfortunately does not have the same concept.
2014-03-17 10:54 AM
thanks again.