Investigation data not available for more than 10 days

[VishamRawat]

I've been told that if my Log Decoder hasn't enough space on its packetdb volume, the data won't be available for investigation as well, and will be rolled over as new data comes in.

So, if the Log decoder stores all the raw logs for the purpose of investigation, then I understand that it has to be allocated 1.9TB worth of disk space.

/dev/mapper/VolGroup01-sessiondb
                       30G   29G  1.8G  95% /var/netwitness/logdecoder/sessiondb
/dev/mapper/VolGroup01-metadb
                      114G  107G  7.1G  94% /var/netwitness/logdecoder/metadb
/dev/mapper/VolGroup01-logcoll
                       64G   15G   50G  23% /var/netwitness/logcollector
/dev/mapper/VolGroup01-packetdb
                      1.9T  1.8T  111G  95% /var/netwitness/logdecoder/packetdb

What I'm unclear about is the purpose of allocating 1.8 TB to the metadb volume on the Concetrator. What exactly is stored within?

/dev/mapper/VolGroup01-sessiondb
                      180G  171G  9.2G  95% /var/netwitness/concentrator/sessiondb
/dev/mapper/VolGroup01-index
                       84G   20G   65G  24% /var/netwitness/concentrator/index
/dev/mapper/VolGroup01-metadb
                      1.8T  1.7T  101G  95% /var/netwitness/concentrator/metadb

Also, which volume is responsible for the availability of data on the investigation console? Is it the packetdb volume on the Log Decoder or the metadb volume on the Concentrator? As per the meta.oldest.file.time value is concerned, it seems like the value for this key found on the Concentrator is the indicator of console availability.  

Please clarify.

[JohnKisner · Accepted solution]

Visham,

On the concentrator the metadb, sessiondb and index partitions are all important. The data in them combine to create what you see in the Navigate view of Investigation. The metadb and sessiondb are used to store the same data that comes from the log decoder. The metadb specifically contains all the meta generated on the log decoder from the parsers, feeds, and app rules. the sessiondb is used to contain the pointers that connect meta data to session data to log/packet data. The index contains the indexing of the meta data on the concentrator and is the most important for Investigation as it is what Investigation used directly to create the Navigate page. When you want to know how far back you can investigate you need to look at the oldest file time on each of the sessiondb, metadb and index of the concentrator. Once you have the time/dates you take the one that is the closest to current and that will be how far back you will be able to investigate. As mentioned before this is because the three rely on each other.

On log decoders a lot of meta is generated per unit of time compared to a packet decoder. This is why a large metadb on the concentrator is a must when connected to a log decoder. The metadb, and sessiondb on the log decoder itself is just a temporary holding location for this data until the concentrator can consume it from the log decoder. This allows for a buffer in case the concentrator is offline or unable to connect with the log decoder. This is also why these partitions are so much smaller than what is on the concentrator. The index on the log decoder is only there to index what the decoder needs and is not transferred to the concentrator as the concentrator generates its own indexes at time of ingestion of the session and meta data from the log decoder. It uses the index-concentrator.xml and index-concentrator-custom.xml to create the required indexes.

I hope this helps answer your question.

[RajveerSingh]

Dear Visham, 

That's true, if Log Decoder hasn't enough space on its packetdB volume, then data won't be available for investigation and will be rolled over as new data comes in. 

Data will not be available for investigation that means you will not be able to see raw logs. but you will be able to see meta data as concentrator already aggregated meta from that decoder and it is indexed and stored in concentrator storage. Yes you will not be able to perform proper investigation as concentrator can provide only Meta value and key, event analysis can not be performed. 

[JohnKisner]

Visham,

Rajbir does have a point in that if the packetdb does hit its limit and starts to roll over data those raw logs will no longer be available during investigations. This is the reason we always suggest to our customers to have an archiver available to provide for longer term storage of the raw logs. In my personal experience I've seen a customer's log decoder packetdb have more retention then their concentrator due to logs, in general, taking up very little space compared to the meta generated from those logs. If the indexes or meta on a concentrator roll out before the logs on the log decoder, the logs become inaccessible anyway.

Please note that under most circumstances the raw logs, once processed, should no longer be needed for a comprehensive investigation of an issue. Each log is parsed extensively to provide all the meta you should need to perform your investigation even without the raw logs. I'm sure there are those in the community that may disagree with me on this but I suspect it is because they started their network analysis career using Wireshark and other similar tools. There is nothing wrong with this, I suspect they are just more comfortable or familiar with that way of analyzing data in these tools. Netwitness is different from these tools in that its specifically designed to generate a massive amount of meta data from each raw log so that the raw log is secondary and used primarily for compliance purposes only. If you have to review the raw logs during an investigation to find information then our parser may need an update to allow more meta to be created. This is why more stress is placed on the storage of the concentrator than of the log decoder. The concentrator's meta, session, and indexes are what your Investigation relies on 98% of the time. The remaining 2% or less is reviewing the raw logs. 

I wanted to reply so that everyone had a clearer picture of how Netwitness handles data retention, why certain database partition sizes are important, and to show that sometimes when working with Netwitness it requires a different way of looking at the data captured. 

[VishamRawat]

Thank you Rajbir, very clear now!

[VishamRawat]

Hi John,

Thanks for the detailed explanation. Much clearer now.

Just a quick query, and I'm glad you brought it up, since I've been wondering about it.

What if a built-in parser isn't creating meta for a particular piece of data within the event log, which is important for a certain use case? Is there a way to update it? Does RSA make such updates?
To be specific, there is a popular attack called Kerberoasting, which needs the parsing of Ticket Encryption Type and Service Name from within the event log. However, I see that RSA's built-in parser isn't meta-tagging these values, and simply ignoring them.

What do you think my options are, in such a case?

[JohnKisner]

In this situation you can put in a case with Netwitness Support for an update to the base parser for that log type and our Content team will make the needed update. If you do submit such a ticket please make sure to provide several samples of the log type and show what pieces you are looking for that need to be parsed. Any details as to its usage helps as well, such as saying it is for an attack called Kerberoasting. Please note the content updates do not happen immediately. It can take some time depending on complexity of the request and their current backlog of updates/additions.

The other option is to take the existing parser and using the Log Parsing Tool https://community.rsa.com/docs/DOC-94172 and create your own parser. This tool allows you, with a sample of the log, to designate what information you want to put into what meta keys. 

If you are looking for a new event source to be supported I suggest going to the RSA Ideas page https://community.rsa.com/community/products/netwitness/ideas and putting the request there. This allows others to vote on new features/event sources for addition to the product.

I hope this helps to answer your question.

[VishamRawat]

Thanks John! I'll do so accordingly