PacketDB retention exceeds MetaDB retention

[Anonymous]

I've got a question about the different sizes of databases that I'm still confused about.

If you have a PacketDB (log or packet) that has say a 60 day retention and a MetaDB that has say a 30 day retention, what happens if you query data past the retention period of the MetaDB?

Does the decoder or concentrator just take longer to go back through that older data to pull the meta out from it?

Thanks.

[KhaledGamal]

Hi Jeremy,

If the meta DB on the concentrator has a retention period of 30 days, you won't be able to investigate previous than that as the investigation normally happen on the concentrator/broker where no packet db is available. Packet DB has only raw data and it is only available on Decoders whether packet decoder or log decoder. It is nice to have a retention period almost the same for the meta db of the concentrator and the packet db of the decoders. The meta db on the decoder is not the one investigated but it is only used as a temp meta db till the concentrator aggregates the meta from the decoders. A 30 day retention period on the decoder is enough but for the concentrator it should be bigger than that.

If you have a packet db of 60 day retention on the decoder, there should be a meta db on the concentrator with around the same retention. If you need any clarification or you need me to elaborate on any part, please tell me.

Best regards

Khaled

[Anonymous]

Thanks for the explanation Khaled,

If I understand correctly, the MetaDB on a decoder acts as a cache, so when looking at the storage calculations on sadocs.emc.com and it says something like "x amount of space for an 8 hour cache" means that if the concentrator were to go offline the MetaDB on the decoder would act as a cache for up to 8 hours until the concentrator comes back on line and starts consuming from the decoder again.

In terms of storage space how does the SessionDB fit into equation with the PacketDB and MetaDBs?

Thanks.

[JohnSnider]

This might help Explain, something put together by John Wu

[KhaledGamal]

Hi Jeremy,

You are totally correct, When the concentrator goes offline for some time and comes back online, it goes and completes aggregation from the decoder from the point it last stopped. This means for example if the retention of the metadb is small and an outage happened on the concentator for a time period more than the retention, this will cause data loss as some data from the meta db of the decoder will be rolled out from the DB. From my perspective, I guess 30 days for the meta db on the decoder is enough.

Regarding the Sessiondb, RSA Security Analytics sessionizes the raw data to be able to reconstruct data like emails and web pages. Without session DB, SA won't be able to reconstruct sessions. Session DB is the data base which maps the raw data together into sessions to know which packet is related to which session and also links packetdb and sessiondb together.

The attached document from my colleague above might help give you the big picture.

Thanks and BR

Khaled

[Anonymous]

Thanks again Khaled,

The document does help, I understand how all the pieces fit together, what I'm trying to get my head around is how the storage requirements for those pieces impact on the retention of the data.

For example, if the packetDB has a longer retention than the metaDB or sessionDB, does that mean that the data that the packetDB has stored over the metaDB and sessionDB no longer accessible or is it a case of that you have to talk to the decoder directly and the time it takes to retrieve that information slower.

Cheers.

[KhaledGamal]

Hi Jeremy,

Here are a couple of scenarios that may help:-

- If the metaDB has data for January,February, March and April while the Packetdb has data for February, March and  April. Then if we investigate the data in January, the result from the investigation will return meta data but if you try to open the session "raw packet or raw log" it will give you an error that it can't get it.

- If the packetDB has data for January,February, March and April while the metaDB has data for February, March and April. When you try to investigate the Concentrator it won't get you any results for January as there is no meta data to retrieve.

If you try to investigate the decoder it self, you get results but not indexed so it is pretty much useless this happens because the decoder only index time meta while the rest of the meta is not indexed so you won't get very satisfying results.

That is why the optimum solution is to have all databases round up around the same time.

Hope this helps.

Best Regards

Khaled

[Anonymous]

Thanks Khaled,

That helps a lot and has cleared up the questions I had surrounding the DBs.

Cheers.

[KhaledGamal]

You are always welcome Jeremy.

Don't hesitate to reach out if you need anything in the future.

Cheers.