Archiver Storage

[VishamRawat]

#

Okay, maybe I'm not getting the whole picture here, but I'm wondering what exactly is sent to the Archiver from the Decoders.

For example.

I have 3.5 TB of Archiver storage. And this has been designed for 9 months of hot storage. Additionally, the Log Decoder has an ingestion rate of 150GB per day. This means 3.5 TB in about 20-25 days.

Now, as per my understanding, the Archiver is sent a copy of all raw logs (and meta generated on the Log Decoder). So, shouldn't the Archiver be full and start rolling over logs in less than a month? Yet, the oldest meta file I see if of December? How is this possible? Or is it that not all data from the Log Decoder is sent to the Archiver?

[GuyWilliams]

Hi Visham,

The data on the Archiver is compressed by default as it is for long term storage and not investigation.

Thanks,

Guy

[VishamRawat]

Thanks Guy. Completely missed that. Just another quick query. I've recently seen the Duration on the Archiver Hot Storage change from 9 months to 8 months. Does this mean, that the volume of logs streaming into the Archiver has increased, resulting in the rollover of a substantial volume of older logs?

[GuyWilliams]

Yes, that's most likely the case.  Either more logs or bigger logs came in. 

The Archiver saves the logs in database slices.  Once the disk usage exceeds 95% the oldest slices are removed to bring it below 95% usage.

You can configure warm (still online) or cold (offline, inaccessible) storage to roll the older slices over to.