Concentrator Losing Data

[RenatoGoncalves]

Hello

Im noticing that the concentrator is losing data....since yesterday we lost 3 days of logs: We could query since 19 of September and today we can only see data since 22 of September. I need to stop that....is that possible?

How can i put the logs that are in the log decoder because the log as data since 9 September. Only with the suggestion from Dave Glover? https://community.rsa.com/message/917738? How can i set da bigger data retention?

[DaveGlover]

Renato

Have you checked the concentrator to see what the oldest meta or index time is?  Sounds like you are rolling the data in the database.

Did you get a burst of logs yesterday that would have filled the DB?

Dave

[EricPartington]

There are many factors that will determine how much data (meta) you keep relative to the raw data (log/packet raw) that are too complicated to figure out over Link.  I would suggest that you contact RSA support and open a case for them to review the architecture and make sure all settings, key values are set properly.

Data will roll in the NW system, once a partition gets to 95% full it will be FIFO (first in first out) .  The rate of meta into the concentrator will determine how much data you save, storage is fixed (unless you buy/add more) so in general, more data in, less data kept for the same storage volume.

[JohnKisner]

Renato,

This may be normal data roll out due to storage being full. Here is how you can check.

1. SSH to the Concentrator and run df -h. What you are looking at is the /var/netwitness/concentrator/metadb, /var/netwitness/concentrator/sessiondb, /var/netwitness/concentrator/index. Depending on if you have a DAC attached you may have a sessiondb#, and metadb#, where # is some number. You are looking for any of these partitions that are at about 95% utilized.

2. Next you go into the Concentrator's service Explore view. Once in the Explore view go to the database -> stats node. In here there are two stats you want to look at meta.oldest.file.time and session.oldest.file.time. 

3. Then go to the index -> stats node in Explore and look at time.begin.

4. With these three dates compare them to each other. Which ever one has a date that is the closest to current this is how far back you can Investigate on this concentrator.

You can only investigate as far back as you have data in all three locations on the concentrator. If you can provide these three dates here we can try to help you discover if there is any way to extend your investigation retention without adding additional storage.

[RenatoGoncalves]

Hi Dave,

How can i see that?

[RenatoGoncalves]

Hi Eric,

I opened....but i'm put against the wall by me boss and the support is still waiting for the sosreport that my colleague sysadmin is trying to get. I to top that my sysadmin colleague is going on vacation today

[RenatoGoncalves]

Hello John,

This is what i can see:

Meta.Oldest.File.Time - 24/9/2018 15:28

Session.Oldest.File.Time - 18/06/18 03:12

Time.begin - 20/09/2018 14:17

[DaveGlover]

Renato

Admin -> Services -> Concentrator -> Explore

Open database and click on stats.  The two highlighted values are a great place to start

[JohnKisner]

Renato,

Based on this the meta databases only go back to September 24th so that is as far back as you will be able to investigate on this concentrator. The only way to extend this time is to produce less meta data. Which means either less logs coming into the log decoder or adding additional storage to the concentrator. There is no way to bring those back. This is why most customers have archivers as they compress and collect only a subset of all meta created. It allows a customer to run reports against older data that has already rolled out of the system. 

I highly suggest looking at all your concentrators so you have a good idea of what kind of retention your system has. I suspect that your concentrators may have many different retention rates depending on the EPS of the underlying decoders.

Remember the more meta you produce the quicker storage is consumed and the faster meta, sessions, and indexes roll out.

[RenatoGoncalves]

We only have one concentrator.....:(

Can we re-inject he logs that are in the log decoder like in the article of Dave Glover, for report creation? We need to create a report for our client....