Raw logs and metadata storage and flow

[VishamRawat]

I just wanted a clearer understanding of the flow of raw logs and metadata and how and where they're stored within the RSA SA architecture componentry.

My current understanding is, the raw logs are sent via the VLCs to the Log Decoder. They are processed by the built-in Log Collector service on the Log Decoder before being parsed, is that right? Or are they sent directly for parsing to the Log Decoder from the VLCs?

Now, once parsed, the raw logs are stored on the Decoder as well as sent to the Archiver (if deployed and configured), and the metadata (metakey + metavalue) is sent to the Concentrator for indexing.

At the Concentrator, the metadata is stored and indexed, and then sent to the ESA (if deployed and configured), where alerts are generated and stored.

In summary, here's my understanding of storage on each component.

1. Raw logs are stored on the Decoder and Archiver only

2. Metadata is stored on the Concentrator only

3. Correlation alerts are stored on ESA only

4. Basic alerts are stored on the Decoder only

5. The Broker does not store raw logs, metadata or alerts

However, if the above is true, my question is when we investigate via the SA console and select a Concentrator, how do we get a raw log view of the event, if the Concentrator doesn't store raw logs?

[Anonymous]

Visham, you have that all pretty much correct. I really just want to clarify some things to make sure and avoid questions you might have in the future.

1) The VLCs send the logs they collect to a Decoder where they are parsed. While you do have to have a VLC setup as a Remote Collector and a Local Collector on the Decoder, the Local Collector service on the Decoder doesn't do much other than accept the logs from the Remote Collector and get some information about the log. For example, if the VLC knows it's sending a SQL log, the Decoder can spend less time finding the right parser.

2) After the Decoder parses the logs, it actually stores the raw logs and metadata locally on the Decoder, but the metadb size on the Decoder is small since it only needs to hold as much meta as it needs to in case the Concentrators go down for any reason (could be an hour's worth, a day's worth, etc.). Concentrators and Archivers pull data from the Decoder. The Concentrator pulls only meta from the Decoder, while the Archiver pulls both raw logs and metadata. They both index the metadata they pull, but while Concentrators have SSDs for the indexes, Archivers generally do not. Also, the Archivers only pull the meta keys they're told to pull.

3) If you are looking at an event (network session or log) and you want to see the raw data, the UI sends the session ID to the Decoder in order to pull the raw event.

4) An "Alert" in the system is, I would say, something that makes it into the Respond queue (assuming you're on v11) as an Alert and can be used to create an Incident (assuming an Incident Rule criteria is met). The ESA's rules generate those. You can also "alert" on an App Rule that runs on a Decoder (Log or Packet), but when you check the "alert" box, you then choose which meta key you want to send that "alert" to. All that does is tag that event with an entry in the meta key you chose with a name matching the name of the App Rule that triggered. You could build an ESA Rule that looks for just the existence of that entry in that meta key to generate an ESA "Alert" to make it work both ways, but you can do whatever makes sense in your environment.
5) Since the "alerts" generated by App Rules are really just metadata, I would say that really those are "stored" (to use your terminology) on the Concentrator with the rest of the metadata. ESA Alerts (and Incidents) are stored in a MongoDB on the ESA.

Did I cover everything you were curious about?

[VishamRawat]

Thanks Sean, real helpful.

Just to confirm my understanding, and whether it is right or not.

1. Decoder stores raw logs and metadata

2. Archiver stores raw logs and metadata

3. Concentrator stores metadata and the index

4. ESA stores alerts and incidents

5. App rule alerts are metavalues tagged to the alert ID metakey. Hence, these are stored on the Concentrator as well as Decoder (and Archiver for LTR)

What about the alerts generated under the Report module? where exactly are these stored, and which component generates them?

[DaveGlover]

Visham you have it mostly correct, however:

1. Decoder stores raw logs and metadata -> Meta on the decoder is only held for a short amount of time to allow for the concentrator to pick it up.. 

What about the alerts generated under the Report module? where exactly are these stored, and which component generates them? -> this info is stored on the head unit.  The reporting engine is the originator of the alert.

[VishamRawat]

Thanks Dave