Everything in RSA NetWitness Platform is based on the concept of meta information ("information about information"). Both logs and packets are parsed at the time they are captured by the platform and meta information is then generated, even before the raw data is written to disk. Investigation, reporting, alerting, all are based on meta.
RSA NetWitness Platform provides 300+ unique meta keys out of the box, but only 70 or so are enabled by default. This implies that not all the parsing variables populated by log parsers shows up in Investigation, and thus not all of this information is available for reporting unless the platform is configured accordingly.
This topic documents the steps needed to configure the Log Decoder and Concentrator services to make additional parsing variables available within NetWitness.
The meta generation process
Within RSA NetWitness Platform, the Decoder is responsible for parsing the incoming events, generating the meta upon them and storing the raw data. The Concentrator retrieves the meta from the decoder and stores and indexes it.
This implies both the modules are involved in the meta generation and storing process.
This approach implies that a potentially huge amount of meta data can be generated out of the platform. That is, it is similar to running a report containing all the possible columns for each incoming event.
To avoid this situation, to save disk space and to prevent unnecessary information being stored, only the most relevant meta (around 70) are kept and stored by default; all the others are silently discarded. This behavior can be fully customized.
From a high level prospective, a meta can be generated by one of the following means:
- Log Parsers
- Feeds
- Application Rules
- Basic Correlation Rules.
Log parsers, feeds, and application rules are applied by the Decoder. Basic Correlation Rules can be applied by the Decoder or the Concentrator.
Understanding which meta the Log Decoder is generating
The Log Decoder applies the log parsers to recognize log messages and to extract relevant information. The parsers are enabled on the Log Decoder (Administration > Device > Log Decoder > Config). The log parsers are applied for each new log message, and parsing variables are populated with the values coming from the parsing process.
In order to conserve storage and to filter out not-relevant information, not all the variables generated by the parser are kept. Some of those are discarded: if they are marked as Transient by the Decoder, they are discarded. The following picture describes the process:
This behavior is configured within a configuration file called table-map.xml which is maps each parsing variable into a valid meta, and specifies to the Decoder service which meta has to be kept (flags=”None”) and which meta has to be discarded (flags=”Transient”).
In the example below, the parsing variable msg_id is mapped to the meta called msg.id and since its flag is set to None, the meta is kept andis available to the Concentrator for indexing.
<mapping envisionName="msg_id" nwName="msg.id" flags="None" />
Where envisionName is the name of the parsing variable, nwName” is the meta key.
On the other hand, in the example below, even if the parsing variable content_type is mapped to the meta called content.type, its flag is set to Transient and so the meta is discarded and is not available to the Concentrator.
<mapping envisionName="content_type" nwName="content.type" flags="Transient" />
How to make a transient variable non-transient
If we are interested in a specific parsing variable which is not available by default, we need to make it non-transient. To do so the flag needs to be changed from Transient to None.
You need to update the table-map-custom.xml file on the Log Decoder.
To update the table-map-custom.xml file:
-
Depending on your version:
- For NetWitness 11.x: Navigate to ADMIN > Services.
- For Security Analytics 10.x: In the Security Analytics menu, select Administration > Services.
- In the Services grid, select a Log Decoder.
- From the Actions menu, select View > Config, then select the Files tab in the Services Config view.
-
Select table-map-custom.xml from the drop-down list.
The table-map-custom.xml file opens in edit mode.
Note: If you edit the table-map.xml file instead of table-map-custom.xml, your changes are overwritten when you update RSA NetWitness Platform.
-
Change the line from:
<mapping envisionName="content_type" nwName="content.type" flags="Transient" />
to:
<mapping envisionName="content_type" nwName="content.type" flags="None" />
- Click Apply to apply your changes.
Understanding Which Meta the Concentrator is Indexing
Even if a meta key is generated and kept by the Log Decoder following the instructions above, this does not mean it shows up in Investigation. That is, the Concentrator service needs to be instructed accordingly to make some room in its index database for the new meta key.
This behavior is configured within a configuration file called index- concentrator.xml, which lists all the meta for which the concentrator is allocating a buffer within its index, resulting in the information being available for investigation and reporting.
To do so, the index level needs to be changed from IndexNone (the default setting if the meta is not listed in the file) to IndexKeys or IndexValues.
The following picture describes the process:
How to make the Concentrator index a Meta Key
Referring to the example above, if we are interested in the content.type meta, we need to alter the index-concentrator-custom.xml file by adding the following line if it does not exist:
<key description="Content Type" format="Text" level="IndexValues" name="content.type" valueMax="10000" defaultAction="Open"/>
Where:
- The format value has to be the same as the format used in table-map-custom.xml.
- The description is the nice name shown in Investigation
- The name is the meta name
- The valueMax” is the size of the buffer allocated for the meta (FIFO of distinct values)
- The defaultAction is how the information appears in Investigation
Note: The index-concentrator.xml file is overwritten upon each upgrade so to make the change persistent, you must edit the index-concentrator-custom.xml file.
You need to update the table-map-custom.xml file on the Log Decoder.
To update the index-concentrator-custom.xml file:
-
Depending on your version:
- For NetWitness 11.x: Navigate to ADMIN > Services.
- For Security Analytics 10.x: In the Security Analytics menu, select Administration > Services.
- In the Services grid, select a Concentrator.
- From the Actions menu, select View > Config, then select the Files tab in the Services Config view.
-
Select index-concentrator-custom.xml from the drop-down list.
The index-concentrator-custom.xml file opens in edit mode.
-
Add the following line:
<key description="Content Type" format="Text" level="IndexValues" name="content.type" valueMax="10000" defaultAction="Open"/>
- Click Apply to apply your changes.
IndexKeys vs. IndexValues
If a meta key is set to IndexKeys in the Concentrator index configuration file (index-concentrator-custom.xml), this means that the index structure links for each meta key all the events IDs whose meta exists (regardless of the value):
Filename:
- Event 10;
- Event 23;
- Event 78;
This implies that a query like “filename exists” would result in a prompt answer, whereas “filename = a.php” would require the Concentrator to generate the index on the fly by accessing the metadb and retrieving the value of the meta key for each event.
If the meta key is set to IndexValues, the index stores in a different data structure FOR EACH value the corresponding events IDs “tagged” by the specific value:
Filename:
a.php:
- Event 10;
- Event 23;
index.html:
- Event 78;
This means using IndexValues generates an index which is 10-100 times faster to be accessed, but larger in size than if you use IndexKeys.
Putting everything together
To summarize, the process for making a parsing variable (not enabled by default) available within RSA NetWitness Platform is the following:
