The content you are looking for has been archived. View related content below.
Custom, XML-based parsers can be deployed to parse and tag a variety of traffic. Often, these parsers may have been written to make use of new types of meta. This is not a problem since the NextGen engine is designed to be extensible.
There are two configuration items that must be considered when deploying a custom parser that makes use of new meta types.
Configuration Item 1
When you deploy a new custom parser and that parser requires meta definitions that haven't previously loaded into the system, you may receive the following error from the system:
Parser Custom_Parser registered for meta entry custom.meta but entry was not found.
In this case, a new parser called Custom_Parser was loaded into a Decoder and it required a new custom meta type called custom.meta. If the entry for custom.meta wasn't already loaded in the Decoder's index-decoder.xml file, then the system would generate the error shown above. We would expect this from the system since it is being asked to parse on a meta-type for which it doesn't have a definition.
Configuration Item 2
XML parsers can be read in any text editor. Many custom parsers have comments in them that help you define the custom meta. The issue that may arise is that you need to load the new custom meta definition into your entire infrastructure if you want to make use of this parser across your infrastructure. This includes Decoders, Concentrators and (if you use them) Brokers. While meta needs to be defined on these devices, it only needs to be indexed at the Concentrator and Broker. Indexing a new meta value at the Decoder-level can adversely affect performance on the Decoder and serves no purpose when it comes to reporting and drilling into data.
Before deploying any custom XML parser, open the parser file in a text editor and read all the comments. All of the comments provide worthwhile information on the operation of the parser but the meta definition is of particular importance to the operation of the system. It may look something like this:
Keys required by this parser:
<key description="Custom Meta" format="Text" level="IndexValues" name="custom.meta" valueMax="5000" />
Brokers:
This section only applies if you are using a three-tier infrastructure that includes Brokers. If not, you may skip to the next section titled "Concentrators".
Take a look at the value: level. In this case, it's set to level="IndexValues". In this case, what you are seeing is the entry you need to add for index-broker.xml file for the Broker. You can copy the line <key description="Custom Meta" format="Text" level="IndexValues" name="custom.meta" valueMax="5000" /> and paste it into the index-broker.xml file before the final </language> tag. This can be done easily through the "files" menu in NetWitness Administrator.
Note: Any time you change an appliance's index-"appliance".xml file, you need to restart that appliance's service so that the changes can be loaded into the NetWitness NextGen engine. Once the changes are applied they will apply to data collected or aggregated from that point on, but not to previous data.
Concentrators:
Take a look at the value: level. In this case, it's set to level="IndexValues". In this case, what you are seeing is the entry you need to add for index-concentrator.xml file for the Concentrator. You can copy the line <key description="Custom Meta" format="Text" level="IndexValues" name="custom.meta" valueMax="5000" /> and paste it into the index-concentrator.xml file before the final </language> tag. This can be done easily through the "files" menu in NetWitness Administrator.
Note: Any time you change an appliance's index-"appliance".xml file, you need to restart that appliance's service so that the changes can be loaded into the NetWitness NextGen engine. Once the changes are applied they will apply to data collected or aggregated from that point on, but not to previous data.
Decoders:
On the Decoder, you'll remember in Configuration Item 2 above that we don't want to index the new meta value at the Decocder-level but we must define it at a minimum for it to work. We can easily do this by copying the meta value from the Parser above but changing the part that reads from level="IndexValues" to level="IndexNone". So using the example we have been using all through this solution the following line will work in index-concentrator.xml and index-broker.xml for your Concentrators and Brokers:
<key description="Custom Meta" format="Text" level="IndexValues" name="custom.meta" valueMax="5000" />
But for your Decoders, you'll want to modify the line before adding it to index-decoder.xml to look like the example that follows:
<key description="Custom Meta" format="Text" level="IndexNone" name="custom.meta" valueMax="5000" />
Note: Any time you change an appliance's index-"appliance".xml file, you need to restart that appliance's service so that the changes can be loaded into the NetWitness NextGen engine. Once the changes are applied they will apply to data collected or aggregated from that point on, but not to previous data.
Conclusion:
Adding a custom XML-based parser may require that you add new meta to your system. Custom meta should be defined in the comments of the XML-based parser which can be read in any text editor. This new custom meta will need to be defined in all appliances that will be making use of it. Such meta will need to be indexed by your Concentrators (and Brokers if you use them) but must not be indexed by your Decoders. Decoders merely tag and store data, indexing is done at the higher tiers by Concentrators and if you use them, Brokers.