Create Custom Typespec for File Collection

RSA NetWitness uses type specification (typespec) files for ODBC and file collection. These files act on raw log files, and are used for two main purposes:

  • Define where in the log file data resides. For instance, some log files contain header information that is not considered data to be parsed.
  • Replace certain types of characters that the log parser cannot parse correctly. For instance, the tab character can sometimes cause problems.

This topic tells you how to create a custom typespec for the Log Collector. The topic includes:

  • Create Custom typespec procedure
  • File Collection typespec syntax
  • Sample typespec files

Create Custom Typespec File

To create a custom typespec file:

  1. Open an SFTP client (for example, WinSCP) and connect to a Log Collector or remote Log Collector.
  2. Navigate to /etc/netwitness/ng/logcollection/content/collection/file, and copy an existing file, for example apache.xml.
  3. Modify the file according to your requirements. See File Collection Typespec Syntax for details.
  4. Rename and save the file to the same directory.
  5. Restart the Log Collector.

Note: You will not be able to see new Event Source type in Security Analytics until you restart the Log Collector.

File Collection Typespec Syntax

The following table describes the typespec parameters.

Parameter Description

name

The display name of your File event source (for example, apache). Security Analytics displays this name in the Sources panel of the View > Config > Events Sources tab.

Valid value is an alphanumeric string. You cannot use - (dashes), _(underscores), or spaces . The name must be unique across all typespec files in the folder.

type

Event source type: file. Do not modify this line.

prettyName

User-defined name for the event source. You can use the same value as name (for example, apache) or use a more descriptive name.

version

Version of this typespec file. Default value is 1.0.

author

Person who created the typespec file. Replace author-name with your name.

description

Formal description of the event source. Replace formal-description with your description of the event source.

<device> Section

parser

This optional parameter applies to Security Analytics 10.6.1 and newer. This parameter contains the name of the log parser. This value forces the Log Decoder to use the specified log parser when parsing logs from this event source.

Note: Please leave the field blank when unsure of the log parser to be used.

name

Name of your File event source (for example, apache).

<collection><file> Section

parserId

Reserved for future use.

processorType

Examples of a processor-type are generic, xml, tagvalmap, and oracle. Processor types are similar to handlers in RSA enVision.

Note: If the processorType is XML, the typespec file contains an <eventGroups> section, described below.

dataStartLine

The line number in the log file at which Security Analytics starts collecting events. Default value is 1.

fieldDelim

Specify the delimiter that separates the fields in the log file being parsed. Specify any of the following values:

  • || (piping)
  • ^ (caret)
  • , (comma)
  • : (colon)
  • 0x20 (to represent a space)

idField

Msg ID (message ID) field number. For example, specify 6 to identify the sixth field from the space-delimited event as the Msg ID.

lineDelim

Line delimiter that detects the end of an event. For example, specify \n to provide values for CR and LF.

File collection uses the following tags during event transformation

transformType

Transform type. Example of a transform-type is ias. The Internet Authentication Service (IAS) allows you to parse IAS logs when transformType = ias and ProcessorType = generic.

transformPrefixTag

Inserts the specified prefix in front of the transformed event. For example, if you specify APACHE, Security Analytics inserts %APACHE as the prefix.

transformReplaceFieldDelim

Specifies whether or not to replace the delimiter during transformation. Values:

  • 0 (default): do not replace
  • 1: replace

transformPrefixFilename

Specifies whether or not to add the prefix to the filename during transformation. Values:

  • 0 (default): do not add
  • 1: add

transformMultipleDelimiterAsOne

Specifies whether or not to combine multiple sequential delimiters as one. Values:

  • 0 do not combine
  • 1 (default): combine

transformReplacementFieldDelim

Replace raw field delimiters with the given values with the specified values, if the transformReplaceFieldDelim flag = 1.

Entries in the <eventGroups><eventGroup> section:

  • globalInfo: The globalInfo xpath. Reads parent node information and adds it to each level.
  • eventXPath: xpath of events.

Sample File Collection Typespec Files

The following sample is the typespec file for the CA ACF2 event source.

<?xml version="1.0" encoding="UTF-8"?>
<typespec>

<name>acf2tvm</name>
<type>file</type>
<prettyName>ibmacf2</prettyName>
<version>1.0</version>
<author>administrator</author>
<description>File Collection specification for event source
type "CA ACF2" using file handler type "ACF2TVM"
</description>

<device>
<name>ibmacf2</name>
<parser>ibmacf2</parser>
</device>

<configuration>
</configuration>

<collection>
<file>
<parserId>file.acf2tvm</parserId>
<processorType>generic</processorType>
<dataStartLine>1</dataStartLine>
<fieldDelim>|</fieldDelim>
<idField></idField>
<lineDelim>0x0a</lineDelim>
<transformPrefixTag>ACF2TVM</transformPrefixTag>
<transformReplaceFieldDelim>0</transformReplaceFieldDelim>
<transformPrefixFilename>0</transformPrefixFilename>
<transformMultipleDelimiterAsOne>0</transformMultipleDelimiterAsOne>
<transformReplacementFieldDelim></transformReplacementFieldDelim>
</file>
</collection>
</typespec>

The following sample is the typespec file for the Tripwire Enterprise event source. Note that this file contains an <eventGroups> section (because the <processorType> value is XML).

<?xml version="1.0" encoding="UTF-8"?>
<typespec>
   <name>tripwire</name>
   <type>file</type>
   <prettyName>tripwire</prettyName>
   <version>1.0</version>
   <author>administrator</author>
   <description>FileCollection specification for eventsource type
   "Tripwire Enterprise" using file handler type "tripwire"
   </description>
   <device>
      <name>tripwire</name>
      <parser>tripwire</parser>
   </device>
   <configuration>
   </configuration>
   <collection>
      <file>
         <parserId>file.tripwire</parserId>
         <processorType>xml</processorType>
         <eventXPath></eventXPath>
         <eventGroups>
          <eventGroup>
          <globalInfo> //ReportHead/Report[@type!='systemlog_rpt']/../../ReportBody/ReportSection/@name |
             //ReportHead/Report[@type!='systemlog_rpt']/../../ReportBody/ReportSection/ReportSection/@name |
             //ReportHead/Report[@type!='systemlog_rpt']/../../ReportBody/ReportSection/ReportSection/ReportSection/@name |
             //ReportHead/Report[@type!='systemlog_rpt']/../../ReportBody/ReportSection/String |
             //ReportHead/Report[@type!='systemlog_rpt']/../../ReportBody/ReportSection/ReportSection/String
          </globalInfo>
          <eventXPath>
         //ReportHead/Report[@type!='systemlog_rpt']/../../ReportBody/ReportSection/ReportSection/ReportSection/ReportSection
          </eventXPath>
       </eventGroup>
       <eventGroup>
          <globalInfo></globalInfo>
          <eventXPath>
         //ReportHead/Report[@type='systemlog_rpt']/../../ReportBody/ReportSection/ReportSection/*
          </eventXPath>
       </eventGroup>
        </eventGroups>
        <dataStartLine></dataStartLine>
        <fieldDelim></fieldDelim>
        <idField></idField>
        <lineDelim>\n</lineDelim>
        <transformPrefixTag></transformPrefixTag>
        <transformReplaceFieldDelim>0</transformReplaceFieldDelim>
        <transformPrefixFilename>0</transformPrefixFilename>
        <transformMultipleDelimiterAsOne>0</transformMultipleDelimiterAsOne>
        <transformReplacementFieldDelim></transformReplacementFieldDelim>
      </file>
   </collection>
</typespec>