Create Custom Typespec for ODBC Collection

NetWitness uses type specification (typespec) files for ODBC and file collection. These files act on raw log files, and are used for two main purposes:

  • Define where in the log file data resides. For instance, some log files contain header information that is not considered data to be parsed.
  • Replace certain types of characters that the log parser cannot parse correctly. For instance, the tab character can sometimes cause problems.

This topic tells you how to create a custom typespec for the Log Collector. The topic includes:

  • Create Custom typespec procedure
  • ODBC Collection typespec syntax
  • Sample ODBC Collection typespec files

Create Custom Typespec

To create a custom typespec file:

  1. Open an SFTP client (for example, WinSCP) and connect to a Log Collector or remote Log Collector.
  2. Navigate to /etc/netwitness/ng/logcollection/content/collection/odbc, and copy an existing file, for example bit9.xml.
  3. Modify the file according to your requirements. See ODBC Collection Typespec Syntax for details.
  4. Rename and save the file to the same directory.
  5. Restart the Log Collector.

Note: You will not be able to see new Event Source type in NetWitness until you restart the Log Collector.

ODBC Collection Typespec Syntax

The following table describes the typespec parameters.

Parameter Description

name

The display name of your ODBC event source (for example, activeidentity). NetWitness displays this name in the Sources panel of the View > Config > Events Sources tab.

Valid value is an alphanumeric string. You cannot use - (dashes), _(underscores), or spaces. The name must be unique across all typespec files in the folder.

type

Event source type: odbc. Do not modify this line.

prettyName

User-defined name for the event source. You can use the same value as name (for example, apache) or use a more descriptive name.

version

Version of this typespec file. Default value is 1.0.

author

Person who created the typespec file. Replace author-name with your name.

description

Formal description of the event source. Replace formal-description with your description of the event source.

<device> Section

parser

This optional parameter contains the name of the log parser. This value forces the Log Decoder to use the specified log parser when parsing logs from this event source.

Note: Please leave the field blank when unsure of the log parser to be used.

name

Name your ODBC event source (for example, ActivIdentity ActivCard AAA Server).

maxVersion

The version number of the event source (for example, 6.4.1).

description

Description of the event source.

<collection> Section

odbc

The syntax under <odbc> is used for event collection and processing. You can provide multiple queries for the same event source type by adding <query> tags.

query

This section contains the details of the query used to collect information from the event source.

tag

The prefix tag you want to add to events during transformation (for example ActivIdentity).

outputDelimiter

Specify the delimiter to use to separate fields. Specify any of the following values:

  • || (piping)
  • ^ (caret)
  • , (comma)
  • : (colon)
  • 0x20 (to represent a space)

interval

Specify the number of seconds between events. Default value is 60.

dataQuery

Specify the query to fetch data from the ODBC eventsource database for SQL-syntax. For example:

SELECT acceptedrejected, servername, serveripa, sdate, millisecond, suid, groupname, ipa, reason, info1, info2, threadid FROM A_AHLOG WHERE sdate > '%TRACKING%' ORDER BY sdate

maxTrackingQuery

The query used on the initial pull of events to identify the starting point within the data set to begin pulling logs from. After the initial pull, this query is no longer used, unless the maxTracking value has been reset or altered. For example:

SELECT MAX(Event_Id) from ExEvents

trackingColumn

The tracking column value used when the ODBC collector pulls a new set of events.

Sample ODBC Collection Typespec Files

The following sample is the typespec file for the IBM ISS SiteProtector event source.

<?xml version="1.0" encoding="UTF-8"?>
<typespec>

   <name>siteprotector4_x</name>
   <type>odbc</type>
   <prettyName>SITEPROTECTOR4_X</prettyName>
   <version>1.0</version>
   <author>Administrator</author>
   <description>Collects events from SiteProtector</description>

   <device>
      <name>Internet Security Systems, Inc. RealSecure SiteProtector v 2.0</name>
      <maxVersion>2.0</maxVersion>
      <description></description>
      <parser>iss</parser>
   </device>

   <configuration>
   </configuration>

   <collection>
      <odbc>
         <query>
            <tag></tag>
            <outputDelimiter></outputDelimiter>
            <interval></interval>
            <dataQuery></dataQuery>
            <maxTrackingQuery></maxTrackingQuery>
            <trackingColumn></trackingColumn>
            <levelColumn></levelColumn>
            <eventIdColumn></eventIdColumn>
            <addressColumn></addressColumn>
         </query>
      </odbc>
   </collection>
</typespec>

The following sample is the typespec file for the Bit9 Security Platform event source.

<?xml version="1.0" encoding="UTF-8"?>
<typespec>

   <name>bit9</name>
   <type>odbc</type>
   <prettyName>BIT9</prettyName>
   <version>1.0</version>
   <author>Administrator</author>
   <description>Bit9 Events</description>
	
   <device>
      <name>Bit9</name>
      <parser>bit9</parser>
   </device>

   <configuration>
   </configuration>
				
   <collection>
      <odbc>
         <query>
            <tag>BIT9</tag>
            <outputDelimiter>||</outputDelimiter>
            <interval>10</interval>
            <dataQuery>
            SELECT
            Timestamp,
            Event_Id,
            Computer_Id,
            File_Catalog_Id,
            Root_File_Catalog_Id,
            Priority,
            Type,
            Subtype,
            IP_Address,
            User_Name,
            Process,
            Description
            FROM
            ExEvents
            WHERE
            Event_Id > '%TRACKING%'
            </dataQuery>
            <trackingColumn>Event_Id</trackingColumn>
            <maxTrackingQuery>SELECT MAX(Event_Id) from ExEvents</maxTrackingQuery>
            <eventIdColumn></eventIdColumn>
         </query>
      </odbc>
   </collection>
</typespec>