Map an IP Address to a Time Zone

This topic discusses NetWitness support for Event Time.

Often times logs do not fully specify timestamps and may be missing time zone information. To properly normalize such timestamps to UTC, the Log Decoder provides the ability to associate devices from a specific address (IPv4 or IPv6) or hostname to a time zone or a fixed offset.

Timestamp consideration for log files:

  • Some logs are in UTC format.
  • Some logs that are not in UTC include a timezone offset value to adjust the time accordingly.
  • For logs that use local time, with no offset provided, you can create a source mapping to manually adjust times for logs from that event source.

Three time zone formats are currently accepted and are shown in the following examples:

  • Olson format: America/Anguilla
  • POSIX format: AST2:45ADT0:45,M4.1.6/1:45,M10.5.6/2:45
  • Offset by Minutes format: = -500

NetWitness maps the device address (IPv4 or IPv6) or hostname to a specific time zone or offset. Event time meta that is parsed from a log that is from a mapped address and does not include an offset or time zone as part of the timestamp is adjusted to UTC according to the mapping.

To map an IP address to a time zone, do the following:

  1. Go to netwitness_adminicon_25x22.png (Admin) > Services.
  2. In the Services view, select a Log Decoder, and in the Actions column, select netwitness_ic-actns2.png> View > Explore.
  3. Go to /decoder/parsers node, right click Parsers, and select Properties.

  4. In the Properties view, specify the iptmzone command with the following parameters:

    op=add entries="ipaddress=timezone"

    for example: op=add entries="10.10.10.10=Africa/Addis Ababa"

  5. Click Send.

tzinfo Command

You can view the strings that can be used to specify the time zone by using the tzinfo command.

To view the time zone strings, do the following:

  1. Go to netwitness_adminicon_25x22.png (Admin) > Services.
  2. In the Services view, select a Log Decoder, and in the Actions column, select netwitness_ic-actns2.png> View > Explore.
  3. Go to /decoder/parsers node, right click Parsers, and select Properties.

  4. In the Properties view, specify the tzinfo command with the following parameters:

    op=tznames

  5. Click Send.

The list of time zone strings is returned in the Response Output text box.

iptmzone Command

In the iptmzone command, three operations are available:

  • add: This operation adds or updates entries in the iptmzone map. Multiple space delimited address/type pairs may be specified.

    op=add entries="address=time zone"

  • remove: This operation removes entries in the iptmzone map. Multiple space delimited address/type pairs may be specified.

    op=remove entries="address"

  • describe: This operation returns the values currently in the iptmzone map.

Examples

The following examples provide instances for mapping IP addresses to time zones:

  • If you want to map two different entries with different IPV4 values and time zone, enter the following parameter in the iptmzone command and click Send

    op=add entries=”10.10.10.10=America/Anguilla 10.10.10.11=Pacific/Rarotonga”

  • If you want to remove an entry for a single IPV4 value and time zone, enter the following parameter in the iptmzone command and click Send.

    op=remove entries="10.5.245.9"

  • If you want to create a single entry for an IPV6 value and time zone, enter the following parameter in the iptmzone command and click Send.

    op=add entries=”2001:DB8:85A3::8A2E:370:7334=America/Anguilla”

  • If you want to create a single entry to map an IPV4, IPV6, or hostname with the Minute Offset, Olson, or POSIX format, enter the following string in the iptmzone command and click Send.

    op=add entries="10.168.0.2=America/Anguilla 2001:DB8:85A3::8A2E:370:7334=0500 nwappliance21=EST5EDT,M3.2.0/2,M11.1.0"

Change the Date format

The Log Decoder parsing engine uses the event.time meta key to keep track of the time an event occurs for incoming log messages. You can use the mdformat for the iptmzone command to change the date format. The Log Decoder currently has the ability to change the dates in the logs to have the following format:

mdy or dmy (month/day/year or day/month/year)

Notes:

  • Event time metadata in a parser does not account for dmy and mdy.
  • Currently, there is no functionality to detect this scenario and adjust parsing automatically.

To change the date format, do the following:

  1. Go to netwitness_adminicon_25x22.png (Admin) > Services.
  2. In the Services view, select a Log Decoder, and in the Actions column, select; netwitness_ic-actns2.png > View > Explore.
  3. Go to /decoder/parsers node, right click Parsers, and select Properties.

  4. In the Properties view, select iptmzone from the drop-down list, and specify the command with the following parameters:

    op=add entries="<ipaddress>" mdformat=[dmy | mdy | none]

    for example: op=add entries="1.1.1.1" mdformat=dmy

    Where:

    • ipaddress is the Device IP address, and
    • mdformat can be dmy, mdy, or none.

    netwitness_eventtimesupport.png

  5. Click Send.

NetWitness maps the IP address along with the date format in the Log Decoder. Event time meta items are updated according to their respective mappings.

Event Time Filter Example

If you want to see logs that have an event time between 3:02 PM and 3:15 PM on May 17, 2019, create the following filter:

netwitness_eventtimesupport_filter.png

Note: The query builder may tell you “invalid expression,” but that is a validation error in the User Interface. The query works because even though this is a text (string) search, the system actually compares the ASCII values. So, for example the expression A < B would be true.

Below is a sample of the event meta values returned from the query:

netwitness_eventtimesupport_events.png

Missing Year Support

If there is no year associated with a format string, the parser attempts to populate the year heuristically. In most cases, the value is populated based on the current year. The current year is assigned as the message year (UTC), as per the clock on the Log Decoder.

The following are the various other scenarios:

  1. Latent log during transition to new year.
  2. Logs from forward time zones (or skewed clocks) just before new year transition.
  3. Latent logs received where a leap day cannot be successfully assigned to an appropriate leap year.

Latent log during transition to new year

If the day is more than 31 days into the future, the year is decremented.

For example,

The message date is Dec-31. The current date is 2018-Jan-1. The temporary assigned date, 2018-Dec-31, will be more than 31 days into the future.

This will cause the year component to be decremented to 2017, resulting in a reasonable message time.

Logs from forward time zones (or skewed clocks) just before new year transition

If the day is more than 334 days into the past, the year is incremented.

For example,

The message date is Jan-1. The current date is 2017-Dec-31. The temporary assigned date, 2017-Jan-1, will be more than 334 days into the past.

This will cause the year component to be incremented to 2018, resulting in a reasonable message time.

Latent logs received where a leap day cannot be successfully assigned to an appropriate leap year

If the day (Feb 29) is invalid (say, the assigned year is NOT a leap year), the relative position to the current time cannot be calculated. To do this, the year is decremented in an attempt get a valid time stamp.

Because the position of the leap day relative to the transition to the new year, along with the expectation that no logs would be received this far into the future, we do not ever make an attempt to increment the year to produce a valid time stamp.

After the year is decremented, the same logic is then followed (refer to statement 1 and 2). If the result is still an invalid day. The parser throws and a valid message time cannot be parsed.

Limitations

Note: Currently, there is no mechanism to assign a year to the import of logs.

As this pertains to leap days, if you find the correct leap year, an inconsistent data would result. For example, if a set of messages is two years old and during a leap year, only a single day would remain which is accurate. Because of the heuristic described above, the remaining set of messages would have time stamps one year too.

For data consistency, usually you cannot find valid leap years beyond 1 month and less than 11 months.

Examples

The following examples provide instances for logs with year and logs without year:

  • If the log is with year, event time is generated as below:

    %emcavamar: 1704^^2017-04-07^^07:50:02^^1^^<event-source NodeID="avamar" ProgramName="com.avamar"/>^^1270626^^SYSTEM^^ERROR^^OK^^MCS:DPN_Proxy^^Internal server error^

    <MESSAGE

    id1="1:01"

    id2="1"

    eventcategory="1605020000"

    functions="&lt;@msg:*PARMVAL($MSG)&gt;&lt;@event_time:*EVNTTIME($MSG,'%W-%G-%F %H:%U:%O',fld2,fld3)&gt;"

    content="&lt;fld1&gt;^^&lt;fld2&gt;^^&lt;fld3&gt;^^&lt;fld4&gt;^^&lt;&lt;event-source NodeID=&quot;&lt;hostname&gt;&quot; ProgramName=&quot;&lt;fld8&gt;&quot;/&gt;^^&lt;fld9&gt;^^&lt;category&gt;^^&lt;severity&gt;^^&lt;event_type&gt;^^&lt;agent&gt;^^&lt;event_description&gt;" />

  • If the log is without year, event time is still generated, The current year is assigned as the message year (UTC), as per the clock on the Log Decoder.

    %emcavamar: 1704^^04-07^^07:50:02^^1^^<event-source NodeID="avamar" ProgramName="com.avamar"/>^^1270626^^SYSTEM^^ERROR^^OK^^MCS:DPN_Proxy^^Internal server error^

    <MESSAGE

    id1="1:01"

    id2="1"

    eventcategory="1605020000"

    functions="&lt;@msg:*PARMVAL($MSG)&gt;&lt;@event_time:*EVNTTIME($MSG,'%G-%F %H:%U:%O',fld2,fld3)&gt;"

    content="&lt;fld1&gt;^^&lt;fld2&gt;^^&lt;fld3&gt;^^&lt;fld4&gt;^^&lt;&lt;event-source NodeID=&quot;&lt;hostname&gt;&quot; ProgramName=&quot;&lt;fld8&gt;&quot;/&gt;^^&lt;fld9&gt;^^&lt;category&gt;^^&lt;severity&gt;^^&lt;event_type&gt;^^&lt;agent&gt;^^&lt;event_description&gt;" />