Configure Decoder Rules

This topic provides procedures for creating and managing rules for Decoder or Log Decoder traffic capture in the Services Config view > Rules tabs . Services Config View - Rules Tabs provides details about the Rules tab options.

The following figure illustrates commonly used settings on a Decoder. For a quick basic setup with only the required steps, see Decoder and Log Decoder Quick Setup.

netwitness_deccfgwf-cfgrul.png

Capture rules can add alerts or contextual information to sessions or logs. They can also define which data is filtered out by a Decoder or Log Decoder. Rules are created for specific metadata patterns, which result in predefined actions when matches are found. For example, to keep all traffic that fits certain criteria, but discard all other traffic, you can create a rule to perform the necessary actions. When applied, rules affect both packet capture file importing, as well as live network capture.

Rule and Query Guidelines provides guidelines that all queries and rule conditions in NetWitness Core Services must follow.

By default, no rules are defined when you first install NetWitness. Until rules are specified, the packets are not filtered. You can deploy the latest rules from Live. You can define three types of rules: Network Rules, Application Rules, and Correlation Rules.

  • Network rules are applied at the packet level and are made up of rule sets from Layer 2, Layer 3, and Layer 4. Multiple rules can be applied to the Decoder. Rules can be applied to multiple layers (for example, when a network rule filters out specific ports for a specific IP address). Network rules are only available on Network Decoders.
  • Application rules are applied at the session level. If the first rule listed is not a match, the Decoder then attempts to match the next rule listed, until a match is found.
  • Correlation rules are applied over a configurable sliding time window. When a match is found, the service creates a new super session that identifies other sessions that match the rule, then creates a session list for analysis.

The two most common uses of rules are:

  • To alert, and thereby create a custom alert meta value, when certain conditions are found.
  • To filter out certain types of traffic that do not add value to the analysis of the data.

Groups of capture rules form rule sets, which you can import and export. This feature enables use of multiple rule sets for various scenarios. You can import the exported rule set, in the form of an .nwr file, to other NetWitness services, simplifying the deployment and configuration of multiple services.

Rule Processing

These are the principles governing capture rule processing:

  • Multiple rules can be applied to the Decoder.
  • Capture rules are executed one after the other, in sequence.
  • Rule processing stops when all rules are processed or after a rule configured to stop rule processing is matched.
  • A default rule can be used to either include or exclude all traffic not otherwise selected by a rule. A default rule, if used, must always be placed at the bottom of the rule list. Otherwise, rule processing stops as soon as the default rule is evaluated since, by definition, all traffic is selected by the default rule.
  • When rule processing stops, the session is saved using the configured session options and debug options.

Rule and Query Guidelines

All queries and rule conditions in NetWitness Core services must follow these guidelines:

All string literals and time stamps must be quoted. Do not quote numbers, MAC, or IP addresses.

  • extension = 'torrent'
  • time='2015-jan-01 00:00:00'
  • service=80
  • ip.src=192.168.0.1

Note: The space on the right and the left of an operator is optional. For example, you can type a rule as service=80 or service = 80 .

Rule Examples

The following table shows examples of rule conditions. You can use rule conditions for log retention collections in an Archiver and for application, network, and correlation rules on a Decoder, Log Decoder, or Concentrator. Rule conditions are also used in all WHERE clauses in all Core database queries.

For detailed information on rule syntax in NetWitness, see "WHERE Clauses" in the "Queries" section of the Core Database Tuning Guide .

Rule Name Condition
ComplianceDevices device.group='PCI Devices' || device.group='HIPPA Devices'
HighValueWindows device.group='Windows Compliance'
MediumValueWindows device.type='winevent_nic' && msg.id='security_4624_security'
LowValueWinLogs device.type='winevent_nic' && msg.id='security_4648_security'
LowValueProxyLogs device.class='proxy' && msg.id='antivirus_license_expired'
GeneralWindows device.type='winevent_nic'

Invalid Rules

NetWitness uses a rule parser that strictly defines valid syntax for rules and queries. When a Core service encounters invalid syntax, it writes a warning in the NetWitness logs indicating the error.

Note: NetWitness Platform does not support parsing of legacy syntax rules. After you update to NetWitness latest version, rules with invalid syntax are highlighted in the user interface, and no rules will be applied until the invalid rules are corrected. The Rule Editor provides additional tooltips. After you fix the rules, the highlights disappear. See Fix Rules with Invalid Syntax.

The /decoder/config/rules/rule.errors and /concentrator/config/rules/rule.errors stats, contain the count of rules with errors. If rule.errors is nonzero, NetWitness generates a Health and Wellness alert to indicate that you need to fix the rules.

General Syntax Guidelines

  • All text values must quote literal values. Example: user = 'user1'
  • Quotes can use single or double quotes; but they must match. (You cannot start with a single quote and finish with a double quote.)
  • If the literal value has a quote, you can escape it using double quotes. Example: user = "User's"

The following are valid syntax rules:

  • All time values should use quotes for dates in this form: time = 'YYYY-MM-DD HH:MM:SS'
  • All time values that are the number of seconds since EPOCH (Jan 1, 1970), should not be quoted.
    Example: time = 1448034064
  • Everything else is unquoted: IP addresses, MAC addresses, numerics, and so on. Example: service = 80 && ip.src=192.168.1.1/16

Capture Rule Syntax

Capture rules compare fields to values or to other fields. This is an example of a simple expression with a meta key on the left side of the operator and a value on the right side.

ip.dst=192.168.1.1

The syntax allows a meta key on the right side of the operator in Decoders and Log Decoders for application and network rules. Meta key comparison does not apply in the where clause in queries. This is an example of a simple expression with a meta key on the left side of the operator and a meta key on the right side.

ip.src=ip.dst

Rules that include a meta key comparison support renamed meta keys; if a rule queries a meta key that has been renamed, the rule is parsed for the renamed meta key. For example, if the meta key ip_dst is used in a rule, it is transparently mapped to the renamed meta key: ip.dst. Existing rules that include original keys will trigger alerts that include data for the renamed meta key.

This is an example of a rule that finds packets having the same ip.src address and ip.dst address on a Decoder, and generates an alert on the Concentrator.

alert=ioc name=testRule8 rule="ip.src=ip.dst" order=38

This rule would generate an error because eth.src and ip.src are incompatible formats.

rule="eth.src=ip.src" name="testRule99" alert=ioc

Values can be expressed as discrete values, a range of values, an upper or lower bound, or a combination of these three. You can create a greater than or less than comparison, and test equality or inequality against a range of values or an upper/lower bound.

key 0-5 (a range of values)

key = 0-u is the same as key >= 0 (upper bound, greater than or equal to)

The following table summarizes the operators on meta keys.

Left Operand Format Operator Right Operand Format Description
any = compatible with left operand Equality operator. You can use values or meta keys on the right side of the equality operator.

any

 != compatible with left operand Inequality operator. You can use values or meta keys on the right side of the inequality operator.
any < compatible with left operand Less than operator. You can use values or meta keys on the right side of this operator.
any <= compatible with left operand Less than or equal to operator. You can use values or meta keys on the right side of this operator.
any > compatible with left operand Greater than operator. You can use values or meta keys on the right side of this operator.
any >= compatible with left operand Greater than or equal to operator. You can use values or meta keys on the right side of this operator.
text contains text Find values that contain the right operand. You can use meta keys or values on the right side of this operator.

text

begins

text

 Find values that begin with the right operand. You can use meta keys or values on the right side of this operator.
text ends text Find values that end with the right operand. You can use meta keys or values on the right side of this operator.

text

length

integer

Find strings of a certain length. You can use meta keys or values on the right side of this operator.

any

count

integer

Find values with a specific number of occurrences within the session. You can use meta keys or values on the right side of this operator.

any

ucount and unique

integer

Finds a number of uniquely occurring values. You can use meta keys or values on the right side of this operator. For example, if the results include instances of a meta key with five unique values and three of the same value, the ucount is six.
N/A exists any Finds any values for the meta key. You can use meta keys or values on the right side of this operator.
N/A !exists any Finds any sessions in which the meta key does not occur. You can use meta keys or values on the right side of this operator.

text

regex

text

Finds values matching a regular expression. You can use values on the right side of this operator.

The following table summarizes other syntax elements used in rules.

Syntax element Description

*

Default rule. By using an asterisk (*) as the sole character in a rule, that rule will select all traffic.
u Upper bound of a range of times, IP addresses, or numeric formats. For example, to select all TCP ports above 40000, the syntax would be:
tcp.port = 40000-u
l Lower bound of a range of times, IP addresses, or numeric values. For example, to select all TCP ports below 40000, the syntax would be:
tcp.port = l-40000
- (dash) Denotes a range. This is only applicable to time values, IP or MAC addresses, or numeric values. Separate the lower and upper bounds of the range with a dash (-) character. For example, to select TCP ports between 25 and 443, the syntax would be:
tcp.port = 25-443
, (comma) Denotes a list of ranges or values or meta keys. Single values may be used as well as any combination of ranges and upper or lower bounds. Single meta keys may be used in a list. Meta keys and literal values cannot both appear on the right-hand side of an operator. For example, the following is valid syntax:
tcp.port = 1-10,25,110,143-225,40000-u
( ) Grouping operator. An expression can be enclosed in parentheses to create a new logical expression. For example, the following would select traffic on port 80 to/from 192.168.1.1 OR traffic on port 443 to/from 10.10.10.1:
(alias.ip=192.168.1.1 && tcp.port=80) || (alias.ip=10.10.10.1 && tcp.port=443)

~

Logical NOT operator, a negation of an expression.
&& Logical AND operator, a conjunction of two expressions.
|| Logical OR operator, a disjunction of two expressions.

Configure Capture Rules

The Decoder and Log Decoder rules are editable in the Services Config view. While each type of rule (network, application, and correlation) has its own tab; the functions are similar for all types of rules. You can:

  • Add, edit, and delete rules
  • Enable and disable rules
  • Change the execution sequence of rules
  • Import rules from a file
  • Export rules to a file
  • Push rules to another service
  • Revert or apply rule changes
  • Restore one of the last ten rule configurations from a snapshot

To configure rules in the Rules tabs

  1. Go to netwitness_adminicon_25x22.png (Admin) > Services.
  2. In the Services view, select a Decoder service and netwitness_ic-actns.png​ > View > Config​.
  3. In the Services Config view, select one of the Rules tabs: Network Rules, App Rules, or Correlation Rules.
    The rules list for the selected rule type is displayed.
    netwitness_12.1_apprule52_1122.png

Each type of rule has a list with slightly different columns and different parameters. Several basic guidelines apply to all rule management activities:

  • The rules are executed in the sequence they are displayed in the list. To change the execution sequence of rules, drag and drop rules to the appropriate location in the list or use the context menu options to arrange the rules in the list.
  • To select a single row, click the row.
  • To select a group of adjacent rows, click the first, then shift-click the row at the end of the group.
  • To select multiple non-adjacent rows, click the first, then control-click the others.
  • When editing rules in the Rules tab, you must apply the configuration changes in order to activate.
  • Until changes are applied, you can discard edits to the list and revert to the unedited rules.
  • Once rules are applied, you can recover the last ten rules configurations using the History option in the Actions menu.

To add a rule in any Rules tab, do one of the following:

  • Click netwitness_ic-add.png.
  • Right-click a rule, and select Insert Above or Insert Below from the context menu.
    The Rule Editor dialog for that type of rule is displayed.

To remove a rule:

  1. From any Rules tab, select the rules to remove from the rules list.
  2. Click netwitness_ic-delete.png.
    The selected rules are removed from the list, but still exist on the service.

To edit a rule

  1. From any Rules tab, select the rule to edit.
  2. Click netwitness_ic-edit.png or double-click the rule row.
    The Rule Editor dialog for that type of rule is displayed.

To disable a rule:

  1. From any Rules tab, select the rules to disable.
  2. Click netwitness_ic-disable.png.
    The status changes to disabled in the rules list, but the rule is still enabled on the service.

To enable a rule:

  1. From any Rules tab, select the rules to enable.
  2. Click netwitness_ic-enable.png.
    The status changes to enabled in the rules list, but the rule is still disabled on the service.

Import Rules from a File and Export Rules

You can import network, application, and correlation rules to a Decoder from a file that contains rules of the same type. After the rules are imported, you can edit and manage them as you would any other rules.

When you attempt to import a group of rules, NetWitness Administration checks the type of rules imported. If you are successful, a message displays the number of rules imported. If the rule type differs from the active tab type, the rules are not imported. You must re-import the rules under the correct tab or select another file to import.

To import rules to a service:

  1. From any Rules tab, select netwitness_ic-actionsbolt.png> netwitness_ic-import2.png.
    ​The Import dialog is displayed.
    netwitness_impruldg_400x375.png
  2. Click netwitness_ic-add.png.
    A view of the directory structure is displayed.
  3. Choose one or more NetWitness rules (.nwr) files to import, and click Open.
    The file is added to the list in the Import dialog.
    netwitness_104importrulefile.png
  4. Click Import.
    The rules are imported into the user interface. Imported rules have a red corner in each edited column.
  5. Edit or reorder the rules if needed.
  6. To save the rules to the service, click Apply.
    The rules for the service are updated with the changes.

To export a rule to a file:

  1. To export a subset of the rules, select the rules to be exported.
  2. Do one of the following:
    • In the toolbar, select netwitness_icon_actionsmenu-rule.png > Export > Selection. (Export > All exports all rules in the rules list even if you have a subset selected for export.)
    • Right-click the selected rules and select Export Selection.

A prompt for the filename is displayed.

  1. Enter the filename and click Export.
    The .nwr file is downloaded.

Push Rules to Other Services

You can apply (push) rules or selected rules to other services (Decoders or Log Decoders) or service groups. When you push all rules to other services, all rules on the target services are removed and replaced with all of the rules on the source service.

To push selected rules from this Decoder to other Decoders:

  1. From any Rules tab, select the rules that you want to push to another Decoder.
  2. Do one of the following:
    • Select netwitness_icon_actionsmenu-rule.png > Push > Selection.
    • Right-click the selected rules and select Push Selected Rules.
      The Push Selected Rules dialog is displayed.
      netwitness_pshselruldg.png
  3. Select a Push Option:
    • Select Replace All to delete all rules on the target services and replace them with the selected rules. This is the default selection.
    • Select Merge to merge the selected rules with the existing rules on the target services.
  4. On the Services tab, select the target services to receive the pushed rules, or select the groups of services from the Groups tab.
  5. Click Push.
    The rules are pushed to the selected services and become effective immediately.

To push all rules from this Decoder to other Decoders:

  1. From any Rules tab, select netwitness_icon_actionsmenu-rule.png > Push > All.
    (Push > All pushes all rules in the rules list even if you have a subset selected to push.) The Push Selected Rules dialog is displayed.
    netwitness_pshallruldg_400x401.png
  2. On the Services tab, select the target services to receive the pushed rules, or select the groups of services from the Groups tab.
  3. Click Push.
    All rules from the target services are deleted and replaced with all of the rules from source service. The rules become effective immediately.

Change Execution Order of Rules

Capture rules are applied in the order they are displayed in the rules list. To reorder rules, use either of these methods:

  • Drag and drop the rules in the appropriate location in the rules list.
  • Right-click a rule to display the context menu, and use the Cut and Paste options.

Restore a Rule Snapshot from History

NetWitness keeps the last ten snapshots of rules applied to a service.

To restore a rules snapshot from history:

  1. Select netwitness_icon_actionsmenu-rule.png > History.
    A submenu of snapshots is displayed.
    netwitness_historymenu.png
  2. Select the snapshot time from the submenu.
    The rules from the snapshot are loaded into the rules list, replacing the current set. But the current set is still in use on the service.
  3. To apply the rules to the service, click Apply.
    The rules are applied to the service.