NetWitness Community

Yes

Decoder Snort DetectionDecoder Snort Detection

Snort is a free and open-source intrusion prevention system that uses a rule-based language to detect malicious network traffic. The NetWitness Platform Decoder offers compatibility with Snort detection rules, sometimes referred to as Snort signatures. NetWitness supports importing existing Snort rules into the Decoder, extending the known threats that can be detected. There are many sources for Snort rules, as it is an open source solution. In most cases, the behavior of Snort rules in the Decoder is the same as the rules in other network analysis tools. However, the Decoder does not have coverage for all Snort rule options. The topics in this section highlight NetWitness Platform Decoder-specific details of how to set up Snort parsers, how Snort rules are handled, and explain cases where Decoder might differ in behavior from Snort.

Using Snort Rules in Decoders Using Snort Rules in Decoders

The Snort Detection engine is implemented as a built-in native parser in Decoders. The parser's identifier is Snort, and can be found in the Decoder Parsers Configuration page along with the other native parsers enabled by default. The Snort parser reads files from the directory /etc/netwitness/ng/parsers/snort. All files in this directory are read during the Decoder service restart. Files that end in the extension .conf are treated as configuration files. The format for configuration files matches the configuration file format for Snort, although not all configuration directives are supported. Further details are in Configuration Directives . Files that end in the extension .rules are treated as Snort rules files. A rules file can contain one or more rules, and the Snort directory can contain more than one rules file. The rule file format is the same as for Snort itself. The Snort rule files can be loaded into the Decoder by any of the following methods:

The Decoder service starts.
The reload message is sent to the /decoder/parsers folder. This allows for rules to be reloaded while capture is running.

However, note the following:

Any rule that does not properly parse is ignored.
Any valid Snort rule should successfully parse; however, there are rule options that are not supported by Decoders which are not fully parsed. For more information, see Snort Parser Capabilities .

Note: NetWitness recommends that the maximum number of Snort rules in use per Decoder is 1,000.

Initial Decoder Snort Configuration Initial Decoder Snort Configuration

Follow these steps to set up an initial configuration of Snort on a Decoder.

Edit the snort.conf file in the /etc/netwitness/ng/parsers/snort directory, and configure it according to the instructions in Configuration Directives , which describes the details of the variables inside the configuration file. If the variables HOME_NET, EXTERNAL_NET and HTTP_PORTS are not defined, they default to the value of any.
A sample snort.conf file is shown below, but depending on where the Decoder is placed in the network, and which threats need to be captured, you might need to alter the configuration.
For example, you can be more specific about the internal subnets by defining ipvar HOME_NET [192.168.0.0/16, 10.0.0.0/8, 172.16.0.0/12]. However, if the Decoder is capturing traffic on a DMZ, those subnets might not be accurate.
Also, if HOME_NET has been defined, and the EXTERNAL_NET has also been narrowed down to ipvar EXTERNAL_NET !$HOME_NET, then you may miss lateral movement traffic (for example, 192.168/16 to 192.168/16) within your environment. In general we suggest you align this configuration file with how you configure the traffic_flow configuration for the corresponding Traffic Flow Lua parser.

An example snort.conf file that is a good starting point:
Generate or upload a Snort rules file into the same directory (/etc/netwitness/ng/parsers/snort). Read the Snort documentation at https://www.snort.org/#documents for details on how to construct rules. There are many free community-generated Snort rules available as well, that can be uploaded from https://www.snort.org/downloads/#rule-downloads. Several other sites publish Snort signatures as responses to new threats that are found. The basic rule structure and supported sections are mentioned in Rule Sections.

Note: Only the Snort V2.x rules are supported. There are some rule options that are not supported, which are described in Snort Parser Capabilities .

Make sure the Snort configuration and rules files do not cause any errors when initialized by the Decoder by running the service restart command on the Decoder.
The reload will check in the Snort rules without restarting, but does not log any messages about whether there are any issues with the rules or configuration.
1. In the NetWitness user interface, go to (Admin) > Services > Decoder > > View > System.
2. Select Shutdown Service and confirm. (The Decoder will automatically restart.)
3. Go to > Services > Decoder > > View > Logs.
4. Search for Snort to validate that the rules files loaded.
  The log will indicate if the rules files are loaded completely (full), partially (partial), or if any rules railed to load (failed).
  The following are example log entries, using mysnort.rules, with two fully supported rules in the file. If any of the rules are marked partial or failure, examine the rules to determine why the load was not successful.
  Loaded mysnort.rules, full 2, partial 0, failures 0
  Loaded 2 snort rules, 2 small tokens, 0 with pcres, 0 partial, 0 unsupported
  When you search for Snort, you can find details on configuration variables and warning messages if they are not defined. The following is an example log entry for a snort.conf file that is not fully populated with the minimal IP and port variables.
  Undefined snort ip variables (will default to any): EXTERNAL_NET, HOME_NET
  Undefined snort port variables (will default to any): HTTPS_PORTS
5. Alternatively, you can perform the service restart and review of the logs from the Decoder console:
  systemctl restart nwdecoder
  tail -f /var/log/messages | grep -i snort
Import or capture network traffic that is known to trigger the Snort rules to validate they are functioning.
Validate that the Snort rules are getting triggered by the network traffic or by sample uploaded PCAPs. The Snort statistics can be used to track which rules have been evaluated, compared to generated metadata. For more information, see Rule Statistics . The default metadata generated by Snort rules are: sig.id, sig.name, threat.category, threat.source, and risk.num. The mapping of Snort rule information to NetWitness Platform metadata, and how to change it, is described in Snort Parser Output.

Snort Parser OutputSnort Parser Output

When a Snort rule matches a session, it produces meta items. The meta items generated may change depending on the configuration of the Snort parser. Snort parser meta key usage has been updated with a new option for the Snort parser. The option, Snort="udm=true" (set to true by default), uses the Aligned Unified Data Model (UDM) key set. For information about UDM, see https://community.netwitness.com/t5/netwitness-platform-unified-data/introduction-to-the-rsa-netwitness-platform-unified-data-model/ta-p/565620. By default, the Aligned UDM key set is used.

Note: To pass options to parsers, you must first give the name of the parser and then the options to be passed in this format: <ParserName>="<ParserOptions>"<Whitespace><ParserName2>="<Parser2Options>"
Each ParserName=Value option must be separated by whitespace. Normally, the Value must have double quotes around it. The Value itself can sometimes list multiple Option=Value pairs, each separated by whitespace, and if those values have whitespace, they must be in escaped double quotes. To escape a quote, place a backslash before it: \".
This is an example of defining options for Parser1, Parser2, and Parser3:
Parser1="Option1=\"Option1 Value With Space\" Option2=Option2ValueNoSpace" Parser2="Option1=Value" Parser3="op1=val1 op2=val2 op3=\"another value\""

The following keys are generated in the Aligned Unified Data Mode:

sig.id with the value of the rule's sid field
threat.category with the value of the rule's classtype field
sig.name with the value of the rule's msg field
risk.num with the value of the rule's priority

The following keys are generated in Legacy Key Mode:

alert.id with the value of rule's sid field
threat.category with the value of the rule's classtype field
risk.info, risk.suspicious, or risk.warning depending on the priority of the rule. The value of the risk meta is the rule's msg field.

If you want to use the legacy key set which contains keys that are consistent with previous releases for Snort parser meta keys, follow these steps.

In the NetWitness User Interface, go to ADMIN > Services.
Select a Decoder and then click > View > Explore.
In the left panel, select decoder > parsers > config.
In the right panel, in parser.options, add Snort="udm=false".

Rule Statistics Rule Statistics

The Snort parser maintains counters to monitor the activity of each rule. These counters can be retrieved using the running snrtStat on the Decoder's /decoder/parsers node, which can be accessed through the NwConsole utility or the REST interface. The statistics reported for each rule are:

evaluations: This counts the number of times the rule was evaluated. A rule can be evaluated once for each stream in a session. Typically a rule will only match one side of a session, and will be evaluated, at most, once. The content strings determine if a rule will be evaluated in a session. A content pattern must match somewhere in the session in order for it to be evaluated.
hits: This counts the number of times a rule matched and the corresponding alert meta item was created.

To review the Snort statistics when using the Decoder console:

Log in to the console locally or remotely
Type the command NwConsole
Log in to localhost:50004 <admin> <netwitness> (change login credentials accordingly)
Enter /decoder/parsers snrtStat

The output is a JSON structure listing the number of evaluations and hits performed, along with the options evaluated, for each Snort ID.

Configuration Directives Configuration Directives

Decoder supports a limited number of configuration directives.

Variable Definitions Variable Definitions

Decoder supports definitions of variables that hold the value of IP ranges using ipvar or Port ranges using portvar. For example, Snort rules usually make use of HOME_NET, EXTERNAL_NET and HTTP_PORTS variables. These have to be defined in your configuration file. If they are undefined, they default to the special value any.

ruletype ruletype

The definition of additional rule types is supported. However, only rules that have a base rule type of alert are supported. Rule type definitions follow the same definition format as Snort configuration.

Other Configuration Entries Other Configuration Entries

config detection: debug

Adding this configuration line turns on debug messages that are generated as the rule is being processed. These debug messages are generally emitted only emitted when a rule matches a session by one of the content patterns, but then fails to match other options within the rule. This is useful for debugging complex filters in Snort rules, to determine why they may not be matching as expected.

config nopcre

Adding this configuration line turns off the PCRE (regular expression) capability.

config classification

Classification configuration influences how meta is generated when a rule matches a session. Meta output from Snort rule matches is described in Snort Parser Output.

Snort Parser Capabilities Snort Parser Capabilities

The general format of a Snort rule is that it contains a rule header and rule options. The rule header consists of the rule action, protocol type, source-destination criteria (IP addresses, ports), and the direction of traffic. The rule options consist of the message (msg) which describe what the rule has detected, the references related to the threat or where the rule was generated, the classification of the rule, the unique signature (rule) identifier, and a long list of attributes (for example, flow, content, pcre) to define how the traffic is identified. The specific details on which header and rule options are supported are outlined in the remainder of this section.

Rule SectionsRule Sections

Section	Description
Header	The header conditions are evaluated when a rule receives the first token callback for a stream. The header is evaluated once per stream, and prevents any further consideration of a rule against a specific stream if the conditions are not met.
Actions	The specified action or a rule must be defined (either one of the native Snort actions, or defined in the configuration using the ruletype statement) for the rule to be considered valid. The Decoder only uses rules with alert actions.
Protocols	The Decoder supports the current Snort protocol keywords (tcp, udp, icmp, ip).
IP Addresses	The full language for defining IP addresses is supported, including lists, CIDR, and negation.
Port Numbers	The full language for defining port numbers is supported, including lists, ranges and negation.
Direction Operator	The directional operator supports the from-to (‘->’) and bidirectional (‘<>’) values. The to-from (‘<-’) value is invalid and causes the rule to fail to load.

Content Content

The Snort content statement makes up the bulk of the Snort pattern detection capability. For best performance, Decoder strives to map content patterns to the Token Parser used by most NetWitness parsers. Therefore, the content strings are the main mechanism to activate the rule matching engine on a session for a particular rule. If the rule does not have any content patterns, it is effectively unsupported. The Snort parser allows for very short strings patterns in content, but be aware that a rule that contains only very short content may have to be evaluated for every session, which will make the Snort Engine run much more slowly.

The Snort parser supports most forms of content statements, including the following modifiers. All modifiers effect the last content pattern declared in the rule.

Option	Description
nocase	Case-insensitive pattern matching is used.
offset	This option is applied to the distance of the token from the beginning of the packet. If the number of bytes between the offset point and the start of the token is less than this value, it is not a match.
depth	This option is applied to the distance of the token from the current offset position, or the beginning of the packet if no offset is set. If the number of bytes between the 'offset' point and the beginning of the token is greater than this value, it is not a match.
distance	This option is applied to the distance of the token from the current Detection Offset End (DOE) point. If the number of bytes between the current token and the DOE is less than this value, the token is not a match.
within	This option is applied to the distance of the token from the current Detection Offset End (DOE) point. If the number of bytes between the current token and the DOE is greater than this value, the token is not a match.

These content modifiers are supported if the Snort parser is used in conjuction with AppHTTP native parser:

Option	Description
http_client_body	The token must appear in the body of an HTTP request.
http_cookie	The token must appear in an HTTP cookie header.
http_header	The token must appear any where in in an HTTP request or response header.
http_method	The token must appear in the HTTP method field. The detection engine does not currently restrict the method tokens, such as GET or POST, to the HTTP method field, so these matches will work even if AppHTTP is turned off.
http_uri	The token must appear in the normalized HTTP header. Non-normalized header matches are not supported.

Further detail on the HTTP related modifiers is described below in HTTP Parsing .

FlowFlow

The flow directive verifies that the rule is only applied to the client or server stream. It supports the following options.

Option	Description
to_client	Limits the rule to only matching on a stream that a Decoder has defined as Server.
from_server	Synonym for to_client.
from_client	Limits the rule to only matching on a stream that a Decoder has defined as Client.

Fast Pattern Fast Pattern

The Snort parser uses content tagged with fast_pattern to indicate tokens that should always be used as tokens in the Decoder's token scanner. Decoder supports the fast_pattern and fast_pattern:only directives, but it does not support the fast_pattern:offset,length directive.

PCRE PCRE

The Snort parser implements pcre regex matching using the same libpcre library as other tools. It supports most of the flags documented for the pcre tag, with a few minor exceptions as described later in this document.

HTTP Parsing HTTP Parsing

The Snort parser relies on the native HTTP parser to notify it when HTTP is being parsed. The HTTP parser feeds information to the Snort engine to let it know which regions of the session should be used to evaluate the http content flags. The pcre implementation has option flags analogous to the http content options, and these are supported as well. If the native HTTP parser is disabled, then these features are also disabled in the Snort parser.

File DataFile Data

The Snort parser supports the file_data tag for HTTP response bodies only. This functionality relies on the native HTTP parser being enabled. If the Native HTTP parser has the decompression feature turned on, the Snort parser also uses decompressed responses to implement rule matching inside of file_data regions. You can enable the HTTP decompression feature by adding the token HTTP="decompress=true" to the configuration field /decoder/config/parsers.options.

The Decoder has a Javascript normalization parser that functions similarly to the normalize_javascript feature in Snort's HTTP preprocessor. It also affects data presented to Snort rules using file_data. To enable the Javascript normalization parser, add the token JSNormalize="unescape=true" to the configuration field /decoder/config/parsers.options.

The Snort parser emulates the file_data behavior in the same way as Snort. Directives that appear after file_data are implicitly evaluated whenever a content match is found inside of a file_data region. Because of this, file_data has side effects on other statements that appear after it in a rule.

Binary (byte) Directives Binary (byte) Directives

The Snort parser supports the byte_test, byte_extract and byte_jump directives.

The detection engine supports most options on these directives:

The options for extracting the value, such as the offset, the little endian / big endian options, and relative options are supported. The relative flag causes the extraction of the byte value to occur relative to the DOE (Detection Offset End) pointer.
byte_jump and byte_extract support the align and multiplier options to manipulate the extracted value.

The 'byte_extract directive stores values in variable names that can be used to substitute a number into a few specific locations elsewhere in the rule. These locations are supported:

The value field in a subsequent byte_test
The value of a within statement on a subsequent content pattern
The value of a distance statement on a subsequent content declaration.

The byte_jump option explicitly moves the DOE (Detection Offset End) pointer for use by subsequent directives, like byte_test, content marked with distance or within, or PCRE patterns marked as relative.

There are some less common byte operations that are not supported:

The byte options do not support the string option for converting text into a binary value
The byte options do not support the bitmask option.
The byte options do not support the dce option.

Default option / fallback Default option / fallback

When the Snort parser encounters rule options it does not yet support, it skips those options and assumes they did not prevent the rule from matching. In such cases it relies on the implementation of options it does support, such as the content or pcre statements, to determine if the rule matches. This behavior means that the Snort parser is more likely to fail in the false-positive direction rather than the false-negative direction when using features that Decoder does not yet support.

Packet Scanning/Scoping Packet Scanning/Scoping

Depending on how the rule is formed, it may be intended to match content entirely within single packets, or it may be intended to match data that spans multiple packets. The Snort parser attempts to handle both cases. Rule statements that are implicitly packet-scoped may be evaluated for every packet available in a session at parse time, and the Snort parser correctly scopes such rules so they only match if the elements of the rule match in a single packet. Conversely, if the rule uses a feature that implies reassembly and reconstruction, such as file_data, the Snort parser will allow the rule to be evaluated for the reconstructed portion of the stream that the rule references. This allows the Snort parser to emulate the various buffers that the Snort product provides.

Known Limitations for Snort Parser Known Limitations for Snort Parser

Decoder Token Match Limits Decoder Token Match Limits

The Snort Parser is activated by token scanning in Decoder, and therefore is subject to the same limitations on matches as all Decoder parsers. Configuration items such as parse.bytes.min, parse.bytes.max, timeouts, and size limits all apply to the Snort parser. The Decoder configuration guide describes these parameters in great detail. The Snort parser is implemented on top of reassembled sessions only so that features that require an assembled session can be supported.

Flow Bits Flow Bits

Flow bit directives, which start with the keyword flowbits, are currently mostly ignored by the Snort detection engine. Statements in rules that test flow bits currently always pass. However, the detection engine will honor flowbit directives that specify noalert. Rules marked as noalert never generate meta, even if they match.

DCE Preprocessor DCE Preprocessor

The Snort parser does not currently implement any DCE preprocessors, so any features that rely on the DCE preprocessor are not implemented.

Thresholds and Filters Thresholds and Filters

The Snort parser implements the threshold filters option only. It does not yet implement the detection_filter filter.

Raw Byte Scanning Raw Byte Scanning

The Snort parser does not currently allow rules to individually bypass the normalization or decoding using tags like rawbytes.

IPv6 IPv6

The Snort rule parser does not currently allow rules to utilize IPv6 addresses as part of their header.

Note: Only the Snort V2.x rules are supported.

Performance Considerations Performance Considerations

The performance of the Snort parser is dependent entirely on the rules that are being evaluated. Therefore it will vary greatly from one installation to another.

Content Tokens Content Tokens

Content tokens form the main entry point into the Snort engine. Hence they are the main factor affecting rules performance. The more times that a content string is found in the packet stream, the more work the Snort parser performs, and the more CPU time it uses. Therefore, it's important to use content tokens that are long, unique, and unlikely to appear in random data. Short tokens will spuriously activate the Snort parser, wasting CPU time.

Negation Negation

Statements like content and pcre support negation, meaning they only allow the rule to match if a content string is not found or if a regex doesn't match. These are usually detrimental to performance because they still tend to contain strings that will activate rule evaluation, yet will often fail since they are intended to filter out patterns. For example, a negated pcre search still has to be evaluated if it is negated, and if it is used to filter out common text. it will produce lots of CPU activity on traffic that does not actually match the Snort rule.