NetWitness Community

2 Likes

Traffic Flow Lua Parser Traffic Flow Lua Parser

The Decoder identifies the host which initiated a session as ip.src, and the responding host as ip.dst. However, there is no indication which hosts are internal to your network and which are external to your network. So, there is no indication whether a session was initiated by an internal host to an external host, whether a session was initiated by a external host, or whether a session was between two internal hosts. Nor, for internal hosts, is there any indication where on your network an internal host resides. The traffic flow parser provides this directionality information.

The netblock name (netname meta) provides the context of where on your network a host resides. Directionality (direction meta) provides the context of whether a session was initiated from an internal host to an external host (outbound), from an external host to an internal host (inbound), or was between two internal hosts (lateral).

You download the parser from Live to deploy to a network decoder, in the same manner as you download and deploy all of the RSA parsers. In addition to the parser, there is an options file. You only need the options file if the default settings are not sufficient for your use case. At this time, only manual deployment to a Log Decoder is supported. The screenshot below shows the Investigation view of these two pieces of meta being populated.

Setup

The Lua parser and options file are supported on both the packet and log decoder, but will only deploy through Live to the network decoder. Deployment will silently fail for the Log Decoder; if you want the Traffic Flow parser installed on the Log Decoder, you must deploy manually.

The parser and options file are separate packages in Live. This means that any options file customizations you make do not get overwritten if you update the parser itself. Thus, we recommend not subscribing to the options file in Live.

If you need to edit any options, you should deploy the Options file at the same time you deploy the parser itself. Note, however, that if the default settings cover your network to your satisfaction, then you do not need the Options file.

Deploy to Network DecodersDeploy to Network Decoders

To deploy the parser to a Network Decoder:

In the NetWitness UI, select Live > Search.
In the Search Criteria section, type "traffic flow" for Keywords.
In the Matching Resources pane, select both files returned: traffic_flow_options and traffic_flow.
Click Deploy.

This launches the Deployment Wizard.
Use the Deployment Wizard to deploy to a network decoder.
1. The two files, traffic_flow_options and traffic_flow, are shown in the Resources screen. Click Next to proceed.
2. In the Services screen, select a decoder, and click Next.
3. In the Review screen, click Deploy.
4. Once the two files deploy, click Close.
You can navigate to the General tab in the Decoder and view the parser in the Parsers Configuration pane:

Deploy to Log Decoders (For versions Prior to 11.0)

For RSA NetWitness Platform versions 11.0 and later, you can deploy to log decoders using Live, in a similar manner as described in Deploy to Network Decoders above.

For earlier versions of RSA NetWitness, perform the following steps to install the Traffic Flow Lua parser on Log Decoders.

Warning

The direction meta is a string; it only contains the last value written to it. If a log parser writes direction meta and the Lua parser is deployed to a log decoder, then that existing direction meta may be overwritten by the parser.

To deploy the parser manually on versions earlier than RSA NetWitness Platform 11.0:

Download the Traffic Flow Parser and Traffic Flow Options File from Live.
1. In the NetWitness UI, select Live > Search.
2. In the Search Criteria section, type "traffic flow" for Keywords.
3. In the Matching Resources pane, select both files returned: traffic_flow_options and traffic_flow.
4. Select Create from the Package menu.
  
  A resource bundle ZIP file is downloaded to your computer.
Unzip the file that you downloaded. In the download path, there are two ZIP files, traffic_flow_options.zip and traffic_flow.zip.
Unzip these files: you will see traffic_flow_options.lua, traffic_flow.luax and traffic_flow.luaxtoken.
Upload these three files to your Log Decoder.
1. In the NetWitness UI, select Administration > Services and select a Log Decoder.
2. Select Parsers > Upload.
3. In the Upload Parsers dialog box, click , and navigate to where you saved the parser files on your computer.
4. Select all three files: traffic_flow_options.lua, traffic_flow.luax and traffic_flow.luaxtoken.
5. Click Upload.
  
  The files get uploaded, and you can view them in the file list.
Restart the Log Decoder service.

Update XML Files

Note: This section only applies to versions prior to 10.6.2. For all newer versions, these entries are delivered automatically.

Add Entries to the Index Concentrator File

You need to add entries to the index-concentrator-custom.xml on the Concentrators you would like to index to the meta.

Depending on your version:
- For NetWitness 11.x: In the NetWitness menu, select ADMIN > Services.
- For Security Analytics 10.x: In the Security Analytics menu, select Administration > Services.
Select a Concentrator.
Select View > Config from the Actions menu.
Select the Files tab, then select the index-concentrator-custom.xml file.
Add the following lines:

<key description="Network Name" level="IndexValues" name="netname" format="Text" valueMax="10000"/> <key description="Traffic Flow Direction" level="IndexValues" name="direction" format="Text" valueMax="10000"/>

Add Entries to the Table Map File (Log Decoders Only)

You need to add entries to the table-map-custom.xml file on the Log Decoder.

Depending on your version:
- For NetWitness 11.x: In the NetWitness menu, select ADMIN > Services.
- For Security Analytics 10.x: In the Security Analytics menu, select Administration > Services.
Select a Log Decoder.
Select View > Config from the Actions menu.
Select the Files tab, then select the table-map-custom.xml file.
Add the following lines:

<mapping envisionName="netname" nwName="netname" flags="None"/> <mapping envisionName="direction" nwName="direction" flags="None"/>

Update ESA Configuration

To be able to create ESA rules against the netname meta, you need to make netname an array.

To make netname an array:

Log onto RSA NetWitness Platform as an administrator.
Depending on your version:
- For NetWitness 11.x: In the NetWitness menu, select ADMIN > Services.
- For Security Analytics 10.x: In the Security Analytics menu, select Administration > Services.
Select the ESA service, then View > Explore.
From the left pane, select Workflow > Source > nextgen.
Add the meta key name 'netname' to the ArrayFieldNames.

Tuning

The following sections describe the parser default values, the options file, and present examples.

Parser Defaults

The default settings for the parser are as follows (these values are in the internal table, which replaces the definitions table from the earlier version of the options file):

["0/8"] = "broadcast"
["10/8"] = "private"
["127/8"] = "loopback"
["169.254/16"] = "link-local"
["172.16/12"] = "private"
["192.168/16"] = "private"
["224/4"] = "multicast"
["240/4"] = "reserved"
["255.255.255.255/32"] = "broadcast"

If these settings are sufficient, then you do not need to deploy the Options file.

Options File

The Options file contains extensive comments, as well as all the defined default values. Note the following:

Local subnets must be defined in "internal"
Named external subnets must be defined in "external". They will result in netname meta, but be considered as "external" for purposes of directionality.
If a subnet is not listed, it results in other, for example , netname: other src.
Aggregation is recommended wherever possible.

For IPv4, note the following:

Use only CIDR notation. Specifically, don't use a netmask like "255.255.255.0"
Both shorthand and normal CIDR are valid.
If netmask is omitted, /32 is assumed.

For IPv6, note the following:

Use only prefix notation.
Both compressed and expanded addresses are supported.
If prefix is omitted, /64 is assumed (not /128).
Only IPv6 prefixes aligned with nibble (4-bit) boundaries are supported (/128, /124, /120, ... /12, /8, /4). Subnets specified using non-aligned prefixes will not be used.

If you need to define subnets with prefixes that are not nibble-aligned, define multiple smaller prefixes. For example, instead of

["2000:a:bc:dee::/63"] = "campus",

Use the following:

["2000:a:bc:dee::/64"] = "campus",
["2000:a:bc:def::/64"] = "campus",

Caution: RSA strongly suggests that you do not subscribe to the options file. Subsequent downloads of this file will overwrite all changes that you have made to the file.

Note the following:

If you deploy the options file, it can be found in the same directory as parsers: /etc/netwitness/ng/parsers/.
The parser is not dependent upon the options file. The parser will load and run even in the absence of the options file. The options file is only required if you need to change the default settings.
If you do not have an options file (or if your options file is invalid), the parser uses the default settings.

Note: The parser will never use both the defaults and customized options. If the options file exists and its contents can be loaded, then the defaults will not be used at all.

Editing the Options FileEditing the Options File

On the Decoder

To change any of the options for the parser on the Decoder, you must edit the options file itself. You do this from either:

A shell on the Decoder, or
In the NetWitness UI: go to the Decoder > Config > Files.

On the Log Decoder

To change any of the options for the parser on the Log Decoder, you must edit the options file itself. You do this from either:

A shell on the Log Decoder, or
In the NetWitness UI: go to the Log Decoder > Config > Files.

Note: Changes take effect immediately. (This is not true for other parsers.)

Option File Details

Starting with version 2017.03.08.1 of the traffic_flow_options file, the file contains two tables:

internal. This table replaces the previous definitions table. Subnets listed here receive netname meta and are considered internal for directionality.
external. Subnets listed here receive netname meta and are considered external for directionality.

Note: In all prior versions to 2017.03.08.1, there is only one definitions table.

Addresses matching neither table receive netname "other," and are considered external for directionality.

For backwards compatibility, if the parser finds a definitions table rather than an internal table, then definitions is used as internal. If the parser finds both, then it uses internal.

The tables internal and external for 'traffic_flow' are Lua tables that contain a list of netblocks in CIDR notation. The syntax of each entry is:

["ip/prefix"] = "name",

For example:

["192.168.0.0/16"] = "private",

Shorthand notation is valid. For example, the following entry is equivalent to the previous example:

["192.168/16"] = "private",

Caution: All aspects of the syntax are crucial, including the trailing comma. If the options file is edited so as to become invalid, the parser will use default values.

Be sure to account for all of your environment's internal networks (but only these).

If a specific entry in the lua table isn't valid CIDR notation, it will be ignored. However, it will not affect the rest of the table (so long as the table syntax is valid) and an error will be logged. For example:

["10.1.1.0/a"] = "dmz",

This results in the following logged error message:

traffic_flow: invalid cidr '10.1.1.0/a'

Matching Rules

For matching, the most specific prefix for an IP "wins". For example, assume the options file has both of the following entries:

["192.168.0.0/16"] = "private", ["192.168.1.0/24"] = "dmz",

The IP 192.168.1.1 will match "dmz", and the IP 192.168.2.1 will match "private".

The following keys will be matched against the networks listed in the table:

ip.src
ip.dst
alias.ip
ip.addr
orig_ip

Examples

For each value of each of the above keys, meta is registered with the key "netname" as follows:

For ip.src: <name from entry> src

For example: private src
For ip.dst: <name from entry> dst

For example: netname: private dst
For the other keys: <name from entry> misc

For example: netname: private misc

If a value does not match an entry in the table, then the name "other" is used instead:

Example for ip.src: other src
Example for ip.dst: netname: other dst
Example for the other keys: netname: other misc

Further, for each src and dst pair meta "direction" is registered:

If ip.src is listed and ip.dst is "other":

direction: outbound
If ip.src is "other" and ip.dst is "listed":

direction: inbound
Both ip.src and ip.dst are listed, or neither are listed:

direction: lateral

Here is an example of what the options file looks like for versions 2017.03.08.1, and newer, where the file contains both the internal and external network definitions:

Note: Some application rules leverage “direction = outbound” to help minimize false positives and reduce noise. By listing web proxies as external, this allows these app rules to function as expected if you have NetWitness visibility on the ‘inside’ of your web proxy. If NetWitness has visibility on the ‘outside’ of your proxy, there is no need to set your web proxies as ‘external’, and the app rules will function as intended.

function internal()
--[=[   INTERNAL NETWORKS
            For proper direction meta:
              (a) add all internal subnets
              (b) DO NOT add any external subnets here
    --]=]
    return {
        ["0/8"] = "broadcast",
        ["10/8"] = "private",
        ["127/8"] = "loopback",
        ["169.254/16"] = "link-local",
        ["172.16/12"] = "private",
        ["192.168/16"] = "private",
        ["192.168.2.101/32"] = "HOST vineyard",
        ["224/4"] = "multicast",
        ["240/4"] = "reserved",
        ["255.255.255.255/32"] = "broadcast",
    }
end

function external()
--[=[   NAMED EXTERNAL NETWORKS
            For proper direction meta:
              (a) DO NOT add any internal subnets here
              (b) add any desired external subnets
    --]=]
    return {
        ["1.2.3.0/24"] = "partner network vpn",
        ["104/8"] = "TESTNET-1",
        ["151.101/16"] = "TESTNET-2",
        ["172.16.0.0/24"] = "Web proxies",
        ["65.52/16"] = "DMZ",
        ["52.84.126.141/32"] = "KEEP_AN_EYE_ON_THIS_SYSTEM",
    }

The following is an example of the definitions table for prior versions of the options file (that contained only one table):

function definitions()
    return {
        ["0/8"] = "broadcast",
        ["10/8"] = "private10",
        ["10.10.72.0/21"] = "topeka-lab01",
        ["10.10.80.0/21"] = "topeka-voip",
        ["10.10.88.0/21"] = "topeka-voip",
        ["10.10.96.0/21"] = "topeka-vpn",
        ["10.10.140.0/19"] = "campus-works",
        ["127/8"] = "loopback",
        ["169.254/16"] = "link-local",
        ["172.16/12"] = "private172",
        ["192.168/16"] = "private192",
        ["192.168.2.253"] = "***<<SourceCode_Server>>***",
        ["192.168.121.0/24"] = "hydrotestlab",
        ["192.168.150.101"] = "corporate-svr01",
        ["192.168.150.102"] = "corporate-svr02",
        ["192.168.150.103"] = "corporate-svr03",
        ["192.168.150.0/20"] = "corporate-netp150",
        ["224/4"] = "multicast",
        ["240/4"] = "reserved",
        ["255.255.255.255/32"] = "broadcast",
        ["201.101.141.0/24"] = "ACMElabB1F01R101",
        ["201.101.142.0/24"] = "ACMElabB1F01R102",
        ["201.101.143.0/24"] = "ACMElabB1F01R103",
        ["201.101.144.0/24"] = "ACMElabB1F01R104",
        ["208.43.253.0/24"] = "brooklyn-b2f1l012",
        ["217.43.253.0/24"] = "brooklyn-b2f2l012",
        ["222.43.253.0/19"] = "brooklyn-b2f3l012",
        ["222.44.253.0/24"] = "brooklyn-b01SF",
        ["228.100.17.0/26"] = "brooklyn-DMZ01",
        ["228.100.18.0/27"] = "brooklyn-DMZ02",
        ["88.56.104.0/20"] = "remoteOffice-R201",
        ["147.22.56.0/21"] = "Branch-Services",
    }

Key Mappings and Defaults

Keys used for source (netname_src) and the determination of directionality:

ip.src
ipv6.src

Keys used for destination (netname_dst and the determination of directionality:

ip.dst
ipv6.dst

Keys used only for miscellaneous (netname_misc) not the determination of directionality:

alias.ip
ip.orig
alias.ipv6
ipv6.orig
ip.addr
ipv6.addr

Full default values:

INTERNAL

["0/8"] = "broadcast",
["10/8"] = "private",
["127/8"] = "loopback",
["169.254/16"] = "link-local",
["172.16/12"] = "private",
["192.168/16"] = "private",
["224/4"] = "multicast",
["240/4"] = "reserved",
["255.255.255.255/32"] = "broadcast",
["fc00::/8"] = "unique-local",
["fd00::/8"] = "unique-local",
["fe80::/12"] = "link-local",
["fe90::/12"] = "link-local",
["fea0::/12"] = "link-local",
["feb0::/12"] = "link-local",
["ff00::/8"] = "multicast",

EXTERNAL

["ff0e::/16"] = "multicast global",
["ff1e::/16"] = "multicast global",