(Optional) Configure Selective Network Data Collection

Selective network data collection gives administrators the ability to apply centrally managed capture policies across their Network Decoders. This results in better use of service resources, including hard drive space, which leads to more predictable costs and lessens the burden of managing multiple services. Administrators can determine which traffic is stored and how it is stored by using policies. Each policy contains a list of supported base protocols and definitions for handling any other protocols that are detected.

Note: Administrators can use this feature to create policies for metadata-only while using the cost-effective Meta-Only license, without having to pay for additional storage or throughput licensing. To use this license, work with your NetWitness Customer Support Representative to purchase the license, and then enable the functionality as described in (Optional) Configure Meta-Only Decoders.

The administrator can choose to deploy predefined policies that capture:

  • All base and other protocols (Full Capture - All Protocols)
  • Only metadata on all base and other protocols (Capture Meta Only - All Protocols)
  • Only metadata on all base protocols and drop all other protocols (Capture Meta on Base Protocols, Drop all other protocols)
  • All base protocols and only metadata on all other protocols (Full Capture on Base Protocols, Meta only on all other protocols)

The predefined policies are not configurable. The only way you can edit predefined policies is by assigning services (such as Decoders) to deploy them.

Administrators can create custom policies to give further control over the deployment. A base set of protocols are available for alterations by the administrator, allowing you to choose what level of capture you prefer on a per-protocol basis. If you are only making slight changes, a good start for customization is to clone one of the predefined polices and alter it. These centrally-managed policies are then applied to services (Network Decoders) to allow handling multiple use cases across your environment.

Here are some example use cases and how an administrator would deploy these.

Use Case 1: Predefined Policy for Internal Assets

To gain further visibility into internal lateral traffic going East-West through the environment, a Network Decoder is deployed in the interior of the network to capture this traffic. Applying full packet capture for this segment may generate too much data to manage at a reasonable cost point. Instead, the predefined "Capture Meta Only – All Protocols" policy can be applied to achieve the same parsing and detection while only saving the metadata, limiting the amount of storage required of the solution. In this situation, the only steps you must complete is to choose the predefined policy, edit it, and then enable it by assigning it to the appropriate internal services (Decoders). For an example of how to set up this policy, see Use Predefined Policies.

Use Case 2: Custom Policy for DMZ Assets

To protect the public-facing web servers, a Network Decoder is deployed into the demilitarized zone (DMZ). The network communications between these web servers and other network devices are a subset of the base protocols. To make sure you can harvest forensic evidence for any communications related to these high-value assets, you would want to do full packet capture on HTTP, SMTP, FTP, and DNS protocols. You still want visibility into any other traffic these servers are engaged in, so you would prefer metadata generated for all other protocols. The suggested steps to create this custom policy would have you choose the predefined policy "Full Capture on Base Protocols, Meta only on all other protocols" and then clone it to adjust it to the protocols aligned with the DMZ scenario. During the clone process, the name and description can be updated to identify the new purpose of the policy. The define policy step allows all the base protocol rules inside the policy except HTTP, SMTP, FTP, and DNS to be altered from "Collect All" to "Collect Meta Only" to achieve the desired policy. The last steps are assigning the policy to the appropriate DMZ services (Decoders). For an example of how to set up this policy, see Create New Policies from Predefined Ones.

Note: When a Decoder is being managed by a selective collection policy, it is possible that the Decoder specifications in the policy could be overwritten if the Decoder configuration is edited outside of the policy. To try and avoid this situation, the following warning message is displayed at the top of the Parsers section of the Decoder Configuration page: This Decoder is being managed by the following capture policy: <policy-name>. Some parsers are required by the capture policy, and changes made here may overwrite that policy. Review of the capture policy is recommended.
Another warning message is displayed at the top of the Application Rule section: This Decoder is being managed by the following capture policy: <policy-name>. Some application rules are required by the capture policy, and changes made here may overwrite that policy. Review of the capture policy is recommended.

The following sections describe configuring and using selective network data collection.

For detailed information about the Selective Collection user interface, see:

Use Predefined Policies

This procedure describes how to use the predefined policies. You can use this procedure to configure a policy for internal assets. For example, as described in Use Case 1: Predefined Policy for Internal Assets , to gain further visibility into internal lateral traffic going East-West through the environment, you could deploy a Network Decoder in the interior of your network to capture the traffic. Applying full packet capture for this segment may generate too much data to manage at a reasonable cost point. Instead, use the predefined "Capture Meta Only – All Protocols" policy to achieve the same parsing and detection while only saving the metadata, limiting the amount of storage required for this solution.

In this situation, the only steps you must complete is to select the "Capture Meta Only – All Protocols" predefined policy to edit it, enable it by assigning it to the appropriate Decoders, and then publish the policy to start collecting the data.

To work with a predefined policy:

  1. In the NetWitness user interface, go to netwitness_configureicon_24x21.png (Configure) and select the CAPTURE POLICIES tab.

    netwitness_12.1_collection-policies-tab_1122.png
    Notice that the PUBLICATION STATUS column lists each of the predefined policies with the status of Unpublished. The only way to edit a predefined policy is to assign Decoders to apply the policy. If changes are made to the Decoder assignments but are not published, the status changes to Unpublished Edits, the date the change was made is displayed in the POLICY UPDATED column, and the user name of the person who made the change is displayed in the UPDATED BY column. In predefined policies, the CREATED BY field is always system.
    The following figure shows the statuses of the policies.

    netwitness_12.1_coll-pol-pub-status-change_1122.png
  2. Select Capture Meta Only - All Protocols, and then click Edit.
    The editing view is displayed with the Identify Policy option active.

    netwitness_12.1_cap-pol-edit-vw_1122.png
  3. Because this is a predefined policy, you cannot change the policy name or description. Click Next.
    The Define Policy: Capture Meta Only - All Protocols page is displayed.

    netwitness_12.1_define-policy_1122.png
  4. Because this is a predefined policy, you cannot edit this page. Review the rules on this page to understand the actions that will be enabled when you publish this policy. Click Next.
    The Assign Policy page is displayed.

    netwitness_12.1_assign-policy_1122.png

  5. Select the Decoders that you want to collect from, and then click Save and Publish.
    The publication status moves to Updating. It changes to Published when the policy has been successfully deployed to all the Decoders that you selected. However, the Decoders that are available start gathering data immediately, before the policy is displayed as Published, so even if not all of the Decoders are available (and the publication status is not Published), data will begin to be collected on the Decoders that are available. When the status moves to Published, all the services are displayed in the Service Assignment column. You can hover over the names of the Decoders to see their full names, as shown in the following figure.

    netwitness_12.1_published-hover-txt_1122.png

Create New Policies from Predefined Ones

This procedure describes the steps for creating a new policy from a predefined one, and provides an example of how you might use this policy.

For example, as described in Use Case 2: Custom Policy for DMZ Assets , suppose a Network Decoder has been deployed into the demilitarized zone (DMZ) to protect public-facing web servers. The network communications between these web servers and other network devices are a subset of the base protocols. To make sure they can harvest forensic evidence for any communications related to these high value assets, you would want to get full packet capture on HTTP, SMTP, FTP, and DNS protocols. To provide visibility into any other traffic these servers are engaged in, you would want to generate metadata for all other protocols.

To configure a policy that would achieve the results described here, you would select the predefined policy "Full Capture on Base Protocols, Meta only on all other protocols," clone it, and then edit it to align the protocols with the DMZ scenario. During the clone process, you can update the name and description to identify the new purpose of the policy. In the step where you define the policy, you can change all the base protocol rules inside the policy (except HTTP, SMTP, FTP, and DNS) from "Collect All" to "Collect Meta Only" to achieve the desired results. The last steps assign the policy to the appropriate DMZ Decoders.

  1. In the NetWitness user interface, go to netwitness_configureicon_24x21.png (Configure) and select the CAPTURE POLICIES tab.

    netwitness_12.1_collection-policies-tab_1122.png
  2. Select Full Capture on Base Protocols, Meta only on all other protocols and then click Clone.
    The Collection Policies view is displayed with the cloned policy at the bottom of the list, identified by appending -1 to the policy name.

    netwitness_12.1_cloned-policy_1122.png
  3. Select the cloned policy and click Edit.
    The editing view is displayed with the Identify Policy option active.
    In this example, we are naming this policy "DMZ Assets" and providing the description "This policy is defined to protect public-facing web servers."

    netwitness_12.1_cloned-policy-dmz-id_1122.png
  4. Click Next. The Define Policy view is displayed, with a list of rules you can use to define the policy.
    You can select the following options for rule actions:
    • Collect Meta Only: Collect metadata
    • Drop All: Drop metadata and network packets
    • Collect All: Collect metadata and network packets

    To collect the targeted data for the DMZ Assets policy, we are going to change all the base protocol rules inside the policy (except HTTP, SMTP, FTP, and DNS) from Collect All to Collect Meta Only, as shown below.
    netwitness_12.1_clone-pol-edit-vw_1122.png

  5. Click Next.
    The Assign Policy page is displayed.

    netwitness_12.1_clone-assign-policy_1122.png

  6. A Decoder can only be assigned to one policy. Decoders that are not available are greyed-out, because they are already assigned to other policies. Select the Decoders to collect the targeted data, and then click Save and Publish. The publication status moves to Updating. It changes to Published when the policy has been successfully deployed to all the Decoders that you selected. However, the Decoders that are available start gathering data immediately, before the policy is displayed as Published, so even if not all of the Decoders are available (and the publication status is not Published), data will begin to be collected on the Decoders that are available.
    Your new policy is displayed in the Collection Polices view, with the services displayed in the Service Assignment column.

    netwitness_12.1_dmz-published_1122.png

Create Custom Policies

You can create a custom Selective Network Decoder policy, where you can select specific protocols on which to collect data, as well as the rules that apply to them.

  1. In the NetWitness user interface, go to netwitness_configureicon_24x21.png (Configure) and select the CAPTURE POLICIES tab.

    netwitness_12.1_collection-policies-tab_1122.png
  2. Click Create New.
    The editing view is displayed with the Identify Policy option active.

    netwitness_12.1_new-pol-edit-vw_1122.png
  3. Enter a name and description for your policy, and then click Next.
    The Define Policy view is displayed.

    netwitness_12.1_new-define-policy_1122.png
  4. Define the actions for each of the protocols. The options for RULE ACTION are:
    • Collect Meta Only: Collect metadata
    • Drop All: Drop metadata and network packets
    • Collect All: Collect metadata and network packets
  5. Click Next.
    The Assign Policy view is displayed.
    netwitness_12.1_new-assign-policy_1122.png

  6. A Decoder can only be assigned to one policy. Decoders that are not available are greyed-out, because they are already assigned to other policies. Select the Decoders to collect the targeted data, and then click Save and Publish. The publication status moves to Updating. It changes to Published when the policy has been successfully deployed to all the Decoders that you selected. However, the Decoders that are available start gathering data immediately, before the policy is displayed as Published, so even if not all of the Decoders are available (and the publication status is not Published), data will begin to be collected on the Decoders that are available.

    Your new policy is displayed in the Collection Polices view, with the services displayed in the Service Assignment column.
    netwitness_12.1_new-published-policy_1122.png

Verify Published Policies

When a policy has been published, the policy is applied to the Decoders that were designated in the policy assignment step. The Decoders that are assigned to the policy start collecting the data defined in the policy. There are a few ways to validate the policy is functioning as expected.

View Publication Status

Publication status is indicated in the Capture Polices view in the PUBLICATION STATUS column. You can also see which Decoders are assigned to the policy in the SERVICE ASSIGNMENT COLUMN.


netwitness_12.1_verify-pub-status_1122.png

The publication statuses are:

  • Unpublished - the rule actions of the policy have been defined and Decoders may have been assigned to the policy, but the policy has not been deployed to any Decoders.
  • Unpublished Edits - policies that have been updated but are not deployed to Decoders. Clicking Save and Close after making edits keeps them from automatically being published, allowing the administrator to wait until a specific outage window to publish them. This would display as Unpublished (for policies not previously published to Decoders) or Unpublished Edits (for policies that have been previously published to Decoders, but the updates have not been deployed to Decoders).
  • Published - the policy has been successfully deployed to all the Decoders that you selected. However, the Decoders that are available start gathering data immediately, before the policy is displayed as Published, so even if not all of the Decoders are available (and the publication status is not Published), data will begin to be collected on the Decoders that are available.
  • Failed - policies that failed to execute, for example, if a Decoder is offline or a system is down.

You can also select a published policy, and then click Edit to verify the Decoders that are assigned to the policy.

netwitness_12.1_verify-pub-status2_1122.png

Verify Policies in Admin > Services

  1. As an administrator, log in to the NetWitness user interface to view the Decoders that have been assigned to the policy.
  2. Go to netwitness_adminicon_25x22.png (Admin) > Services > <decoder name> > netwitness_ic-actns.png(Actions) > View > Config. On the General tab, in the right panel, there is a list of parsers that are installed on the Decoder by default. If a policy is administering this Decoder and requires any of those parsers to function, a message is displayed as shown below.
    netwitness_12.1_parser-message_1122.png

  3. On the App Rules tab, a similar message indicates that some of the rules are required for the policy to function. You can search for "selective collection" to find the specific rules required, as the rule names contain that information.

    Note: In Investigate, you must have the Alert session option selected for the selective collection policy to be displayed.

    netwitness_12.1_app-rules-message_1122.png

By default, the selective collection rules are added to the bottom of the rule set when the policy is applied. However, an administrator can directly affect the policy by altering the selective policy rules or by moving conflicting rules (rules with the same services) above the selective collection rules. In both cases, those changes take effect when they are saved locally on the individual Decoders, until the next time the policy is published from the central configuration page, when that policy will override changes made individually on the Decoders.

Note: There currently is no feedback in the centralized configuration page denoting that the configuration related to the policy has been altered locally on the Decoders. To avoid this situation, manage all the policy rules from the centralized configuration page. If that is not possible, as additional application rules outside of collection policies may be required, follow standard security practices, including separation of duties, to limit people who need access to alter application rules and general parsers.

Verify Policies in the Investigate View

You can further verify that the policy is working as expected (the protocols are being collected as published) in the Investigate view. In the centralized configuration page for selective collection, the administrator can get a translation of what the query parameter would be for each individual protocol when viewing the rules inside the policy.

For example, in netwitness_configureicon_24x21.png (Configure) > Capture Policies > <select a policy> > Edit > Define Policy, if you hover over the DNS protocol name, service=53 is displayed.

netwitness_12.1_define-pol-hover-dns_1122.png

To determine if the appropriate action has been taken for DNS traffic, in Investigate > Events you can add the query parameter (for example, service=53) as a filter in the query bar and run the query against the Broker or Concentrator the Network Decoder is associated with for the time the policy has been active on the Network Decoder. To make sure you are only seeing the traffic from the Decoders that you want, you can add the Decoder Source, or did meta key as another filter in the query bar.

netwitness_12.1_inv-val-query_1122.png

After running the query, if there are any related results, those events are displayed in the table (unless you applied an action to drop them).

netwitness_12.1_inv-val-results_1122.png

If you chose the action to collect all you can click on one of the events and review the metadata and the payload information for that packet session.

netwitness_12.1_inv-val-meta_1122.png

If you chose the action to collect meta only, you can click on one of the events and review the metadata, however, a message would be displayed instead of the packet payloads, as shown in the following figure.

netwitness_12.1_inv-val-error-message_1122.png

To expedite this process, you can also take the application rules from one of the Network Decoders the policy is applied to and use it as the filter in the Investigate > Events query bar.

The following figure shows an entire collect meta only configuration by using the application rule shown here, created on the Decoder (in Verify Policies in Admin > Services😞 (service=0,20,21,22,23,25,53,67,69,80,110,119,137,139,161,443,520,1433,1521,1719,1720,2000,2049,5060,5222,6667) truncate:

netwitness_12.1_inv-val-meta-rule_1122.png

Note: If the findings do not align with what is expected from the policy, make sure that there are not other rules on the individual Decoders that might be conflicting with the selective collection policy rules. In addition, make sure when checking in Investigate that you are looking at only the traffic associated with the Decoders related to the policy.

Troubleshooting Policy Deployments That Fail

Policy deployment can fail for the following reasons:

  • A Decoder service assigned to the policy is not running

  • A Decoder host system is down when you try to deploy the policy

When a policy deployment fails, the Failed icon is displayed in the PUBLICATION STATUS and SERVICE ASSIGNMENT columns as shown in the following figure.

netwitness_12.1_failure_1122.png

Use the following options to research publication failures:

Check the status of the Decoders

Check to see if the Decoder system is down, or if the Decoder service has stopped capturing data.

  1. Select the policy, click Edit, and go to the Assign Policy page to determine which Decoders are assigned to the policy,
  2. Go to netwitness_adminicon_25x22.png (Admin) > Services, select the Decoder, click netwitness_ic-actns.png > View > System to investigate the status of the Decoder.

Review the content-server.log file

You can review the content in this file to troubleshoot errors. This file is located in the following directory on the NW Server host:

/var/log/netwitness/content-server/content-server.log

The following example shows a content-server log snippet that is displayed when publishing a policy fails:

2020-07-08 16:22:45,287 [ nioEventLoopGroup-2-2]
INFO DataAccess|Finished publishing policy 'Full Capture on Base Protocols, Meta only on all other protocols-1' outcome:FAILED

This is an example of a content server log snippet for successfully publishing a policy:

2020-06-23 17:40:50,373 [ nioEventLoopGroup-2-7]
INFO DataAccess|Finished publishing policy 'Policy 1' outcome:SUCCESS

Review /var/log/messages on Decoder Systems

You can view publication status in /var/log/messages on a Decoder system.

The following is an example of successful publication in /var/log/messages:

Jun 15 18:35:17 rsa-nw-1141-Decoder-1 NwDecoder[8260]: [Rules] [audit] User admin (session 26182, 10.237.169.43:55380) has added application rule 'selective-collection:meta-only' at position 38

Unpublish Policies

You can unpublish a policy by unassigning the Network Decoders from the policy, which preserves the policy but makes it inactive so that the data it defines is not being collected. You can then change or remove a portion of policy, or remove the entire policy from the Decoders to which they are assigned.

To unpublish a policy:

  1. In the NetWitness user interface, go to netwitness_configureicon_24x21.png (Configure) and select the CAPTURE POLICIES tab.
    netwitness_12.1_collection-policies-tab_1122.png
  2. Select the published policy and click Edit.
    netwitness_12.1_edit-unpublish_1122.png
    1. To change the name or description of the policy, update the policy name or description in the Identify Policy view.
    2. To change the protocols that are collected, click Next and then edit the rule actions in the Define Policy view.
      netwitness_12.1_unpublish-definepolvw_1122.png
    3. To unassign or reassign Decoders to this policy, click Next.
      • To fully unpublish this policy, deselect all the Decoders and then click Save and Publish.
      • To change the Decoders that are assigned to this policy, select or deselect the Decoders as needed and then click Save and Publish.

When the changes to the policy status have been completed and the Decoders updated, the policy status is displayed in green text as Published on the Collection Policies tab. If you unassigned all the Decoders without assigning any new ones, the publication status is Unpublished Edits.

Delete Policies

You can only delete customized policies. Predefined policies can be unpublished, but cannot be deleted. You can only delete one policy at a time. When you delete a policy, any Decoders that were assigned to the policy immediately stop applying the rules in the policy.

For information about how to unpublish a policy, see Unpublish Policies.

To delete a policy:

  1. In the NetWitness user interface, go to netwitness_configureicon_24x21.png (Configure) and select the CAPTURE POLICIES tab.
    netwitness_12.1_collection-policies-tab_1122.png
  2. Select a customized policy and click Delete.
    The policy is removed from the system.

Caution: If you delete a policy that is published to Decoders, a warning is displayed. If you click Accept in the warning, that policy is removed from the Decoders, the collection status is changed, and the policy is deleted.

Supported Protocols for Selective Network Data Collection

The protocols in this list are supported for selective network data collection. These protocols have descriptive tool tips in the user interface. You can choose how to handle each of these protocols individually by dropping, capturing metadata only, or capturing all data related to them. There is also an additional rule for each policy that handles all other unidentified protocols not included in this list. The policies apply the collect or drop actions based on protocol characteristics, regardless of the port that they are captured on.

Category Protocol Description
Database

TNS

Transparent Network Substrate Oracle database networking protocol messages
TDS Tabular Data Stream for SQL server communications
Email

SMTP

 Simple Mail Transport Protocol messages
POP3 Post Office Protocol v3 messages (Email clients use to retrieve mail from server)
File

SMB

Server Message Block (or CIFS) network communication protocol file sharing communications

NFS

 Network File System protocol messages

FTP

File Transfer Protocol messages

TFTP Trivial File Transfer Protocol messages
Networking

SNMP

Simple Network Management Protocol messages

DNS

Domain Name System messages

DHCP

Dynamic Host Configuration Protocol messages

NETBIOS

Network Basic Input/Output System networking protocol messages

Remote Access

SSH

Secure Shell secure remote connections

TELNET

Teletype network remote interface protocol messages

SCCP

Cisco Skinny Client Control Protocol terminal control protocol messages

Routing

RIP

Routing Information Protocol distance-vector routing protocol messages

Social Media GTalk

Google Talk or Google Chat instant text messages

IRC Internet Relay Chat protocol text messages

NNTP

Network News Transfer Protocol for Usenet news articles
VoIP RTP

Real-time Transport Protocol audio and video streaming protocol messages

SIP

Session Initiation Protocol text-based messaging application control protocol messages
H323 ITU-T Q.931 based protocol for Voice over IP (VoIP)

Web

HTTP/HTTP2

HyperText Transfer Protocol 1.x/2.x messages

HTTPS/TLS HyperText Transfer Protocol Secure over Transport Layer Security or Secure Sockets Layer encrypted messages

Note: When selective network data collection is set up to collect metadata only for the HTTPS/TLS protocol, it truncates the traffic after the SSL/TLS handshake to ensure that all the certificates and other details are retained for analysis purposes while the encrypted packet payloads are dropped.