(Optional) Data Plane Development Kit Packet Capture

The NetWitness Decoder supports capturing packets through the Data Plane Development Kit, also known as DPDK. DPDK is a set of libraries and drivers that support applications that work on directly on packets. DPDK's web site, www.dpdk.org, provides more information.

Advantages

The Decoder can use the DPDK to ingest packets from Ethernet interfaces. It is an alternative to existing packet capture options built into the Decoder. DPDK offers these advantages:

  • It is entirely open source, with no binary-only driver components, unlike PF_RING.
  • It is directly supported by multiple vendors, including our main vendor, Intel.
  • Since the kernel-level interface is in the mainline Linux kernel, there are no separate driver packages to compile and distribute.
  • DPDK devices are completely separate from the Linux network stack, which avoids unintentional interaction between Linux networking and packet capture.
  • DPDK is capable of capturing at greater than 10G, unlike the packet_mmap.

How it Works

DPDK completely detaches the Ethernet interface from the Linux kernel. It does this by doing the following:

  • It uses a Linux User I/O driver mechanism to expose the raw PCI-Express device to normal Linux programs. This means that it can take the raw memory space mapped to the PCI device and expose it, so that a normal Linux program can directly manipulate the hardware. There are a couple of drivers built into the Linux mainline kernel that can accomplish this. Decoder is designed to use vfio_pci_generic.

    • Note: Only one driver can be attached to an Ethernet interface at a time. When vfio_pci_generic is attached to an interface, the Linux network driver for that interface is detached.

  • The DPDK provides its own user-space implementation of the hardware drivers for a wide variety of Ethernet cards. When a DPDK application like Decoder is loaded, it can attach directly to the PCI device and manipulate it using DPDK-specific library functions. When a device is attached to DPDK, the Decoder can see it and automatically sets the device up with the DPDK Capture Device option.

Migrate PF_RING Devices to DPDK

For configurations using multiple adapter packet capture (available in 11.5 and later), you must use the manual procedure outlined in Manually Move a Capture Interface to DPDK . For example, the migration (dpdk migrate) does not support a Decoder when capture is configured with: capture.interface=PFRINGZC,em3; PFRINGZC,em4;packet_mmap,em2.

For configurations prior to 11.5, the procedure that uses the devices parameter to capture from multiple PF_RING-supported interfaces (for example, ixgbe and i40e) will be converted to using DPDK, along with the multiple adapter packet capture, so that one capture thread is used per interface.

For example, the migration will work if configured as follows:

  • For multi-interface capture: capture.interface=PFRINGZC,em3 along with capture.device.params=device=zc:em3,zc:em4
  • For single interface capture: capture.interface=PFRINGZC,em3

Note: dpdk migrate only functions with PFRINGZC, not mmap interfaces.

If you are using PF_RING to capture packets on your Decoder, you can use the Decoder's dpdk migrate command to migrate your existing PF_RING capture devices to DPDK. This command uses your currently-selected capture device configuration to determine which network interfaces are moved to DPDK control.

  1. Go to netwitness_adminicon_25x22.png (Admin) > Services and select a Decoder.
  2. Click netwitness_ic-actns.png > View > Explore, right-click decoder and select Properties.
  3. From the command drop-down list, select the dpdk command, type the parameter migrate, and click Send.
  4. In the command output window, the changes that will be made on the Decoder to perform the migration are displayed. If everything looks correct for the migration, add the parameter commit=1 to the command after migrate to actually make the changes on the Decoder.


netwitness_migpfring3.png

  1. At the reboot prompt, reboot the host to make the driver loading changes.

  2. (Optional) Run the /decoder/devices/<devicename>/msg=prune command to remove configuration options for devices and interfaces that no longer exist after the migration.

    Note: The prune command removes only the undetected interface folders. Also, it ignores the interfaces, stats, nwimport, and flow_events folders available in /decoder/devices/.

Manually Move a Capture Interface to DPDK

  1. Run the new host service command /appliance/nicToDp, that converts a named network interface and sends it to DPDK.
    For example, if you provide nicToDp an interface name like p1p1 it detaches it and sets it up for DPDK to use. It does the following:
    1. Identifies the PCI address of the named interface.

      2. Note: From this point on, the Ethernet interface is identified by its PCI address alone, which is a physical characteristic based on where the Ethernet interface is installed in the computer.

    2. Creates configuration files that describe which PCI addresses DPDK is allowed to use.
    3. Sets up a systemd unit that binds the configured PCI addresses to DPDK instead of to the default Linux drivers on the next system reboot. The following figure shows a successful command.

netwitness_migpfring1.png

  1. Reboot the Decoder host. The reboot accomplishes the following:
    1. Ensures that live capture on the interfaces stops before the network driver is unloaded.
    2. Binds the vfio_pci_generic driver to the configured interfaces.
    3. Pre-allocates some Huge Page memory for each interface, as required by the DPDK libraries.
  2. After the reboot, the Decoder does not start capturing, since the network interface it had been using to capture has been moved to DPDK. Change the Decoder's capture configuration to select the DPDK device that is now available. For information about how to change the capture configuration, see Configure Capture Settings.
  3. Restart the service. The following figure shows the Decoder configuration after a restart.

netwitness_migpfring2.png

Check the device status and stats of DPDK interfaces

When you capture using DPDK interfaces, the link status and stats of the DPDK can be obtained using the command /decoder/devices dpStatus. The traditional commands like ethtool or ifconfig, which are used for other capture interfaces, will not fetch the link status and stats of DPDK devices.

  1. Go to (Admin) > Services.

  2. In the Administration Services view, select the Decoder and View > Explore.

  3. Navigate to decoder > devices, right click on devices and select Properties.

    netwitness_12.1_dpdkdev_1122.jpg

  4. Select dpStatus from the drop-down and click Send.

    netwitness_dpstatus.jpg

    The link status and stats of DPDK devices are displayed in the Response Output section.

    netwitness_dpstatusprop.jpg

Check link status and stats from NwConsole

Run /decoder/devices dpStatus on NwConsole. Sample output when capture is in Start state:

netwitness_dpdkoutput.png

Sample ouput when the capture is in Stop state:

Device : 0000:01:00.0

Link detected: no

Device : 0000:01:00.1

Link detected: no

Utilizing Receive Side Scaling with DPDK

Receive Side Scaling (RSS) efficiently distributes the network receive processing across multiple CPUs in a multi-processor system. RSS ensures that the processing associated with a given connection stays on the assigned CPU.

IMPORTANT: Receive Side Scaling is supported on DPDK devices using the ixgbe or i40e device drivers only.

Advantages

When RSS is enabled, the DPDK capture devices split the incoming packet stream into multiple queues providing following benefits:

  • Each queue is processed with a dedicated capture worker thread. As there are more threads processing the capture work, it allows more CPU time available for Berkeley Packet Filters (BPF) rule and network rule evaluation.
  • Each capture worker thread can feed multiple assemble worker threads simultaneously which allows more CPU time available for assembly.

  • Each CPU thread processes only a fraction of the packet stream. However, the total number of packets processed from all queues working in parallel increases the overall output (packets processed per second) of Decoder.

Configuring Receive Side Scaling

To enable RSS in a DPDK controlled NIC, set the /decoder/devices/DPDK/(PCI_ID)/config/rss value to a value greater than 1. The typical values of RSS must be a small power of 2, for example, 4 or 8. This value determines the number of CPU threads to be allotted for packet processing during the next capture. Similarly, if assembler threading is enabled, a matching number of assembler threads will be created to process the packets on each incoming queue.

For example, the following sample configuration will utilize a total of eight threads (four capture threads + four assembler threads) for multi-adapter packet capture with DPDK and multi-assembler threading turned on. This is the maximum recommended value for an eight core Decoder CPU.

assembler.threading.enabled set to on or true

/decoder/devices/DPDK/0000:01:00.1/config/rss=2

/decoder/devices/DPDK/0000:01:00.0/config/rss=2

To minimize packet drops, it is recommended that you keep the total number of capture and assembler threads less than the number of cores per CPU.

How it Works

When RSS is enabled, Decoder utilizes the RSS hardware acceleration on the capture NIC to apply a symmetric RSS hash function. The hash function sends packets from both directions of a given stream to the same queue. It enables the packets received on that queue to be processed by a dedicated assembler thread. This eliminates the need for Decoder to merge streams from different RSS queues and reassemble the streams. Hence, it is recommended that you enable multi-threaded assembly when using RSS. To enable, set /decoder/config/assembler.threading.enabled to true or on.

Configure the Decoder Capture Failover in Load Balance Deployments

The DPDK capture in the decoder service provides the ability for load balancers to failover and avoid data loss when decoder capture is off or when the service is shut down due to maintenance activity. When the decoder capture is stopped, the DPDK capture interface link status is down. This provides the ability for the traffic ingestion physical load balancers (for example, Keysight Ixia, Gigamon) which monitor interface links to switch to a different decoder and avoid data loss.

Configuration Failover on DPDK Capture Interfaces

Refer to the (Optional) Data Plane Development Kit Packet Capture topic for instructions to configure DPDK capture on the decoder.

  • You can migrate existing PF_RING devices to DPDK

    (Or)

  • Manually move capture interfaces to DPDK

Disable Wake On LAN (WOL) in BIOS settings

Some load balancers monitor interface link status through NIC link lights. You must disable WOL as the interface will not be down if WOL is enabled. To disable WOL in the BIOS, follow the BIOS instructions.

Example: f2 -> Device Settings -> "Integrated NIC 1" (capture interfaces) -> "NIC Configuration" -> "Wake on LAN" = Disabled

Note: When you restart the device and if capture does not auto-start, the lights on the capture interface are on. But if you turn on capture and turn it off again the lights will be off.

(Optional) Remove DPDK on Capture Interfaces

In case you need to revert DPDK on capture interfaces you can use the Appliance service dpToNic command.

  • If you run this command without arguments, it lists the ethernet devices controlled by DPDK.

    • Example: /appliance dpToNic

      DPDK Device List:

      82:00.1 Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection Ethernet Server Adapter X520-2

  • If you specify an ethernet interface using the 'eth' parameter, it removes the interface from DPDK so that when the system reboots it will be treated as a normal Linux network interface.

    • Example: /appliance dpToNic eth=82:00.1

      Detaching PCI Device 82:00.1 from DPDK.

      DPDK Device List: A RESTART IS REQUIRED FOR THIS TO TAKE EFFECT

Validate Link Status

  • On Capture Start

    • During the decoder capture start, the service logs DPDK interface link status

      • [DPDK] eth dev 0000:05:00.0 link is up

    • On the upstream traffic ingest Load Balancer, the link status would be detected.

      • If it is a Linux-based device, you can use the ethtool <interface name> command to check link status

  • On Capture Stop

    • During the decoder capture stop, the service logs that capture is stopped on DPDK interfaces

      • [Thread] Stopped thread: Capture Work (DPDK, 0000:05:00.0) id: 31214

    • On the upstream traffic ingest Load Balancer, the link status would not be detected.

      • If it is a Linux-based device, you can use the ethtool <interface name> command to check link status.

DPDK Interface Configuration and Statistics

  • The dpdk-devbind --status command displays the status of DPDK and other known network interfaces. It displays the PCI domain, bus, slot, and function, along with a text description of the device.

    • Example: dpdk-devbind --status

      • 0000:01:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe_zc

      • 0000:01:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=vfio-pci unused=ixgbe_zc

  • The default MTU is 2048. If you need to adjust the MTU for capture, add the snaplen parameter to /decoder/config/capture.device.params .

  • VLAN offload is disabled.

  • The DPDK interface capture stats are updated under /decoder/devices/dpdk/

    • Example: /decoder/devices/DPDK/0000:01:00.1/stats/

      • Received packets (Rx packets) - capture.received

      • Received bytes (Rx bytes) - capture.total.bytes

      • Received dropped (Rx dropped) - capture.dropped

For information on checking device status and stats of DPDK interfaces, refer to Check the device status and stats of DPDK interfaces