Useful Commands
The following NwConsole commands are useful when interacting with NetWitness Core services:
- feed: Enables you to create and work with feed files.
- makepcap: Converts Packet database (DB) files to PCAP.
- packets: Retrieves packets or logs from the logged in service.
- hash: Creates or verifies hashes of database files.
The following sections as well as the NwConsole help and topic information (man) pages, provide additional information.
Feeds
The feed command provides several utilities for creating and examining feed files. A feed file contains the definition and data of a single feed in a format that has been precompiled for efficient loading by a Decoder or Log Decoder. For a complete reference on feed definitions, see the "Feed Definitions File" topic in the Decoder Configuration Guide. Go to the NetWitness All Versions Documents page and find NetWitness Platform guides to troubleshoot issues.
create
feed create <definitionfile> [-x <password>]
The feed create command generates feed files for each feed defined in a feed definition file. A definition file is an XML document that contains one or more definitions. Each feed definition specifies a data file and the structure of that data file. The resulting feed files will be created in the same directory as the definition file with the same name as the data file, but with the extension changed to .feed (for example, datafile.csv results in datafile.feed). Any existing files with the target name will be overwritten without a prompt.
$ ls
example-definition.xml example-data.csv
$ NwConsole
RSA NetWitness Console
Copyright 2001-2020, RSA Security Inc. All Rights Reserved.
Type "help" for a list of commands or "man" for a list of manual pages.
> feed create example-definition.xml
Creating feed Example Feed...
done. 2 entries, 0 invalid records
All feeds complete.
> quit
$ ls
example-definition.xml example-data.feed example-data.csv
$Optionally, feed files can be obfuscated using the option -x followed by a password of at least 16 characters (no spaces). This will be applied to all feeds defined in the definition file. In addition to the feed file, a token file will be generated for each feed file. The token file must be deployed with the corresponding feed file.
feed create example-definition.xml -x 0123456789abcdef
stats
feed stats <feedfile>
The feed stats command provides summary information for an existing, un-obfuscated feed file. Specifying an obfuscated feed file will result in an error.
> feed stats example.feed
Example Feed stats:
version : 0
keys count : 1
values count: 2
record count: 2
meta key : ip.src/ip.dst
language keys:
alert Text
dump
feed dump <feedfile> <outfile>
The feed dump command generates a normalized, key-value pair listing of an un-obfuscated feed file. You can use the resulting file to validate a feed file or assist in determining which records were considered invalid when the feed was created. Specifying an obfuscated feed file will result in an error. If outfile exists, the command will abort without overwriting the existing file.
feed dump example.feed example-dump.txt
Converting Packet DB Files to PCAP
You can use the makepcap command to quickly convert any Packet DB file to a generic PCAP file, preserving the capture time order. This command offers many options (see help makepcap), but is easy to use. All it really needs is the Packet DB directory (with the source=<pathname> parameter) to get started.
Note: You must stop the Decoder or Archiver service before running this command. If you want to generate a PCAP while the service is running, see the packets command.
makepcap source=/var/lib/netwitness/decoder/packetdb
This command converts every Packet DB file into a corresponding PCAP file in the same directory. If the disk is almost full, see the next command.
makepcap source=/var/lib/netwitness/decoder/packetdb dest=/media/usb/sde1
This command writes all of the output PCAPs to the directory at /media/usb/sde1.makepcap source=/var/lib/netwitness/decoder/packetdb dest=/media/usb/sde1 filenum=4-6
This command only converts the files numbered 4 through 6 and skips all other files. In other words, it converts the Packet DB files: packet-000000004.nwpdb, packet-000000005.nwpdb, and packet-000000006.nwpdb.
makepcap source=/var/lib/netwitness/decoder/packetdb time1="2020-03-01 14:00:00" time2="2020-03-02 07:30:00" fileType=pcapng
This command only extracts packets with a timestamp between March 1st, 2020 at 2 PM and March 2nd, 2020 before or on 7:30 AM. It writes the file as pcapng in the same directory as the source. All timestamps are UTC.
Packets
You can use the packets command to generate a PCAP or log file based on a list of Session IDs, a time period, or a where clause. This command is very flexible, and you can use it on any running service that has access to the raw data from a downstream component. Before running the command, you must first login to a service and then change directory to the appropriate SDK node (for example, cd /sdk). Unlike the makepcap command, which only works on the local file system, you use this command for a remote service.
login ...
cd /sdk
packets where="service=80 && time='2020-03-01 15:00:00'-'2020-03-01 15:10:00'" pathname="/tmp/march-1.pcap"This command writes 10 minutes of HTTP-only packets from March 1st to the file /tmp/march-1.pcap. All times are in UTC.
packets time1="2020-04-01 12:30:00" time2="2020-04-01 12:35:00" pathname=/media/sdd1/packets.pcap.gz
This command writes all packets between the two times to a GZIP compressed file at /media/sdd1/packets.pcap.gz.
packets time1="2020-04-01 12:30:00" time2="2020-04-01 12:35:00" pathname=/media/sdd1/mylogs.log
This command writes all logs between the two times to a plaintext file at /media/sdd1/mylogs.log. Any pathname ending with .log indicates that the format of the output file should be plaintext line-delimited logs.
Verifying Database Hashes
By default, Archiver writes an XML file for every DB file that is written. This XML file ends with the extension .hash and contains a hash of the file along with other pertinent information. You can use the hash command to verify that the DB file has not been tampered with by reading the hash stored in the XML file and then rehashing the DB file to verify that the hash is valid. The command does accept wildcards, so something like /var/netwitness/archiver/database0/default/packetdb/*.hash will verify all hashes in that directory. You can also use the --output-pathname and --output-format parameters to write out the verification in json, xml or text formats for scripting purposes. If the DB files are not found in the location that the XML file says it should be in, then the command will attempt to find the DB files in the current directory or the directory where the hash file is found.
hash op=verify hashfile=/var/lib/netwitness/archiver/database0/alldata/packetdb/packet-000004880.nwpdb.hash
This command verifies that the Packet DB file packet-000004880.nwpdb still matches the hash in the XML file packet-000004880.nwpdb.hash. For proper security, the hash file should be stored somewhere else to prevent the XML file from being tampered with (such as write-once only media), but the hash command itself is not affected by where it is stored.
