One of the less common use cases for NetWitness is to process PCAP data that has been collected elsewhere for analysis. That is, the Decoder is not plugged into a TAP or SPAN port that is collecting live data. Instead, PCAP files can be fed into the Decoder a number of ways but file transfers happen over the management interface... not a capture port. For this to work the Decoder is set to Stop Capture, as below.
Importing a PCAP manually from the Administrator tool is possible by clicking on the Import Packets button. This is great for a small numbers of PCAP files but it doesn't work well out-of-hours or when you want to take leave. An automated, scripted process for importing PCAP files is needed in these situations.
Automating PCAP importing
Other than the Administrator tool, there are a number of other ways to import PCAP files in NetWitness that can potentially be used in a script.
REST API import
The first method is the use of the REST API to upload PCAP files from literally anywhere. This method has the benefit of being very easy to script as it uses html commands to the REST port (50104) on a Decoder.
# curl -u "admin:netwitness" -F "fileupload=@data.pcap" "http://DecoderIP:50104/decoder/import"
<?xml version="1.0" encoding="UTF-8"?>
<import>
<data filename="data.pcap" packets="5230072" size="744026236">Success</data>
</import>
There is only one drawback to using this method that I can see and that is the PCAP file doesn't create meta associated with the filename and path. This was an optional but useful feature of using the Administrator tool to import data.
NwConsole
The other method I am aware of is the use of the NwConsole command.
[root@NWDECODER ~]# NwConsole
NetWitness Console 9.8.5.9
Copyright 2001-2012, NetWitness Corporation. All Rights Reserved.
> login localhost:50004 admin netwitness
Successfully logged in as session 10596
[localhost:50004] /> import data.pcap
Sending packets to Decoder from file "data.pcap"
... 20% ... 40% ... 60% ... 80% ... 100%
[localhost:50004] />
My question to the NW community is have you addressed this issue and come up with an elegant solution you can share? It would be great to hear of your approaches and any tricks you learnt along the way. Also, if anyone knows a magic REST API tag that will enable the Track Filename functionality please let me know too.
Thanks,
1 ACCEPTED SOLUTION
Accepted Solutions
If anyone needs to perform the unnatural act of batch PCAP ingestion from a Windows box here is a useful script. You will need to get hold of a Windows compiled curl binary from here.
e.g. c:\upload.bat
@echo off
setlocal enableDelayedExpansion
set MYDIR=c:\pcaps\
for /F %%x in ('dir /B/D %MYDIR%') do (
set FILENAME=
%MYDIR%\%%xecho +======= Uploading !FILENAME! to Decoder =======+
curl.exe -u admin:netwitness -F fileupload=@!FILENAME! http://DecoderIP:50104/decoder/import
)
@
Running this on PCAP files in a directory will return output like:
Make sure you don't upload any PCAP file larger than 3.99GB in size or the Decoder service seems to stop. This is due to a buffer limitation that can possibly be changed somewhere. However, uploading 4GB PCAP files is asking a bit much either way. Slice and dice larger PCAP files with editcap first.
e.g. "c:\Program Files\Wireshark\editcap.exe" -F libpcap -i 86400 Big.pcap Smaller_pcaps
