Troubleshooting Decoder Packet Drops

[RSAAdmin]

Using freely available web tools*, we have created a browser application that leverages the REST interface of Decoder to determine when packet drops have occurred.  Once identified, on demand analysis graphs out the necessary system statistics and display the service logs to quickly view the state of the decoder at the time packet drops occur and improve the time necessary to make a determination of the root cause of the drops.

To properly use, the attached archive contains a two files the must be extracted placed on the decoder in the directory /etc/netwitness/ng/app.  Once in place, use a browser** to navigate to the path /sdk/app/reports on the decoder's rest port, no service restart necessary:

http://mydecoder:50104/sdk/app/reports

This will initiate a scan of the statistics database of the decoder and display packet drops as they are found.  This process may take a few minutes to complete, but once a drop occurrence is displayed, selecting will immediately populate the graph and log views (pictured above) with statistics and logs within the five minute window in which the drop occurred.

For this to properly work, the credentials used to login via the browser will need to have the sys.manage and logs.manage roles which are both present in the default groups Administrators and Operators.  If the page displays "Searching for drops: 0%" for an extended period or fails to retrieve logs when displaying graphs, it is likely a permissions issue.

To perform a targeted search for packet drops, the optional URL parameters time1 and time2 can be specified to limit the scope of the statistics database scan.  The value for these parameters can be of the form  YYYY-MM-DD hh:mm:ss (browser local time) or as a UNIX timestamp (seconds since 1970/1/1 00:00:00 GMT):

http://mydecoder:50104/sdk/app/reports?time1=1410235200

http://mydecoder:50104/sdk/app/reports?time1=2014-09-09%2000:00:00

http://mydecoder:50104/sdk/app/reports?time1=2014-09-09%2000:00:00&time2=2014-09-09%2001:00:00

Finally, to view the graphs and logs for a time range independent of a specific drop occurrence, the URL parameter analysis (with any value) can be included with both time1 and time2 specified (option is ignored if either are not present):

http://mydecoder:50104/sdk/app/reports?time1=2014-09-09%2000:00:00&time2=2014-09-09%2001:00:00&analysis=true

Note that this may take a significant amount of time for the browser to render depending on the time frame specified.

If the REST port of decoder in question is inaccessible from a workstation with a browser, but SSH access is available, an SSH tunnel can be setup to forward a local port on the workstation to the REST port on the decoder:

ssh -L 50104:localhost:50104 user@mydecoder

and use the same URL's presented above with the hostname replaced with localhost:

http://localhost:50104/sdk/app/reports

Please be on the lookout for a followup posting on how best to interpret the results of this tool and apply that knowledge to resolving packet drop issues.

* - dygraphs, Bootstrap, and jQuery (a dependency of Bootstrap)

** - Developed and tested using Chrome version 37, little cross browser testing performed

[RSAAdmin]

Hi Becca-

I've attached sessions.html which is a (very) rough example of how you can graph arbitrary stats (on any service).  You'll need the attached common.js file installed, drop sessions.html in /etc/netwitness/ng/app and point a browser at /sdk/app/session.

If you've looked at any of these files, they are strictly browser based logic making REST calls back to the decoder.  You can easily use this model to display any data exposed via REST however you please.  The only real restriction atm is that we have only opened up the three URL's (/sdk/app/reports, /sdk/app/sessions, and /sdk/app/common.js) to serve content.