Troubleshooting Decoder Packet Drops

[RSAAdmin]

Using freely available web tools*, we have created a browser application that leverages the REST interface of Decoder to determine when packet drops have occurred.  Once identified, on demand analysis graphs out the necessary system statistics and display the service logs to quickly view the state of the decoder at the time packet drops occur and improve the time necessary to make a determination of the root cause of the drops.

To properly use, the attached archive contains a two files the must be extracted placed on the decoder in the directory /etc/netwitness/ng/app.  Once in place, use a browser** to navigate to the path /sdk/app/reports on the decoder's rest port, no service restart necessary:

http://mydecoder:50104/sdk/app/reports

This will initiate a scan of the statistics database of the decoder and display packet drops as they are found.  This process may take a few minutes to complete, but once a drop occurrence is displayed, selecting will immediately populate the graph and log views (pictured above) with statistics and logs within the five minute window in which the drop occurred.

For this to properly work, the credentials used to login via the browser will need to have the sys.manage and logs.manage roles which are both present in the default groups Administrators and Operators.  If the page displays "Searching for drops: 0%" for an extended period or fails to retrieve logs when displaying graphs, it is likely a permissions issue.

To perform a targeted search for packet drops, the optional URL parameters time1 and time2 can be specified to limit the scope of the statistics database scan.  The value for these parameters can be of the form  YYYY-MM-DD hh:mm:ss (browser local time) or as a UNIX timestamp (seconds since 1970/1/1 00:00:00 GMT):

http://mydecoder:50104/sdk/app/reports?time1=1410235200

http://mydecoder:50104/sdk/app/reports?time1=2014-09-09%2000:00:00

http://mydecoder:50104/sdk/app/reports?time1=2014-09-09%2000:00:00&time2=2014-09-09%2001:00:00

Finally, to view the graphs and logs for a time range independent of a specific drop occurrence, the URL parameter analysis (with any value) can be included with both time1 and time2 specified (option is ignored if either are not present):

http://mydecoder:50104/sdk/app/reports?time1=2014-09-09%2000:00:00&time2=2014-09-09%2001:00:00&analysis=true

Note that this may take a significant amount of time for the browser to render depending on the time frame specified.

If the REST port of decoder in question is inaccessible from a workstation with a browser, but SSH access is available, an SSH tunnel can be setup to forward a local port on the workstation to the REST port on the decoder:

ssh -L 50104:localhost:50104 user@mydecoder

and use the same URL's presented above with the hostname replaced with localhost:

http://localhost:50104/sdk/app/reports

Please be on the lookout for a followup posting on how best to interpret the results of this tool and apply that knowledge to resolving packet drop issues.

* - dygraphs, Bootstrap, and jQuery (a dependency of Bootstrap)

** - Developed and tested using Chrome version 37, little cross browser testing performed

[huanzhou1]

thanks for sharing.

Just wondering this one will overwrite the default reports page, how to make them coexist?

Thanks.

[RSAAdmin]

The default /sdk/app/reports page is served from the process and the functionality can be restored by removing or renaming the installed /etc/netwitness/ng/app/reports.html file.  At this time there is no way to use the packet drops tool without overriding either /sdk/app/reports or /sdk/app/sessions (rename /etc/netwitness/ng/app/reports.html to /etc/netwitness/ng/app/sessions.html).

It is our plan to incorporate this tool into the decoder in a subsequent release so that overriding these existing pages will no longer be necessary.

[huanzhou1]

thanks for the advise.

Some customer may block access to the rest port which mean they only allow port 443 to SA server, is there any way to use this tool from SA UI?

[RSAAdmin]

They can't block access to the REST port for 10.3 and earlier.  SA will not work without the REST port.  Only SA 10.4 works without the REST port enabled.

But to answer your question, currently this is just a javascript tool that uses the REST port to gather the drop metrics and display them for convenient analysis.  There isn't any integration with SA.  But it's not a bad idea.

[Anonymous]

Can't find the /etc/netwitness/ng/app
directory at our SA HybridPacketDecoder (10.3.4.2634-5.el6).

There are just this directories:

/etc/netwitness/ng

./decoder_truststore
./upload
./parsers
./logparsers
./appliance
./appliance/storedproc
./decoder
./decoder/storedproc
./warehouseconnector
./warehouseconnector/storedproc
./feeds
./warehouseconnector_truststore
./appliance_truststore
./lockbox

Could you give me a hint where's the best for storing the reports.html?

[RSAAdmin]

David-

If the app directory doesn't exist, you can create it and place the files in the newly created directory.

Please let me know if you have any other questions or issues.

eq

[Anonymous]

after creating "/etc/netwitness/ng/app" and copying both files into the request:

https://192.168.0.10:50104/sdk/reports

throws error "url not found".

Do you have any idea?

[RSAAdmin]

Correct url is /sdk/app/reports

[RQuinn]

This is some pretty cool stuff. How easy would it be to do additional modifications to see device performance over time as well?

For example - my hybrid is a busy little box. I would love to trend its activity over a 24 hour period to see how it is behaving and identify performance spikes of concern. I would also like to see the rate of packets/sessions over time. I know my internal traffic is increasing slowly and my retention rate is decreasing, but if I could graph these changes over time I can better estimate my storage retention and plan for future growth.

-becca