Filtering Redundant Values from REST API Query

[Anonymous]

I'm using a very simple query to pull back all hosts from NetWitness for offline analysis.  You can see a small sample of the results below.  The query for alias.host returns all values, which have many redundant entries.

The same query using Investigator will not display the redundant entries but rather show each alias.host entry only once.

Is there any way to duplicate this using the REST API?

Thanks!

[Anonymous]

Hi Rob,

Yes, there is. Look at the values message instead of the query one.

Something like /sdk?msg=values&size=1000&fieldName=alias.host&where=<yourwhereclause>&flags=2305

To sort by descending count as in Investigator.

Flags are described in the NwSDK documentation and copied below for ease of access.

Flags = 0x0001 - Total session count

Flags = 0x0002 - Total session size in bytes

Flags = 0x0004 - Total packet count

Flags = 0x0008 - Total field count

Flags = 0x0100 - Sort by Total

Flags = 0x0200 - Sort by Value

Flags = 0x0400 - Order Ascending

Flags = 0x0800 - Order Descending

Flags = 0x8000 - Estimate Total

Hope that helps!

Rui

[Anonymous]

Rui,

Thanks for the response.  Using your method versus a query seems to time out with simple request to bring back 100 values.  Using a WHERE clause to restrict the time to an hour window didn't seem to make a difference either.

http://XX.XX.XX.XX:50105/sdk?msg=values&size=100&fieldName=alias.host&flags=2305

334829	2012-Dec-09 13:59:11	SDK-Values	audit	User admin (session 3083966, XX.XX.XX.XX:63913) has issued values (pipe 3084063): flags=2305 fieldName=alias.host size=100
334830	2012-Dec-09 13:59:42	Rest	audit	User admin (session 3083966, XX.XX.XX.XX:63913) has a running REST request that has timed out, returning HTTP 408
334831	2012-Dec-09 13:59:44	SDK-Values	audit	User admin (session 3083966, XX.XX.XX.XX:63913) has canceled values (pipe 3084063, elapsed time 00:00:33): flags=2305 fieldName=alias.host size=10

http://XX.XX.XX.XX:50105/sdk?msg=values&size=1000&fieldName=alias.host&where+time+%3D+%272012-Dec-09+08%3A00%3A00%27-%272012-Dec-09+09%3A00%3A00%27+%26%26+service%3D80&flags=2305

334870	2012-Dec-09 14:09:35	SDK-Values	audit	User admin(session 3085244, XX.XX.XX.XX:63919) has issued values (pipe 3085250): flags=2305 "where time = '2012-Dec-09 08:00:00'-'2012-Dec-09 09:00:00' && service=80" fieldName=alias.host size=1000
334874	2012-Dec-09 14:10:06	Rest	audit	User admin(session 3085244, XX.XX.XX.XX:63919) has a running REST request that has timed out, returning HTTP 408
334875	2012-Dec-09 14:10:08	SDK-Values	audit	User admin(session 3085244, XX.XX.XX.XX:63919) has canceled values (pipe 3085250, elapsed time 00:00:33): flags=2305 "where time = '2012-Dec-09 08:00:00'-'2012-Dec-09 09:00:00' && service=80" fieldName=alias.host size=1000

The query returned values immediately, even for 10-20,000 size settings.

Regards,

Rob

[Anonymous · Accepted solution]

Rob,

Try adding "&expiry=0" to the URL, that should avoid the timeouts. I'm not sure why your query is causing a timeout even for samll time windows.

Hope that helps!

Rui

[Anonymous]

Rui - you nailed it!

The lack of the "&expiry=0" was the issue.  The values lookup performance versus the query lookup performance is very different.  Since the host.alias is indexed and the lookup was over a short timeframe, I'm very surprised at the time the lookup took to complete.  The query was MUCH faster, although it did return duplicate values.

Example URL bringing back 20,000 values for service=80 and timeframe 12/8/12 08:00 - 12/9/12 09:00

http://concentrator.com:50105/sdk?msg=values&size=20000&fieldName=alias.host&where+time+%3D+%272012-Dec-08+08%3A00%3A00%27-%272012-Dec-09+09%3A00%3A00%27+%26%26+service%3D80&flags=2305&expiry=0

Also if you are reading this thread and it's helpful, post some useful examples of using the API for others to learn

Thanks - Rob