REST API to reload parsers / feeds?

[Anonymous]

Hi all,

I inherited a Netwitness infrastructure (now at version 10.3.2) - and part of the desired processing is

downloading new feed CSV files, massaging them and creating new custom feeds.  We are using

Python to process this on a Windows server.

After the feeds have been uploaded to the correct Informer directory, the script sends a reload message

to the decoders.  This doesn't look like it's working - here's the line of Python code:

urllib.urlopen('http://' + adminID + ':' + adminPW + '@' + ip_address + ':50104/parsers?msg=feed&op=reload')

I've spent some time in a browser going through decoder/parser/  REST API, and not getting anywhere.

I think the "msg=feed" parameter is correct, but I'm not sure about the "op" parameter.

Here is what the GUI is showing for "op".

op - <string, optional, {enum-one:messages|parameters|description|values|roles|extra}> The specific help operation to perform

I can't seem to get a list of permissible values for this.

Any tips for further GUI exploration of REST or permissible values for "op" ?

thanks.

[Anonymous]

http://192.168.99.51:50104/decoder/parsers?msg=reload

parser reload via rest

http://192.168.99.51:50104/decoder/parsers?msg=feed&op=notify

feed reload via rest

[huanzhou1]

[root@NWAPPLIANCE30922 ~]# curl -v http://admin:netwitness@127.0.0.1:50104/decoder/parsers?msg=reload

* About to connect() to 127.0.0.1 port 50104 (#0)

*   Trying 127.0.0.1... connected

* Connected to 127.0.0.1 (127.0.0.1) port 50104 (#0)

* Server auth using Basic with user 'admin'

> GET /decoder/parsers?msg=reload HTTP/1.1

> Authorization: Basic YWRtaW46bmV0d2l0bmVzcw==

> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.14.0.0 zlib/1.2.3 libidn/1.18 libssh2/1.4.2

> Host: 127.0.0.1:50104

> Accept: */*

>

< HTTP/1.1 200 OK

< Content-Length: 130

< Connection: Keep-Alive

< Pragma: no-cache

< Expires: -1

< Cache-Control: no-cache, no-store, must-revalidate

< Content-Type: text/xml; charset=utf-8

<

<?xml version="1.0" encoding="utf-8"?>

<response flags="1073872897">

        <string>The parsers have been reloaded</string>

</response>

* Connection #0 to host 127.0.0.1 left intact

* Closing connection #0

[root@NWAPPLIANCE30922 ~]#

[huanzhou1]

[root@NWAPPLIANCE30922 ~]# curl -v 'http://admin:netwitness@127.0.0.1:50104/decoder/parsers?msg=feed&op=reload'

* About to connect() to 127.0.0.1 port 50104 (#0)

*   Trying 127.0.0.1... connected

* Connected to 127.0.0.1 (127.0.0.1) port 50104 (#0)

* Server auth using Basic with user 'admin'

> GET /decoder/parsers?msg=feed&op=reload HTTP/1.1

> Authorization: Basic YWRtaW46bmV0d2l0bmVzcw==

> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.14.0.0 zlib/1.2.3 libidn/1.18 libssh2/1.4.2

> Host: 127.0.0.1:50104

> Accept: */*

>

< HTTP/1.1 200 OK

< Content-Length: 133

< Connection: Keep-Alive

< Pragma: no-cache

< Expires: -1

< Cache-Control: no-cache, no-store, must-revalidate

< Content-Type: text/xml; charset=utf-8

<

<?xml version="1.0" encoding="utf-8"?>

<response flags="1073872897">

        <string>All feeds flagged to be reloaded.</string>

</response>

* Connection #0 to host 127.0.0.1 left intact

* Closing connection #0

[root@NWAPPLIANCE30922 ~]#

[Anonymous]

Thanks guys - both helpful answers, but my original question was about REST documentation.

The RSA folks all say "browse the rest api on a machine and click on the asterisk for help" -

but I couldn't seem to get the valid "parameters" or "values".

The same thing happens in the Python/TK REST explorer that some Netwitness engineer coded.

re:Patriot3w -

Thanks for the CURL example!  I'm having a heck of a time trying to get pycurl to work on my Windows Informer

box, but I can just call "curl.exe" from Python, and it should work perfectly to reload the feeds.

[Anonymous · Accepted solution]

In that case, you could just issue the 'help' call.

http://192.168.99.51:50104/decoder/parsers?msg=feed&op=help

The output:

Bad Request

Parameter op: The value 'help' is invalid, must be one of the values 'notify|reload|remove|delete'

The possible values are listed in the message.

I also do this from NwConsole.  I haven't tried it from EVERY node.  Much of what I had done via NwConsole, I do now via REST.  Much of what I had done via REST I stored in a text file and use as a reference.  This was done based on trial and error.

I am attaching that notes file here.  It isn't 'documentation'...just a brain dump of what worked for me.

Chris

[Anonymous]

ahearc - you da man!

Thanks SO much for the REST doc. 

Of course I plan to keep and expand it as I explore more areas of NW.

Always trying to automate more and more to speed up our IR process.

Gotta love curl command line.

I solved a sticky problem in REST by calling curl command line from Python.

warm regards,

don

[Anonymous]

op=notify

--

Chris Ahearn

[Anonymous]

http://192.168.99.51:50104/decoder/parsers?msg=reload

parser reload via rest

http://192.168.99.51:50104/decoder/parsers?msg=feed&op=notify

feed reload via rest

--

Chris Ahearn

Senior Security Practice Consultant

RSA Security Analytics

Cell: 845.461.5985

[Anonymous]

Actually…as I re-read this a third time, are the feeds first compiled as a feed or are they just CSV’s?

You could create a recurring feed task in SA Live custom feed wizard to point to the CSV via an http url.

In order for the feed op=notify to really work, the compiled feed (filename.feed) needs to be in the /etc/netwitness/ng/feeds directory first.

It wasn’t clear from the original post if the file was on the decoder or not.

Chris

--

Chris Ahearn

Senior Security Practice Consultant

RSA Security Analytics

Cell: 845.461.5985