Disabling automatic parser reload for REST file upload in 9.8

[RSAAdmin]

Hello,

I'm in the process of automating the deployment of some of our custom parsers and feeds.  I see that the REST API apparently allows for the automatic reload to be disabled, but I cannot make that work.  Even if using the NW-built web form at <decoder>:50104/decoder/parsers/upload and leaving the "reload" checkbox turned off, the device still logs "Issuing automatic parser reload" and reloads all parsers.

I've tried to find a real REST API document, but haven't found anything that details the things I need.  I first thought that I could use /decoder/parsers?msg=upload&op=start, sending the file in the POST body... but that didn't work.  I *can* successfully send the file to /decoder/parsers/upload but then all attempts to disable the auto reload have failed.

Can someone please point me to a real API document, or perhaps an example of how to disable this automatic reload?

Thanks!

Josh

[RSAAdmin]

And before anyone replies to look at the REST API PDF from the User Conference, I already have.  It doesn't cover this subject, unfortunately.  It also does not cover the usage of /decoder/parser?msg=upload (nor does anything else I can find).  Thanks in advance!

[RSAAdmin]

Does anyone have any input on this?  This REST API seems very powerful but so much of it is apparently completely undocumented.  I can't tell if this piece is working as intended or if it's broken... because I have no way to know exactly how it's supposed to be used.  Surely someone involved with development of the REST API read these posts...

[RSAAdmin]

It's working as intended.  The reload command is meant to reload immediately.  If you don't issue a reload, it will wait to see if more parsers are uploaded within a small amount of time.  If no additional feeds/parsers are uploaded within the time window, it will issue the reload automatically.

There's no point to updating parsers and feeds on the decoders if they aren't reloaded.  The automatic reload is to prevent customers from adding new content but forgetting to reload.  It's also meant to keep reloads to a minimum (reloads are expensive), which is why there's a timer to automatically reload after uploads stop.

[RSAAdmin]

That timer must be less than a second.  If I upload two files in a row, it issues an "automatic reload" after each.  What I'd like to do is *disable* that automatic reload.  There is a point to doing that, since I want to perform several operations at once, including uploading several parsers/feeds.  After which I'll manually issue one reload.

The help message says that you can set reload to 0 on a "finished" op... but then I can't find a way to successfully issue a finished op.  I can issue a start op and get a "Success" response, but any finished op results in an error message.  Something to the tune of "no files were uploaded on pipe xxxxxxxx".

[RSAAdmin · Accepted solution]

Hi Josh-

What version of 9.8 are you using?  I have verified that 9.8.5.14 works as intended using /decoder/parsers/upload with out setting the reload argument.

To be a little more specific, once a file is received (upload completed), decoder will delay the reload by 3 seconds to provide a window in which additional content can be received.  Each subsequent upload within this window will push the delay another 3 seconds from the completion of the upload. This can continue up until 1 minute from the initial upload at which point a reload is issued regardless of subsequent content upload.

There is no way to disable or alter the parser reload time windows.

The /decoder/parsers/upload REST interface accepts content as multipart/form-data (as used by the html page) or application/zip.  The multipart/form-data has a single part with the following headers:

Content-Disposition: form-data; name="feed/parser"; filename="test.lua"

Content-Type: application/octet-stream

The application/zip requires that your content be placed within a flat zip file (no directory structure within the archive) and posted with a content type of application zip.  The posted zip file can only contain a single piece of content (feed, parser, etc...).  The following demonstrates using curl to upload a zip file:

curl -u admin:netwitness --verbose  --header "Content-Type: application/zip" --data-binary @test.zip "http://localhost:50104/decoder/parsers/upload"

The following logs show the upload process for the above curl command including the 3 second reload delay:

id=495493  time=1383330807  level=audit  module=Engine  msg=User admin (session 596, 127.0.0.1:49855) has logged in

id=495494  time=1383330807  level=info  module=Rest  msg=Received 3971-42cb-0513-22a8.zip of size 586 for upload

id=495495  time=1383330807  level=audit  module=Parse  msg=User admin (session 596, 127.0.0.1:49855) has started uploading file '3971-42cb-0513-22a8.zip'

id=495496  time=1383330807  level=audit  module=Parse  msg=User admin (session 596, 127.0.0.1:49855) has finished uploading file '3971-42cb-0513-22a8.zip'

id=495497  time=1383330807  level=audit  module=Parse  msg=User admin (session 596, 127.0.0.1:49855) has finished uploading file 'test.lua'

id=495498  time=1383330807  level=audit  module=Engine  msg=User admin (session 596, 127.0.0.1:49855) has logged out

id=495499  time=1383330810  level=audit  module=Parse  msg=Issuing automatic parser reload

I hope that helps.  Please let me know if you have any additional questions.

Edward Quackenbush

[RSAAdmin]

I guess I just wasn't looking closely enough.  The problem was that I was waiting for the success response and doing some processing on the back end before sending the next file.  It was indeed taking about 3 seconds.  I changed that to make sure the next upload starts less than 3 seconds after the previous one ends and only one automatic reload is issued.

I do wish that there was a way to control this myself (as the help message seems to indicate), but I'm satisfied with the 3 second window.  At least I now know what it is.

Thanks!

Josh