detailed.stats not taking effect

[RSAAdmin]

Hello,

For our rather large netwitness 9.8 infrastructure, I have written quite a few scripts.  One of which is intended to audit the various feeds, parsers, etc - ensuring that each decoder has deployed what we intend it to have.  I use the "REST" API (it's not truly REST, but hey it works...) to accomplish this and many other things.  The only way (that I know of) to enumerate the parsers for each decoder is to set the variable /decoder/parsers/config/detailed.stats to "on", wait a few seconds while the list populates, then look at /decoder/parsers/definitions to see the list.  with detailed.stats set to the default "off", the URL /decoder/parsers/definitions gives a 404.  with it "on" that URL gives a list of queryable nodes.  after polling, I return the variable to its original "off" state to conserve resources.  Works great, and has for the past couple of years.

Recently, I've noticed that some of our decoders (9.8.5.x) are either silently ignoring the value of detailed.stats, or are silently failing to populate that list.  I can see in the logs "user has changed detailed.stats to 'on'", etc.  I can verify that the variable is indeed set to "on" as it should be, but /decoder/parsers/definitions still results in a 404.  I've tried waiting several minutes, even several hours for the list to populate, but it never does.  Service restarts, even a reboot, nothing seems to make this bit start working again.  The obvious course of action is to look for something that changed between working and not working... but from what I can tell, nothing has changed.

So - has anyone run across this before?  Any recommendations for a fix?  Thanks in advance!

Josh

edited to add: some of our decoders, running the exact same 9.8.5 release, work as expected.  there seems to be no rhyme nor reason for which ones have developed this little quirk.  it is currently affecting only a few decoders.

[RSAAdmin]

I've never heard of such a problem.  Do the detailed stats show up in Explorer View of the Administrator client?  Can you do a "ls" of them in NwConsole?  Exactly what version of 9.8 are you running?  The latest release is 9.8.5.20.  I know early releases of 9.8 had some issues with the REST interface.

Another way to get the list of parsers is to call /decoder/parsers?msg=schema.  Between that and the parsers.disabled config node you can get the list of enabled parsers without using detailed stats.

[RSAAdmin]

the ones that are experiencing this issue are 9.8.5.15 but in fairness they did work before, on that same version.  Others that are running .19 and .20 are not having this problem... which is not to say they won't in the future.  I think that part is probably pure coincidence.  The result is the same whether I use the web service, NwConsole, or the admin client.  The problem seems to be internal, but it's hard to be sure.

I will look into using /decoder/parsers/msg=schema.  It'll mean some new code, but it might just be more reliable, so I'm game.  at first glance, the output from that command (on one of the affected decoders) does appear to contain most of the info I need.  What I'm not seeing is the actual filename (i.e. supercoolstuff.parser or findthebadguys.lua or whatever).  The filename is part of what I get from the detailed stats, and is integral to how we handle automated deployment of custom stuff - is there a way to get that bit of info from the schema, or somewhere besides detailed.stats?

[RSAAdmin]

The filenames can be had with the content call, on the same parsers node.

[RSAAdmin]

well right... but that can't be reliably tied to the other data gotten from the 'schema' call.  one of the things we like to see is which parsers are using which meta.  in our system of management scripts, all of these things tie back to the filename, including the file itself (obviously).  it has helped tremendously to ensure that all of our devices are configured as expected and to identify any shortcomings with custom parsers, feeds, etc.  with detailed.stats, it's not difficult at all to tie all of that together - that part is already done.  I'm not sure that what is in the 'schema' and 'content' calls is enough.  I'll keep digging around though.

[RSAAdmin]

Please post the output of a "ls" command on the /decoder/parsers node on a system that should be showing detailed stats but is not.  Please pass depth=10 to the ls command so we get a list of everything recursively.

Thanks

[RSAAdmin]

Can you verify whether or not your content is consistent across all decoders, both those presenting this issue and not?

My thoughts are that there may be an issue building the definitions tree with a specific parser, feed, or application rule that is not present across all of your decoders.

The quickest way to check this would be to remove all parsers, feeds and application rules from one of the affected decoders (after backing them up) and perform a parser reload or service restart if that doesn't work.

If this is the case, and you are able to identify a specific content item that is causing this behavior, please share as much detail as possible so that we can replicate and address this issue.

[RSAAdmin]

well the funny thing is, I left detailed.stats on overnight, and when I came in this morning the definitions list was populated.  I did the same thing the night before though, and it wasn't populated in the morning.  Today I turned detailed.stats off again, waited a few minutes, turned it back on.  This time the definitions list was populated within the normal few seconds.  I have no idea what kicked it into gear, but something did... so troubleshooting is going to get a bit more difficult.

FWIW though, yesterday I did an 'ls' and only got the expected 3 entries (all but definitions).

edit: and yes - our content is identical across each of our environments.  That's one of the things my management system ensures, and one of the things this feature is needed to verify in an automated fashion. 

[RSAAdmin]

so, the issue is back... an "ls depth=10" results in a lot of output about feeds, plus the various config options but no definitions node and of course nothing deeper in that tree.  I'm not comfortable posting the exact output of it on a public forum, but if there is something specific you're looking for, please let me know.  It does look identical to the output with detailed.stats set to off, with the only difference being that it reflects detailed.stats as "on"