Decoder Stuck Initializing Database

[MichaelPochan]

Has anyone ever had packet decoders (10.6.2) stuck in the initializing database phase for more than 24 hours? We have two that are going on 26 hours now in this state. Rebooting the boxes and even restarting the nwdecoder processes hasn't worked either. 

[NaushadKasu1]

Are you talking about in the UI it shows in yellow/orange that it's initializing? Because if so, that's a UI bug and the Decoder is actually working fine. You can get rid of the yellow icon by clicking the service, edit, test connection, ok/save and it will go green. Note, the licensing will go red for about 30-45 secs while it syncs up as well but will go green eventually. 

[MichaelMcGillic]

Hello Michael:

As a follow-up to what Naushad posted, if the issue is as he described, you may see the the orange status come back (usually after a restart of the service). RSA Engineering is aware of the issue and this is being addressed in the 10.6.4.0 release.

If you are seeing a different issue, I would recommend getting a Technical Support case opened so one of our Engineers can review your environment to determine the underlying issue.

Thank you.

[MichaelPochan]

Yeah, I don't think this is the UI bug. Checking nwdecoder status from a shell session produces this output: 

status nwdecoder
nwdecoder start/running

Deactivating the health policies then re-enabling them only temporarily clears the alarms. I'm seeing associated alarms for the concentrators also since they're not aggregating any meta from the decoders: 

[NaushadKasu1]

What if you restart the nwdecoder service via "restart nwdecoder"?  Are there errors in the logs?  Tail the /var/log/messages and see if any errors show or if there is informational message about it rebuilding the index which sometimes could take awhile if the DB is in an error state and index is being rebuilt -- the logs would show you ETA to uptime in that case.

[MichaelPochan]

Thank you! After much testing, it looks like there are synchronization issues between the meta and session databases. Nwdecoder will run for a few minutes, then will get killed by an ABRT signal. Cannot discern anything further so will hopefully be working with support at some point in the future. 

NwDecoder[133138]: [Database] [warning] The session and meta databases are out of sync by 744714 entries
NwDecoder[133138]: [Database] [info] Database file /var/netwitness/decoder/metadb/meta-000006843.nwmdb found an error: Do you want to remove the FILE_NOT_CLOSED

init: nwdecoder main process ended, respawning
init: nwdecoder main process (9278) killed by ABRT signal
init: nwdecoder main process ended, respawning
init: nwdecoder main process (4810) killed by ABRT signal