NwLogDecoder Failed to start capture

[Anonymous]

Our LogDecoder refuse to work today. When we  "Start Capture" an Initialization Error occurs.

Failed to start capture: Failed to process message start for /decoder com.rsa.netwitness.carlos.transport.TransportException: terminated

[Anonymous]

Is it having database issues?

in 10.4 you are now able to access the services before they are fully started.  I see that it has only been up for one minute so it might still be opening the database files depending on the size of your data. 

[RSAAdmin]

The logs generated by Log Decoder need to be checked to see if there are any warnings or errors.

Examples:

grep "warning\|failure" /var/log/messages

tail -f /var/log/messages "warning\|failure"

You can do the same thing using SA-UI as well by going to Log Decoder > Logs > Historical and then filtering the logs once by "WARN" and another time by "ERROR".

Usually, when an "Initialization Error" happens in Log Decoder, the reason behind it can be explained by seeing the warning and failure logs that were generated by Log Decoder while the Log Decoder was starting.

Thanks,

Susam

[Anonymous]

There were really appropriate entries in the syslog. So I conducted a data reset, now it works.

Here are the appropriated syslog entries.

Jan 16 16:30:11 logdecoder nw[28920]: [meta] [warning] There are core files taking up 2.18 GB on the partition /var/netwitness/logdecoder/metadb. Please open a support ticket to troubleshoot.

Jan 16 16:30:11 logdecoder nw[28920]: [Database] [failure] One of the databases (session or meta) is missing data.  To correct this error, first make sure the databases are configured properly and the drives are correctly mounted.  If this is the case and there's very little data in one of the databases, perform a data reset to correct.

Jan 16 16:30:11 logdecoder nw[28920]: [Engine] [warning] Module logdecoder failed to load: One of the databases (session or meta) is missing data.  To correct this error, first make sure the databases are configured properly and the drives are correctly mounted.  If this is the case and there's very little data in one of the databases, perform a data reset to correct.

Jan 16 16:30:11 logdecoder nw[28920]: [Engine] [warning] Module logdecoder failed to load: Diagnostic information: Throw in function bool nw::AssemblerDatabase::trimCorruption(bool)Dynamic exception type: N5boost16exception_detail10clone_implIN2nw9ExceptionEEEstd::exception::what: One of the databases (session or meta) is missing data.  To correct this error, first make sure the databases are configured properly and the drives are correctly mounted.  If this is the case and there's very little data in one of the databases, perform a data reset to correct.[PN5boost16errinfo_at_line_E] = 1366

Jan 16 16:30:11 logdecoder nw[28920]: [ObjectStore] [warning] The file '/var/netwitness/logdecoder/statdb/stats-000000096.statsdb' was not properly closed.

[JonathanSaxon · Accepted solution]

The logs you sent indicated that there are CORE files located in /var/netwitness/logdecoder/metadb.  This indicates that the nwlogdecoder service is crashing and generating core dumps and filling up that filesystem.

While performing a data reset did wipe all the data on the appliance, it won't have deleted the core files in that folder. 

You will want to open a support case to see if the service is crashing due to a known issue and whether there is a hot fix already released to address the cause of the core dumps.

[huanzhou1]

Did you see red color "Initialization error" ? I think missing database path caused the issue. How you reset your data? Did you reset config as well?

[Anonymous]

As you can see in my post from 16.01.2015 7:39, there was the red "Initialization error".

I've done the reset for all databases

reset data=1 index=1 log=1  stats=1 config=1

After that I stopped the NWLogDecoder service via shell command and cleaned the directories:

/var/netwitness/logdecoder/index

/var/netwitness/logdecoder/metadb

/var/netwitness/logdecoder/packetdb

/var/netwitness/logdecoder/sessiondb

/var/netwitness/logdecoder/statdb

So that all core.* files are purged.

After all I've done a restart of the NWLogDecoder service.

Do you have a blurb "How to repair a corrupt NetWitness Core Database"?

[JonathanSaxon]

You can use the NwConsole tool to check databases.  There is a dbcheck function that will check various databases.

For more details see KB article 26605 at https://rsaportal.force.com/customer/articles/How_To/a59828-How-to-manually-run-dbcheck-on-an-RSA-NetWitness-appliance?popup=false&navBack=H4sIAAAAAAAAAIuuVipVslLSTy4tLsnPTS3Sjy_N1M_Oyy_PSU1JT9UHcrxhHA-gvH1xamJRcoatkZmZgamSjlIxUC-KAqBYNlCsIDE9NSSzJCd...

[Anonymous]

Thanks a lot. I will try it.

"NextGen 9.5" was not realy my focus at SCOL search 😉

[JonathanSaxon]

It is possible that the KB article needs an update. But this tool is available on Security Analytics core appliances as well as NextGen core appliances.