The NextGen or Security Analytics core appliance experiences Initialization Errors when the service starts and the service fails to start or capture or aggregation fail to start.
This may be due to corrupted meta, session or packet database files on the core appliance.
Use the steps in this article to identify and fix corrupted database files using the dbcheck feature of the NwConsole tool.
Background
- The Decoder/LogDecoder service has the following databases: sessiondb (nwsdb), metadb (nwmdb), packetdb (nwpdb) and NwServerLog (log)
- The Concentrator service has the following databases: sessiondb (nwsdb) & metadb (nwmdb) and NwServerLog (log)
- The Appliance service has the following databases: NwServerLog (log)
- 9.8 also adds: statdb for appliance and service (statsdb)
Note: Appliances DACs may have multiple packetdb file systems. The first DAC will be found /var/netwitness/decoder/packetdb0. Subsequent DACs will be found at /var/netwitness/decoder/packetdb1, etc. Prior to running dbcheck, you should assess the contents of /var/netwitness to determine which database files you want to verify.Running NwConsole and dbcheck
- SSH to the appliance as root
- Stop services
For CentOS 5.x-based appliances
For a decoder appliance:
monit stop nwdecoder
For a concentrator appliance:
monit stop nwconcentrator
For a hybrid appliance:
monit stop nwdecoder; monit stop nwconcentrator
For CentOS 6.x-based appliances
For a decoder appliance:
stop nwdecoder
For a concentrator appliance:
stop nwconcentrator
For a hybrid appliance:
stop nwdecoder; stop nwconcentrator
For CentOS 7.x-based appliances
For a decoder appliance:
systemctl stop nwdecoder.service
For a concentrator appliance:
systemctl stop nwconcentrator.service
If you don't stop the service, you may receive the following error when attempting to run dbcheck on the latest db file:fcntl failed: fd: 4 action: F_SETLK [OS Error: (11) - Resource temporarily unavailable]
- Run NwConsole
NwConsole
- Run dbcheck
Specific examples are provided below. The syntax for the command is:
dbcheck [-header] [-autofix] [-chatty] [-dump] <filename or pathname, wildcards accepted>
- Exit NwConsole
Exit
- Restart services on the appliance
For CentOS 5.x-based appliances
For a decoder appliance:
monit start nwdecoder
For a concentrator appliance:
monit start nwconcentrator
For a hybrid appliance:
monit start nwdecoder; monit start nwconcentrator
For CentOS 6.x-based appliances
For a decoder appliance:
start nwdecoder
For a concentrator appliance:
Start nwconcentrator
For a hybrid appliance:
start nwdecoder; start nwconcentrator
For CentOS 7.x-based appliances
For a decoder appliance:
systemctl start nwdecoder.service
For a concentrator appliance:
systemctl start nwconcentrator.service
Sample Output from dbcheck commandThe output of the dbcheck command will look something like the following. ---------------------------------------------PERFORMING TEST: DB INTEGRITY---------------------------------------------Verifying file /var/netwitness/decoder/packetdb/packet-000000002.nwpdbThe index flags are 0, which translates to INDEX_64, fidelity is 1The index file /var/netwitness/decoder/packetdb/packet-000000002.nwpdbindex does not contain a user defined header.The index contains offsets for 3095886 objects.The file "/var/netwitness/decoder/packetdb/packet-000000002.nwpdb" has a magic number of 1884076754 (0x704CBAD2).Version is 8.The file header size is 20. File creation date from header is 2015-Jan-20 02:48:08 UTC.The file flags value is 4128: . Second file flags is 1.This is a VARIABLE object size database file.The file flags value is 4128: . Second file flags is 1.This is a VARIABLE object size database file.This file does not contain a serializer header.The user defined header length is 8.Header Dump:00000000 DC C7 F9 00 00 00 00 00 [........]Header value as a unsigned 64 bit int: 16369628 0% Object 0 has a size of 765 at offset 48.1% Object 31240 has a size of 1552 at offset 7774079.2% Object 60383 has a size of 781 at offset 15547386.3% Object 90369 has a size of 112 at offset 23320313.[Output Snipped]99% Object 3066707 has a size of 104 at offset 769566957.The object store contains 3095886 objects.100% File scan complete. Total file size is 741.32 MBFile is validExample commands for running dbcheck on a decoder Test specific databse files: dbcheck -autofix /var/netwitness/decoder/sessiondb/session-000003057.nwsdbdbcheck -autofix /var/netwitness/decoder/metadb/meta-000006446.nwmdbdbcheck -autofix /var/netwitness/decoder/packetdb/packet-000102363.nwpdbTest multiple files with wildcards:Check and automatically fix the last 100 packetdb files:dbcheck –autofix /var/netwitness/decoder/packetdb/packet-0000000??.nwdbCheck and automatically fix the last 1000 packetdb files:dbcheck –autofix /var/netwitness/decoder/packetdb/packet-000000???.nwdb