How to use the NwConsole and dbcheck to validate databases on an RSA NetWitness Platform or Security Analytics Appliance

[#1]

Article Number

000001320

Applies To

RSA Product Set: Security Analytics, NetWitness Logs & Network
SA Product/Service Type: Core Appliance
RSA Version/Condition: 10.x, 11.x
Platform: CentOS
O/S Version: EL6, EL7
 

Issue

The NextGen or Security Analytics core appliance experiences Initialization Errors when the service starts and the service fails to start or capture or aggregation fail to start.  

This may be due to corrupted meta, session or packet database files on the core appliance.  

Use the steps in this article to identify and fix corrupted database files using the dbcheck feature of the NwConsole tool. 

Resolution

Background

• The Decoder/LogDecoder service has the following databases: sessiondb (nwsdb), metadb (nwmdb), packetdb (nwpdb) and NwServerLog (log)
• The Concentrator service has the following databases: sessiondb (nwsdb) & metadb (nwmdb) and NwServerLog (log)
• The Appliance service has the following databases: NwServerLog (log)
• 9.8 also adds: statdb for appliance and service (statsdb)

Note: Appliances DACs may have multiple packetdb file systems. The first DAC will be found /var/netwitness/decoder/packetdb0.  Subsequent DACs will be found at /var/netwitness/decoder/packetdb1, etc. 
Prior to running dbcheck, you should assess the contents of /var/netwitness to determine which database files you want to verify.

Running NwConsole and dbcheck

1. SSH to the appliance as root
2. Stop services
   For CentOS 5.x-based appliances

For a decoder appliance:
monit stop nwdecoder
For a concentrator appliance:
monit stop nwconcentrator
For a hybrid appliance:
monit stop nwdecoder; monit stop nwconcentrator

For CentOS 6.x-based appliances

For a decoder appliance:
stop nwdecoder
For a concentrator appliance:
stop nwconcentrator
For a hybrid appliance:
stop nwdecoder; stop nwconcentrator

For CentOS 7.x-based appliances

For a decoder appliance:
systemctl stop nwdecoder.service
For a concentrator appliance:
systemctl stop nwconcentrator.service
 

If you don't stop the service, you may receive the following error when attempting to run dbcheck on the latest db file:
fcntl failed: fd: 4 action: F_SETLK [OS Error: (11) - Resource temporarily unavailable]

3. Run NwConsole

NwConsole

4. Run dbcheck

Specific examples are provided below.  The syntax for the command is:
dbcheck [-header] [-autofix] [-chatty] [-dump] <filename or pathname, wildcards accepted>

5. Exit NwConsole

Exit

6. Restart services on the appliance

For CentOS 5.x-based appliances

For a decoder appliance:
monit start nwdecoder
For a concentrator appliance:
monit start nwconcentrator
For a hybrid appliance:
monit start nwdecoder; monit start nwconcentrator

For CentOS 6.x-based appliances

For a decoder appliance:
start nwdecoder
For a concentrator appliance:
Start nwconcentrator
For a hybrid appliance:
start nwdecoder; start nwconcentrator

For CentOS 7.x-based appliances

For a decoder appliance:
systemctl start nwdecoder.service
For a concentrator appliance:
systemctl start nwconcentrator.service

 
Sample Output from dbcheck command
The output of the dbcheck command will look something like the following. 
---------------------------------------------
PERFORMING TEST: DB INTEGRITY
---------------------------------------------
Verifying file /var/netwitness/decoder/packetdb/packet-000000002.nwpdb
The index flags are 0, which translates to INDEX_64, fidelity is 1
The index file /var/netwitness/decoder/packetdb/packet-000000002.nwpdbindex does not contain a user defined header.
The index contains offsets for 3095886 objects.
The file "/var/netwitness/decoder/packetdb/packet-000000002.nwpdb" has a magic number of 1884076754 (0x704CBAD2).
Version is 8.
The file header size is 20.  File creation date from header is 2015-Jan-20 02:48:08 UTC.
The file flags value is 4128: . Second file flags is 1.
This is a VARIABLE object size database file.
The file flags value is 4128: . Second file flags is 1.
This is a VARIABLE object size database file.This file does not contain a serializer header.
The user defined header length is 8.
Header Dump:
00000000 DC C7 F9 00 00 00 00 00                          [........]
Header value as a unsigned 64 bit int: 16369628
 
0%  Object 0 has a size of 765 at offset 48.
1%  Object 31240 has a size of 1552 at offset 7774079.
2%  Object 60383 has a size of 781 at offset 15547386.
3%  Object 90369 has a size of 112 at offset 23320313.
[Output Snipped]
99%  Object 3066707 has a size of 104 at offset 769566957.
The object store contains 3095886 objects.
100%  File scan complete.  Total file size is 741.32 MB
File is valid

Example commands for running dbcheck on a decoder
Test specific databse files:

dbcheck -autofix /var/netwitness/decoder/sessiondb/session-000003057.nwsdb
dbcheck -autofix /var/netwitness/decoder/metadb/meta-000006446.nwmdb
dbcheck -autofix /var/netwitness/decoder/packetdb/packet-000102363.nwpdb

Test multiple files with wildcards:

Check and automatically fix the last 100 packetdb files:

dbcheck –autofix /var/netwitness/decoder/packetdb/packet-0000000??.nwdb

Check and automatically fix the last 1000 packetdb files:

dbcheck –autofix /var/netwitness/decoder/packetdb/packet-000000???.nwdb