Index partition is full on an RSA NetWitness concentrator with solid state drives (SSD)

[#1]

Article Number

000016600

Applies To

RSA NetWitness NextGen
RSA NetWitness NextGen 9.5 and above
RSA NetWitness Concentrator

Issue

Index partition is full on an RSA NetWitness concentrator with solid state drives (SSD).
Series 2 or Series 3 Concentrators that were upgraded or pre-installed with Solid State Drives (SSDs) were configured to use the guideline of 90% or 81% respectively of partition size for the Page Database (page.db) to accommodate file rollover.  Due to the fact that the SSDs are significantly smaller than the mechanical drives traditionally used for the Concentrator's index, the remaining free space on an SSD array is significantly less than the remaining free space on the mechanical-disk arrays and may not be enough to accommodate file operations.  In some cases, this may lead to full index partitions and the loss of index integrity.  This usually requires a data reset on the Concentrator.

Resolution

There are two procedures that can be used to implement this solution.  Either procedure will alleviate the issue but as a recommended best-practice you should implement them both if your environment allows.  If your index is already full, it is recommend that you perform a manual data reset on the Concentrator first since the Concentrator service can't run with a full partition.  To do a manual data reset on the Concentrator, refer to the knowledgebase article How to perform a manual data or index reset on an RSA NetWitness appliance.

Procedure 1: Change The Size of the Page Database in the Index Partition

Note: This procedure requires either an index reset or a full data reset of the Concentrator.  A full data reset and re-aggregation from Decoder(s) is significantly faster than a re-index, as a re-index of a Concentrator can take days or even weeks, depending on how much meta is stored.  Please note that Decoders typically store session and packet data for a shorter time frame than Concentrators store metadata, so a full data reset may result in the loss of some historical metadata for sessions that have already been rolled out of the Decoder.  During this time the Concentrator will not be able to process new queries.  If connecting to a Broker, it is advisable to remove the Concentrator from it first so as not to impact other query operations.  Please plan accordingly.

• Connect to the Concentrator with SSH and issue the following command: df -m
• Note the total size of the /var/netwitness/concentrator/index partition, listed in megabytes.  You will use this number to calculate the page.db size.
• Connect to the Concentrator's Concentrator Service using NetWitness Administrator
• Once connected, switch to Explorer View by clicking on the Explorer icon on the left pane
• In Explorer, navigate to the following tree location: YOUR_HOSTNAME (Concentrator) > index > config
• Look for the value titled "page.dir".  It should look something like this: /var/netwitness/concentrator/index/page.db=245510
• Verify the value is 90% or 81% of the partition size you observed previously in bullet 2.  The value is in megabytes.
• Once you've confirmed the value is 90% or 81%, multiply the total index partition size you observed previously in bullet 2 by .8 (series 2 Concentrators) or .72 (series 3 Concentrators), and enter this value in place of the existing value to reduce the size of the page database.  To edit, you can double-click on the value and edit directly in Explorer view.
• After making the above changes, issue the following command in NwConsole (via ssh or though the Console tab in NetWitness Administrator): For a full data reset: "/concentrator reset data=1".  For a re-index of the Concentrator: "/concentrator reset index=1"
• If entering the above command in NetWitness Administrator be sure to hit "Send"

Procedure 2: Change The Scheduler Job for Database Rollover from 6 Hours to 24 Hours

• Connect to the Concentrator's dashboard in NetWitness Administrator
• Click on "Files"
• In the drop-down that reads: "Please select a file to edit" select the file "scheduler"
• Locate the entry that reads hours=6 pathname=/index msg=save and change the "6" to a "24"
• Click the "Save" button to save the changes to the scheduler
• The change will take effect immediately