NW Decoder problem, 9.8

[RSAAdmin]

Hi

Still running Netwitness 9.8 here, all working ok until now. The decoder recently stopped collecting and nwdecoder process shut down, the nwdecoder process attempts to restart but then fails. It appears that the decoder index (10GB) partition is full, this had the effect of turning off capture (just prior to capture stopping, there are messages warning that the index is running low of space).

Should the index manage its own file space? From what I can see, its set to index 'time' values only. Can anyone suggest how I would deal with this issue?

Thanks in advance

[Anonymous]

Hi,

I don't have a 9.8 system to double-check and the best would probably be to open a support ticket but here is something that may help.

If you open "Explorer" on your Decoder config via NwAdministrator, navigate to /index/config and check if index.dir has an "=xxxGB" following the path shown, that value should be around 90% of the volume storage, if it's not there that's probably why data isn't rolling off and the file system is full.

If that's the case going back to /index and using right-click "Properties" should show you a reconfig command that you can issue. Use care when doing this, the decoder will require a restart and it may take some time to come back up, it may actually still require manual removal of some older files.

The other common issue is that actually you may find that the index.dir isn't pointing to the right volume due to a problem with the configuration and that could be why the volume filled up.

Again these are just some tips, the best course is to reach out to RSA Support via their standard methods.

Hope it helps!

Cheers,

Rui

[JonathanSaxon]

Open a ticket with Support and send the /var/log/messages file as well as the results of "df -h". 

Something is probably consuming space on the index volume that ought not be there, or the index size is not configured.

The disk space on an S3 should look something like:

[root@S3-Dell-Decoder ~]# df -h

Filesystem            Size  Used Avail Use% Mounted on

/dev/mapper/VolGroup00-root

                       16G  5.5G  9.4G  37% /

tmpfs                  16G     0   16G   0% /dev/shm

/dev/sdc1             248M   55M  180M  24% /boot

/dev/mapper/VolGroup00-usrhome

                      2.0G  3.1M  1.9G   1% /home

/dev/mapper/VolGroup00-tmp

                       20G   44M   19G   1% /tmp

/dev/mapper/VolGroup00-var

                       12G  686M   11G   7% /var

/dev/mapper/VolGroup00-rabmq

                       20G   35M   20G   1% /var/lib/rabbitmq

/dev/mapper/VolGroup00-nwhome

                       40G  512M   40G   2% /var/netwitness

/dev/mapper/decodersmall-decoroot

                       30G  643M   30G   3% /var/netwitness/decoder

/dev/mapper/decodersmall-index

                       30G  151M   30G   1% /var/netwitness/decoder/index

/dev/mapper/decodersmall-metadb

                      3.4T  241G  3.1T   8% /var/netwitness/decoder/metadb

/dev/mapper/decoder-packetdb

                      6.4T  6.1T  331G  95% /var/netwitness/decoder/packetdb

/dev/mapper/decodersmall-sessiondb

                      250G   20G  231G   8% /var/netwitness/decoder/sessiondb

/dev/mapper/VolGroup00-vartmp

                      5.8G   12M  5.5G   1% /var/tmp

/dev/mapper/decoder0-packetdb

                      9.1T  8.7T  480G  95% /var/netwitness/decoder/packetdb0

The Dell S3 gear should have a 30GB index partition.

A S4 Decoder will look like:

[root@S4-Dell-Decoder ~]# df -h

Filesystem            Size  Used Avail Use% Mounted on

/dev/mapper/VolGroup00-root

                       20G  1.4G   17G   8% /

tmpfs                  48G     0   48G   0% /dev/shm

/dev/sde1             248M   56M  179M  24% /boot

/dev/mapper/VolGroup00-usrhome

                      3.9G  449M  3.2G  13% /home

/dev/mapper/VolGroup01-tmp

                       20G   44M   19G   1% /tmp

/dev/mapper/VolGroup01-varlog

                      9.8G  507M  8.8G   6% /var/log

/dev/mapper/VolGroup01-nwhome

                       30G  596M   30G   2% /var/netwitness

/dev/mapper/VolGroup02-warec

                      400G   34G  367G   9% /var/netwitness/warehouseconnector

/dev/mapper/VolGroup00-vartmp

                      5.8G   12M  5.5G   1% /var/tmp

/dev/mapper/decodersmall-decoroot

                       10G  3.2G  6.9G  32% /var/netwitness/decoder

/dev/mapper/decodersmall-index

                       30G  432M   30G   2% /var/netwitness/decoder/index

/dev/mapper/decodersmall-metadb

                      5.2T  3.9T  1.4T  75% /var/netwitness/decoder/metadb

/dev/mapper/decodersmall-sessiondb

                      278G  264G   14G  96% /var/netwitness/decoder/sessiondb

/dev/mapper/decoder-packetdb

                       28T   26T  1.4T  95% /var/netwitness/decoder/packetdb.

You can apply the "mark 1 eyeball" test and see what files or folders are on the index partition and see what looks odd.  The only things that belong in /var/netwitness/decoder/index are a series of index slices stored in folders named like "managed-values-x" where "x" is a sequential number. 

Each folder ought to contain a similar list of files that are something like:

[root@S3-Dell-Decoder managed-values-68]# ls -lah

total 1.1M

drwxr-xr-x.  2 root root   89 Nov 26 20:37 .

drwxr-xr-x. 78 root root 4.0K Dec  2 20:38 ..

-rw-------.  1 root root  138 Nov 26 20:37 managed-values-68.manifest

-rw-------.  1 root root  51M Nov 26 20:37 page.db

-rw-------.  1 root root  11M Nov 26 20:37 summary.db

-rw-r--r--.  1 root root 1.0M Nov 26 20:37 time.nwindex

Short-term you could perform an index reset on the appliance but that could take 8 hours or more to reindex the entire Decoder. 

Hope this helps a bit.

[RSAAdmin]

Thanks, really useful. I noticed in the index config, that attribute 'time' is being indexed, this seems a really strange to me.... does this sound wrong to you? As would be expected, the index contains very large files called 'time.nwindex'.

Because I can't start the decoder, I can't edit the max size via the Netwitness Manager GUI. Can I do this via CLI? Would it make more sense to reset the index?

[RSAAdmin]

Hi

As an update, the problem was indeed related to a full Index partition. I thought it would rollover and manage its own space, but this didn't seem to be happening. The index partition is sized at 10GB.

Something I did notice, page.dir was set as follows:

/var/netwitness/decoder/index=6GB

The user guide states that 'units are t for Terrabytes, g for Gigabytes, m for Megabytes' so I have changed the parameter to: /var/netwitness/decoder/index=6g

Anyone know if this makes a difference? I've reduced the size of the index contents and everything is now back and working again... obviously keen to make sure that it self-manages the index size in the future!

[JonathanSaxon · Accepted solution]

Look in the /etc/netwitness/ng/index-decoder.xml and you will notice that the only index value declared is for time.  Unless you modified this, or index-decoder-custom.xml, that is the only index value in use.

Normally, the threshold for the index partition size is set to 90% of the allocated space.  If your index was configured to /var/netwitness/decoder/index=10GB then the system would allow the partition to fill up if the partition was only 10GB.

The recommended index size setting for a 10GB partition would be (.90 * 10GB) or /var/netwitness/decoder/index=9GB. 

If the index reaches 100% full, the services requires manual intervention and stops deleting old index slices. 

You can reset value in the NwAdministrator tool by running the "reconfig" option on the Index node in Explorer view.