Health & Wellness High Severity messages

[RomanZeltser]

I was not able to find any explanation on the following problems (Health & Wellness):

Meta(s) overflown in Index Slices and High Swap Utilization.

Can anyone point to the reference or explain it?#highitilization

[NaushadKasu1]

Meta Overflow:

Each Concentrator has a setting on how often it saves index slices (a setting of auto means every 600  million sessions) under View -> Explore -> index -> config.  For each slice, the Concentrator will ONLY index unique values up to a max value i.e. valueMax setting in index-concentrator-custom.xml.  This means that if you overflow a specific key, in your case, event.desc, all values that are written to the metadb past the valueMax setting are not indexed and thus not searchable in Investigator.  You can use the key in select clause but if you do a .. where event.desc = "blah" .. then the Concentrator will only return results for which it has indexed (leading to inaccurate / incomplete results).  Typically, you can increase the valueMax setting and/or lower the index slice write interval so this overflow does not occur BUT I'd highly investigate further before increasing that particular key 'event.desc' because it has so many unique values and very large string lengths -- it could cause degradation of the Concentrator service and thus lead to slow ingest and investigations.

High Swap:
The system is using high swap (used up memory and now it is swapping to disk).  You need to further investigate what is causing that.  It's an SA head as I can see from the image so there could be multiple culprits - jetty, collectd, SMS etc..

[RomanZeltser]

Naushad, thanks for the quick reply.

I will certainly look down on those parameters. Is the max size depend on the disk size and is extended automatically after increasing the disk space, or it should be done manually? If it doesn't depend on the disk size, what is the default/suggested setting and what is the max setting?

The memory utilization was quite small until this event. Usually, I see the alarm on the vCenter for VM if I have high utilization but I did not see any. I would try to find out what is going on but I am not sure where to look...

[NaushadKasu1]

The max size (valueMax setting in index-concentrator-custom.xml) is configurable but needs to be carefully monitored after changes because it could impact the system negatively.  There are a handful of keys we index out of the box and set their valueMax setting appropriate for the type of data we expect to see over some period of time (the index slice).  However, as customers create new / custom keys or have a high number of unique values for specific keys they can override our keys which are in index-concentrator.xml by putting the same entry in index-concentrator-custom.xml and changing the parameters i.e. valueMax.  They can also add their new/custom keys to the same custom file.

[RomanZeltser]

I see. Thank you very much!

[RenatoGoncalves]

I received this message:

Policy Concentrator Monitoring Policy

Rule Meta(s) Overflown in Index Slices

Statistic Index/Meta(s) Overflown

Value ip.dstport But following the directions of the email: You can adjust the valueMax parameter in the index-concentrator-custom.xml, for the meta(s) that have reached the maximum capacity. I go to the index-concentrator-custom.xml, but i dont have any reference to ip.dstport and the reference to valueMAX is  : valueMax = Maximum number of unique values allowed per index slice (typically 1 to 4 slices are created per day). Zero means unbounded. Only applicable to level="IndexValues" How can solve this issue of meta overflown?

[VincentWareham]

Hello Renato

If you don't find ip.dstport in your Concentrator /etc/netwitness/ng/index-concentrator-custom.xml file then the default setting hasn't been changed in the /etc/netwitness/ng/index-concentrator.xml file.

Look for the default ip.dstport setting in the /etc/netwitness/ng/index-concentrator.xml file.

For example,

grep ip.dstport /etc/netwitness/ng/index-concentrator*.xml
/etc/netwitness/ng/index-concentrator.xml:      <key description="Destination Port" level="IndexValues" name="ip.dstport" format="UInt16" valueMax="65536" defaultAction="Closed"/>

Notice the default values of valueMax="65536", and also format="UInt16" which is a 2 byte number in the range 0 to 65,535.

This is a reasonable valueMax as the valid TCP port values can only range from 0 to 65535.

If you are getting this default valueMax setting overflown, please check the sorts of values you are getting for ip.dstport, perhaps there are some invalid values been assigned to this meta field, which will indicate a parser needs fixing.

[JohnSnider]

One caveat to this. For ANY of the port keys, change the valuemax to 65537, the alert for meta overflown triggers when # of unique values = valueMax, not exceeds valuemax. I have an engineering case open to fix this, but the only workaround at this time is to set valuemax 1 higher than the total possible values for these keys. Copy them from index-concentrator.xml and paste then into the index-concentrator-custom.xml (where noted in the file) and modify the valuemax by adding 1to it (65537) then save the file and either restart the concentrator or wait for the next index save to occur.