Alarm H&W "Meta Overflown" and valueMax in index-concentrator-custom.xml

[Philippe]

Hello !

I'm trying to understand the link between inacurate investigations and META OVERFLOWN ALARM.

Meta Overflown is triggered when concentrator has indexed values until ValueMax. New values, then, are not indexed and investigate could be inacurate.

SYMPTOMS :

• Meta overflown Alarm
• Inacurate investigation

FIX :

• Increase ValueMax
• Change save.session.count
• indexSave in Explorer view
• Customize parsing (meta generated with different type of data « Text »..., meta keys with too much values?)  

eg Meta overflown Alarm on MD "referer", with:

<key description="Referer" name="referer" format="Text" level="IndexValues" valueMax="10000"/>

Why not increase valueMax parameter to 1 00 000 0000 000 000 .... !

How to monitor parameter of VALUEMAX without negative impact on concentrator and/or ingest and/or invests ?

Thanks in advance for comments and help !

PhBfr

[JohnSnider]

What you have to understand is the default valueMax for meta keys in the index-concentrator.xml file are historical values based on Network capture (and some early log capture) for data rates (~2 - 4 Gbps) that were the average back when this evaluation was done years ago.  Data rates have increased in the past years (hence 10G capture capability) .  The reason they were set to lower values at the time is to limit the size of the index slices. Most of these values are still from when we used "time" based index saves of every 8 hours, and not aligned with a session.save.count like we have been using since 10.5/10.6 days, when it was set to 600 Million sessions. This session save count was used to try an limit the number of index slices (as high numbers of slices were causing issues with query performance). This was later revised to 200 Million slices (not sure exactly when that change happened) as the 600M was causing the exact issue you are stating, too many times keys were hitting valueMax way before the index save and customers were losing search capability for incidents. At some customers, they decide to force a save every 24 hours, so they know that within their 365 day retention they have a controlled number of slices (~365 - Plus some from service restarts/reboots).  

All that being said, there is NO way to predict how any customer network or logs will affect the meta.  There is no "one size fits all"  the reason we don't set it to 100,000,000,000 is valueMax is a safety valve.  If you have a network parser or log parser that decides to "go nuts" or someone writes a feed or application rule that writes to a particular meta key and is poorly written (i.e. triggers on EVERY SESSION)  it could cause damaging bloat to the index and end up pushing "good" index slices off the system as the bad data fills up the disk. 

Every customer will have to make modifications to the indexing on SOME keys, depending on the data rates, or the log sources they are capturing.  This is know as "tuning and optimization" and is a required process on any install.  In some cases, due to a large number of unique values in keys, you may not be able to fully index that key and instead change it to IndexKeys. A lot will depend on if you use that key for queries of specific values, in some cases it may be better to write the detection in an application rule that writes to a customer specific key (ex: customer.info) for specific matches to values in that key that occur during capture (before indexing), so the sessions of interest can be found in a key that is NOT overrun with too many unique values.  In other cases you may have to increase the valueMax in stages until you find the value that works for the overall traffic.  You may be able find other keys that are never used for searching that could be changed to IndexKeys or IndexNone to help decrease the size of the index, allowing you to increase the valueMax on more relevant keys to higher values.  

[RichardB]

How about setting valueMax="0" for some meta keys? Does that make the number of possible unique values unlimited?

<key description="Time" name="time" format="TimeT" singleton="true" level="IndexValues" valueMax="0"/>
<key description="Event Time" name="event.time" format="TimeT" singleton="true" level="IndexValues" valueMax="0"/>
<key description="Time Start" name="starttime" format="TimeT" singleton="true" level="IndexValues" valueMax="0"/>

I understand this can cause the index for these meta keys to become very large but that is sometimes better than loosing the ability to search for some values.

[Philippe]

Hello John and thanks a lot for your answer. 

Sorry to be late, but further reading was necessary. 

1) 
ValueMax = Size of the buffer allocated [i Think by conc] for the meta (FIFO of distinct values)

« FIFO » ? Does it sounds like a stack ?

eg mk = ip.src mv = <ip values list>

2)

ValueMax = historical limit SIZE of index slices 

save.session.count = limit NUMBER of index slices

On first default install (11.4) how to know number of index slices ? « Index saved by slices of 8 hours (« 28800 secs waited » in /var/log/messages of conc ?)

3)
IndexNone, IndexKeys and IndexValues = index Level in index-concentrator-custom.xml

Which meta, conc is indexing…

IndexNone = default setting if meta not listed in « the file »

IndexKeys and IndexValues refer to index structure. 

Could you explain again (sorry) diff between Two ?

i saw an example which i would like to « map » to eg mk=ip.src and mv=list of unique ip value:

- for mk set to: 

indexkeys -> links for each MK all events id :

filename 

- event 10;

- event 23;

- event 78;

indexValues -> links for each VALUES corresponding event id « tagged by specific value » :

filename

- a.php 

event 10; 

event 23; 

- index.html 

event 78; 

Did you understand that ? 
Could you explain me as if i am 4 years old ?

I would like first, to thank you for your Time John. 

Philippe