Meta count (>100000 - X%)

[VishamRawat]

I want to know the exact (event logs) count for a particular metakey-value.

Now, as per my understanding for the 'Event Outcome' metakey, the (event logs) count for the 'failure' metavalue is 37,003 events.

The (event logs) count for the 'success' metavalue has reached the threshold of 100,000 events. I'm not sure what the -73% means, although as per the document it has something to do with load time. Can somebody please elaborate? The document explanation isn't satisfactory.

Also, if the 100,000 events threshold has been reached - how do I know what the exact (event logs) count is?

So, basically if I need to know the exact count of event logs streaming in within 1 hour - how do I know so, if the count shows (>100000 - 73%)?

[JohnKisner]

Visham,

What you are experiencing is a common question asked by many customers. When you see any meta key say (>100,000 - xx%) this is telling you that 100,000 is XX% of the total values for that blue meta value. In your case you are seeing that the value of Success contains more than 100,000 values and that 100,000 values is 73% of the total values for Success within the time frame you are searching. If you do the math the actual total for the time frame you are searching is about 136,986 actual values for Success. 73% of 136,986 is 100,000. Netwitness does this percentage when it reaches a certain threshold set in the product. It does this to speed up returned results when doing Investigations so it isn't churning through large number of results while you are waiting for the screen to refresh. 

You can change this threshold per person within the Investigation window itself. You would go into the Investigation area. Click on the Settings button at the right end of the action bar, you may need to click on the >> or .. to see the Settings button.

In the image above you would adjust the option in yellow. It is the threshold used when determining where to stop looking at the actual values and provide a percentage. I highly suggest not adjusting this threshold up to high as it can cause performance issues within the Investigation page.

When you see meta values with these percentages it is highly suggested to use other meta values to help focus your search which should help bring these numbers down to a more reasonable value so that it is under the threshold and you get an accurate count within Investigation. 

Please note that this threshold does not apply to items generated by reports. This mechanism is primarily there to allow the Investigation page to load faster while giving you an idea of how many results exist for that value.

For your scenario if you are looking for a 100% accurate count I would use a report, or use smaller time frames until you get a value under 100,000 within each time frame. Once you have that you can manually add the numbers together for the total time frame you are looking for. As you can see that can be cumbersome, that is why I highly suggest going the report method.

I hope this helps to answer your question.

[VishamRawat]

Thanks John.

Just to make sure I understand you correctly, I did a little calculation on my SA.

In the last 30 mins, from 15:23:00 to 15:52:59, the (firewall) metakey values for 'action' return the following count in reports.

allow = 3191546
fw:outbound = 249133
deny = 155569
fw:inbound = 29119

For the same timeframe, I however get differing figures when doing the calculation based on the investigation console meta count.

allow = 1111111.11
fw:outbound = 204081.633
deny = 163934.426
fw:inbound = 28514

Quick questions -

So, I should stick with Reports, correct?

And this count is the no. of events that have streamed in, right?

Also, can the difference in the two (views - investigation console vs. reports) really be as pronounced as it is for the 'allow' metakey?

[JohnKisner]

Visham,

Yes I would stick with reports if you want an accurate count as it will go through all the results for the time frame. The Investigation stops at 100,000 and attempts to estimate the remaining amount. This is why there is an inaccuracy once you get past the threshold setting.

This count is not necessarily the number of events that have streamed into the system. This represents the number of meta generated for that language key over the time period that you have selected. Some meta keys can have multiple entries for a single event/session. If you want to know the number of events that have streamed in over that time period then you need to look at a piece of meta that is guaranteed to be on every session/log that you receive and only appear once on that log/session. In your case above, if the Allow meta key is filled with a value from every log that has come in within that time period then you can use it as an indicator of rate. Or if you are looking for the count of Action and all your logs have Action in them then you can use it as well. Normally when customers ask about how to determine rate we point them to the Decoder Source (DID) meta language key. For one this key doesn't conform to the threshold restriction and it takes into account all traffic being collected by that Decoder.

As to your last question, yes the difference can be very pronounced due to the way the system attempts to estimate the remaining amount of items that exist for a particular meta key after it hits the threshold.

[JohnSnider]

If your numbers from investigation are calculations from the % value for 100,000, what you have to realize, is it is just an estimate, it's not an "exact" percentage.

Let me explain:

As J Kisner said, for the time frame you are viewing (lets use your 30 minutes) and reversing your calculation, you showed

allow (100,000 - 9%)

fw:outbound (100,000 - 49%)

deny (100,000 - 61%)

fw:inbound (28,514)

since fw:inbound is not identical for the report/investigate, you may have run the report without using "relative time" which would skew the numbers between the two, but for this explanation,  we'll assume they are the same exact timeframe.

Notice that the accuracy is higher for the estimate when it is closer to 100%, the allow only being at 9% is why it's so far off. It is saying that for the last 30 minutes, it saw 100,000 instances of "allow" @ (9% of 30 minutes = 2.7 minutes), now if the rate of logs with "allow" in them remained exactly the same for the other 27.3 minutes of the time frame, the calculated number would be accurate, but the rate was much higher during the first 2.7 minutes than it was for the other 27.3 minutes, so the "estimate" was off. 

So the thing to remember, is: the lower the percentage at 100,000 events, the less accurate the estimation is going to be.

 Looking at the "deny" at 61% it ends up being much more accurate with the actual number of events over the time period.

[VishamRawat]

Thank you! much clearer now.

[VishamRawat]

Thanks!

So, DID for determining the exact volume of event logs that've streamed in, right?

And this is because the DID is recorded once for every event log, correct?

[JohnKisner]

Visham,

That is correct. The DID meta is on every log/session that is captured by the system and it is only recorded once. Since the DID represents a specific decoder you can see how many events came in on that decoder for the time period in question. 

On another note you may have noticed that the event.time meta on a log event doesn't always match the time meta. This is because the time meta is set on capture by the log decoder where the event.time meta is set by the event source and shows the time the event happened on the event source itself. The event.time meta is the time inside the log. Time and event.time are very rarely exactly identical. I bring this up because the DID count represents when the event was captured which corresponds to the time meta not the event.time meta. So if you are looking for volume on an event source there may be some variation due to this difference, but it should generally be small. However if you are just looking for volume of the Netwitness system  then you should be good to go.