tcp.srcport Index/Meta(s) Overflown

[RichardB]

We have several appliances that report tcp.srcport is overflown. We are familiar with this concept for meta keys that have limited index sizes like payload or filename, but tcp.srcport in defined correctly in index-concentrator.xml as:

<key description="TCP Source Port" name="tcp.srcport" format="UInt16" level="IndexValues" valueMax="65536">

Inspecting the index indeed shows one of the index slices has 65536 values, so that is all of the TCP ports 0-65535. What we fail to understand is how a new tcp.srcport value could be different and thus overflow this index. By definition a TCP port is a 16 bit unsigned field, so 0-65535 are the only possible values that could ever exist, right?

{
  "msgPack": {
    "keys": {
      "tcp.srcport": [
        {
          "keyIndex": {
            "accesses": 0,
            "key": "tcp.srcport",
            "packets": 28867258,
            "pages": 28,
            "pathname": "/var/netwitness/concentrator/index/managed-values-1/tcp.srcport.nwindex",
            "session1": 59920088,
            "session2": 61918168,
            "sessions": 1772474,
            "size": 27724295402,
            "summaries": 28,
            "summary1": 6866213,
            "summary2": 8140168,
            "values": 64835
          }
        },
        {
          "keyIndex": {
            "accesses": 0,
            "key": "tcp.srcport",
            "packets": 856792452,
            "pages": 728,
            "pathname": "/var/netwitness/concentrator/index/managed-values-0/tcp.srcport.nwindex",
            "session1": 1,
            "session2": 59920086,
            "sessions": 56358097,
            "size": 817822448980,
            "summaries": 728,
            "summary1": 60,
            "summary2": 6864324,
            "values": 65536
          }
        }
      ]
    },
    "meta1": 1,
    "meta2": 4550001864,
    "session1": 1,
    "session2": 61918168,
    "size": 1047923718704,
    "totalKeys": 1,
    "totalMemory": 2084005232,
    "totalValues": 130371
  }
}

[KevinArunski]

You probably have some parser (or other content) generating tcp.srcport as something other than UInt16, for example as Text values.   It would likely be something non-RSA.

[JohnSnider · Accepted solution]

Actually, the issue relates to HOW the alert was coded into our system.  The overflown meta trigger happens when the number of unique values "=" the valueMax setting, instead of triggering on ">" valueMax setting (or valueMax + 1).  This is causing several keys to trigger false positive alerts.  Since the valueMax setting is really meant to keep indexes from growing too large, for values that we have no control over (think alias.host, ip.src, ip.dst, etc.) if they get too many unique values, finite keys like the port keys really should not have a valueMax set at all, since they have an physical upper limit.  Though on the other hand, having EVERY port value seen in a time period would indicate a scanner or other non-normal traffic indication.

you can either set the valueMax to 65537 for the port keys, or remove the valueMax entry entirely and the alerts will no longer fire, since you know the maximum number of values will not exceed 65537 (unless some other custom feed/parser/application rule accidentally  writes to that meta key.

I have opened an internal engineering ticket to address this issue.

[AaronMartin2]

Both Kevin and John are correct. For other keys, particular text ones which aren't bound by a real narrow data type size, it could be a parser or other piece of meta generation that could result in an overflow accidentally occurring. Given we all understand what the max possible values for a tcp port number, we know that is unlikely the case, yet we must also be aware that the alarm triggers when the bucket is completely full; not actually when the bucket actually overflows.

[RichardB]

The few custom parsers we have do not set tcp.srcport, but that would definitely be something to check if they did.

[RichardB]

Thanks for confirming this is probably a false positive. I tried to explain this to support but didn't get past 1st line. I've increased the valueMax to 65537 for now to see if that helps.

[KevinArunski]

I see. I was looking at the total values summary (130371) rather than the per slice number (65536).

[KevinArunski]

The warning isn't written to trigger on 'unique values > valueMax' because that condition is never allowed to occur.  We can't put more values than valueMax in the index.  So if it was written that way, it would never fire.

The valueMax on the port values is important to prevent against the case of parsers somehow writing in the wrong format.

Therefore a valueMax of 65537 is appropriate.  However, I do wonder how they got the full 65536 Uint16 values (0-65535) into tcp.srcport.  I don't think you should see a TCP source port of 0 go across the wire.

[RichardB]

That was my first thought as well. I have verified that a tcp.srcport of 0 is indeed in the index. This occurs when the data is fragmented (ip.frag_offset > 0) and the first fragment is not seen. I suspect a flaky span port to be to blame for that. This causes the first 16 bits after the IP destination address to be interpreted by NW as the tcp.srcport (which is probably wrong looking at RFC791).