If you happen to have F5 LTM providing balancing or HA in front of your VLC for syslog messages then you may have enabled a monitor on the LTM to check for the VLC syslog service being reachable.
To do that you might have followed this guide to enable a UDP monitor that also requires an ICMP check to verify if the UDP 514 port is reachable.
Notice the default string in this example is "default send string"
These health checks are not valid syslog messages and have no priority flag set (and are 0 payload length). At volume these messages cause problems with RabbitMq and should be dropped at the VLC to prevent as much of the noise getting to the decoders as possible,
To filter the messages you can use the Filter option available on the VLC under the syslog collection and implement it for both UDP and TCP syslog.
On the log decoders you can grep /var/log/messages to find these 0 length messages and the VLC that they came from to filter.
If you run tcpdump on the VLC looking for UDP or TCP messages from the 0 length sources you might see this (if the Monitor is configured with defaults). Notice the default.send.string value which correlates to the default F5 LTM config.
Now we need to define a filter for syslog to filter these messages from the syslog pipeline
VLC > Config > Event Sources > Syslog > Filter
Define a new Filter and then define a new rule
we will use the raw key as the 0 length messages don't have proper formatted message to extract the source IP from (lc.srcid)
Now add or update the syslog collection with the filter
If you want to view stats on the drops performed by the filter you can switch to the explore view of the VLC
VLC > Explore >
logcollection > syslog > stats > eventsources
the total_filtered_events count will increase when filtered items are found (this counter is reset when the service is restarted)
If you want to see the debug logging values of what the VLC parses from the messages you can enable debug and event_filter_debug from the explore menu. For my testing (temporary) i enabled debug and set the event_filter_debug to 15
that drops debug messages into the VLC > Logs section under [DEBUG] and will show you this information about the match or no match values
20161122T162858^P^@^@^@SyslogCollection~Q^@^@^@[syslog-udp.udp514] [processing] [Receiver WorkUnit] [processing] Unidentified content from 10.10.10.10 received on receiver: 'default send string'~A^A^@^@^A^@^@^@^O^@^@^@20161122T162858^Z^@^@^@SyslogCollection(TraceLog)H^A^@^@[syslog-udp.udp514] [processing] [Receiver WorkUnit] [processing] Content received on receiver 10.10.10.10 does not conform to Syslog standards. Valid Syslog format is "<PRI> MESSAGE". Probably raw syslog message is not starting with "<"PRIVAL">" field: 'default send string'. Please rectify the issue at syslog event source.~T^B^@^@^A^@^@^@^O^@^@^@20161122T162858^Z^@^@^@SyslogCollection(TraceLog)[^B^@^@[syslog-udp.udp514] [processing] [Receiver WorkUnit] [processing] [EventFilter-Accept] syslog.NOFILTER (not filtering-test hits) 1479831814567
Rule: "no match ident only"
#1 [raw] [Contains] [(ignoreCase)default send string]
Match=Yes
Matched=default send string
Result=Drop
Event:
collection_meta:
"lc.lpid" : "syslog.syslog-udp"
"lc.cid" : "vlcid"
"lc.msgtype" : "0"
"lc.ctype" : "syslog"
"lc.wuid" : "17562157925649023279"
"lc.esname" : "udp514"
"lc.estype" : "syslog-udp"
"lc.wusn" : "93719"
content_meta:
raw_message: default send string
Using this debug message you can determine what values are extracted by the VLC to make activities/filters more accurate. In this case the lc.srcid value has no IP address so we are unable to drop based on that value, requiring the RAW value to be used.
1 Comment
