Netwitness Log Decoder's rabbitmq queue is backlogged with the error "Failed to locate message id field"

[#1]

Article Number

000003129

Applies To

RSA Product Set: NetWitness Platform
RSA Product/Service Type: Log Decoder, RabbitMQ Message Broker
RSA Version/Condition: 11.x,12.X
Platform: CentOS 7

Issue

The Log Decoder's rabbitmq queues show a huge backlog when running the below command.

#rabbitmqctl list_queues -p logcollection consumers messages name

Timeout: 60.0 seconds ...
Listing queues for vhost logcollection ...
consumers       messages        name
1       0       LogDecoder.logdecoder.windowslegacy
1       0       LogDecoder.logdecoder.checkpoint
3       48      LogDecoder.logdecoder.file
1       0       LogDecoder.logdecoder.vmware
1       0       LogDecoder.logdecoder.odbc
10      29248071        LogDecoder.logdecoder.syslog
1       0       rabbitmq.log
1       0       LogDecoder.logdecoder.windows
1       0       LogDecoder.logdecoder.snmptrap
1       0       LogDecoder.logdecoder.cmdscript
1       0       LogDecoder.logdecoder.sdee
1       0       LogDecoder.logdecoder.netflow

In the /var/log/messages file a similar error as below will be indicated.
Jul 11 16:49:35 Logdecoder NwLogCollector[1788]: [GenericLogTransformer] [warning] Failed to locate message id field during event transformation. The message id field number is 5. The raw event is "0x7fd,6ca,956,1e0". The message will still be delivered.

Cause

This issue is due to incorrectly formatted events from syslog or file collection sources. These incorrectly formatted events are identified by checking the Events page using a device.type='unknown'  query to see the below problematic events.

Image description

Resolution

Work with the event source owner to stop or fix the problematic events. The device the event is coming from can be identified by the device.ip meta associated with the event.

Alternatively, Syslog or File Collection filters can be used to drop problematic event collections using the below document.
Configure Event Filters for a Collector