RSA NetWitness VLC a rabbitmq queue exists that does not have any consumers

[#1]

Article Number

000017544

Applies To

RSA Product Set: NetWitness Platform
RSA Product/Service Type: Virtual Log Collector (VLC), RabbitMQ Message Broker
RSA Version/Condition: 10.6, 11.x
Platform: CentOS

Issue

At least one rabbitmq queue on the VLC doesn't have any consumers in RSA NetWitness.

In /var/log/messages on the VLC, an error similar to the following is displayed:

Aug  4 22:20:51 NWAPPLIANCE32722 nw[1434]: [MessageBroker] [warning] warning 2014-08-04T15.20.51Z At least one queue exists that does not have any consumers.  This condition may arise because the Log Collector is not currently running or parts of the system have been shut down.  In some cases this condition may arise becau

The message is truncated in /var/log/message. To obtain the full message, perform the following steps:

1. In the NetWitness UI, navigate to Admin > Devices, select the VLC device, and click on View > Logs.
2. On the Historical tab, enter keywords "at least" and click Search.

The full message will be displayed, as shown below.

Warning 2014-08-05T09.00.51Z At least one queue exists that does not have any consumers. This condition may arise because the Log Collector is not currently running or parts of the system have been shut down. In some cases this condition may arise because a queue consumer (such as an event processor or VLC connection) was removed outside of the normal course of operation. The queue names that have no consumers are are: "shovel.sdee.Q2LC", "shovel.file.Q2LC", "shovel.syslog.Q2LC", "shovel.checkpoint.Q2LC", "shovel.windows.Q2LC", "shovel.vmware.Q2LC", "shovel.windowslegacy.Q2LC", "shovel.snmptrap.Q2LC", "shovel.odbc.Q2LC". Make sure that the Log Collector process is running and that all event processors are in the running state. If this warning message persists and you are certain there are no legitimate consumers of these queues, you may delete them via the 'delete' property on the '/event-broker' node. Supply the queue name to delete...."

Cause

This issue can occur when:

• Deleting a Destination Groups entry in NW UI, Admin > Services > {VLC} > Config, Local Collectors tab, doesn't delete the rabbitmq queues.
• Changing the Destination Groups entry to another name doesn't remove the old queues.
• The Log Collector (LC) is not currently running or communication between the LC and VLC is not working.

This leaves some rabbitmq queues on the VLC orphaned without any consumers.

Resolution

Get a list of the names of the 'orphaned' rabbitmq queues from the VLC

On the VLC device run the following command.

rabbitmqctl list_queues -p logcollection consumers name messages

Look for those queue names without an consumers.

For example:

[root@VLC ~]# rabbitmqctl list_queues -p logcollection consumers name messages
Timeout: 60.0 seconds ...
Listing queues for vhost logcollection ...
consumers name messages
0 shovel.windowslegacy.Q2LC 0
0 shovel.vmware.Q2LC 0
0 shovel.checkpoint.Q2LC 0
0 shovel.netflow.Q2LC 0
0 shovel.sdee.Q2LC 0
0 shovel.file.Q2LC 0
0 shovel.windows.Q2LC 0
0 shovel.cmdscript.Q2LC 97406
0 shovel.odbc.Q2LC 0
0 shovel.syslog.Q2LC 2592
0 shovel.snmptrap.Q2LC 0
1 shovel.vmware.LogHybrid 0
1 shovel.odbc.LogHybrid 0
1 shovel.sdee.LogHybrid 0
1 shovel.file.LogHybrid 0
1 shovel.netflow.LogHybrid 0
1 shovel.syslog.LogHybrid 0
1 shovel.checkpoint.LogHybrid 0
1 shovel.cmdscript.LogHybrid 0
1 shovel.snmptrap.LogHybrid 0
1 shovel.windows.LogHybrid 0
1 rabbitmq.log 0

Where,
First column is the consumers count.
Second column is the rabbitmq queue name, in the format shovel.collection_type.name like, shovel.windowslegacy.Q2LC
Third column is the count of logs remaining in the queue.
 

Delete the 'orphaned' queues via the NetWitness UI

1. In NetWitness UI, navigate to Admin > Devices, select the VLC device, and click on View > Explore
2. Right-click on event-broker and select Properties
3. From the drop-down box on /event-broker properties window, select delete
4. In Parameters, enter: queue="shovel.collection_type.name" (seen in the error message, or from the above output) and click Send.
5. ResponseOutput will show "Success".

 

Or Delete the 'orphaned' queues from the command line

Run the following command to delete the rabbitmq queue.

rabbitmqctl delete_queue -p logcollection shovel.collection_type.name

Substitute the shovel.collection_type.name with the rabbitmq queue name in the error message, or from the above output.

For example:

[root@VLC ~]# rabbitmqctl delete_queue -p logcollection shovel.windowslegacy.Q2LC

Run the following command and confirm the queue is now deleted.

rabbitmqctl list_queues -p logcollection consumers name messages

Notes

Any rabbitmq queue which has any outstanding logs which is deleted will also delete those logs.

From the above example those queues are:

Listing queues for vhost logcollection ...
consumers name messages

0 shovel.cmdscript.Q2LC 97406
0 shovel.syslog.Q2LC 2592