Failed to update remote destination error after modifying IP failover order on RSA Security Analytics virtual log collector (VLC)

[#1]

Article Number

000031697

Applies To

SA Product Set: Security Analytics
RSA Product/Service Type: Virtual Log Collector
RSA Version/Condition: 10.x
O/S Version: CentOS 6

Issue

This occurs in the VLC config > Local Collector > Destination Collectors.  When the order of the IP failover was modified, it threw the VLC into an invalid state. 

The following error messages displayed in the UI:
 

WARNING:  Failed to update remote destination:  Invalid or missing address: " Try adding the device to SA if not already added."

 
This message occurs if you are trying to modifyor delete the Destination Group or the Destination Collectors, as in the images below:

Image description

Image description

In /var/log/messages are the following messages:

Nov  6 21:26:46 myhost nw[1956]: [MessageBroker] [failure] error 2015-11-06T21.26.46Z nw_shovel_worker:init failed: no_endpoints! Retrying in 30 seconds.
Nov  6 21:26:46 myhost nw[1956]: [MessageBroker] [info] info 2015-11-06T21.26.46Z Shovel connected to (Direct) Virtual Host: <<"logcollection">>.
Nov  6 21:26:46 myhost nw[1956]: [MessageBroker] [info] info 2015-11-06T21.26.46Z inbound connected to {amqp_params_direct,<<"logcollector">>,none,    
<<"logcollection">>,sa@localhost,none,[]}.
Nov  6 21:26:46 myhost nw[1956]: [MessageBroker] [failure] error 2015-11-06T21.26.46Z nw_shovel_worker:init failed: no_endpoints! Retrying in 30 seconds.
Nov  6 21:26:46 myhost nw[1956]: [MessageBroker] [info] info 2015-11-06T21.26.46Z Shovel connected to (Direct) Virtual Host: <<"logcollection">>.
Nov  6 21:26:46 myhost nw[1956]: [MessageBroker] [info] info 2015-11-06T21.26.46Z inbound connected to {amqp_params_direct,<<"logcollector">>,none,    
<<"logcollection">>,sa@localhost,none,[]}.
Nov  6 21:26:46 myhost nw[1956]: [MessageBroker] [failure] error 2015-11-06T21.26.46Z nw_shovel_worker:init failed: no_endpoints! Retrying in 30 seconds.
Nov  6 21:26:46 myhost nw[1956]: [MessageBroker] [info] info 2015-11-06T21.26.46Z Shovel connected to (Direct) Virtual Host: <<"logcollection">>.
Nov  6 21:26:46 myhost nw[1956]: [MessageBroker] [info] info 2015-11-06T21.26.46Z inbound connected to {amqp_params_direct,<<"logcollector">>,none,

Cause

The shovel configuration file /etc/rabbitmq/shovel_config was missing the addresses, as shown below:

[{sources,[]},
{destinations,
[{destination,
[{name,"MainCollector2"},
{queue,"MainCollector2"},
{addresses,[]},
{collections,
["file","windows","odbc","checkpoint","vmware","sdee",
"snmptrap","syslog","netflow","windowslegacy"]},
{reconnect_delay,30},
{prefetch_count,3},
{ack_mode,"on_confirm"},
{mandatory_flag,1}]}]}].

Resolution

Since the destination cannot be modified or deleted via the GUI, the solution is to delete the shovel_config file located in /etc/rabbitmq/.  

1. To delete the file perform the following tasks:

service rabbitmq-server stop
stop nwlogcollector
cp /etc/rabbitmq/shovel_config to /root/shovel_config.bak
rm /etc/rabbitmq/shovel_config
service rabbitmq-server start
start nwlogcollector
rabbitmqctl list_queues -p logcollection messages name consumers

2. Remove any queues that have a "0" at the end (0 Consumers).  This can be done with KB article 000017544 (At least one VLC queue exists that does not have any consumers in RSA Security Analytics).  This will create a new blank file.
3. Reconfigure the shovel destinations in the UI.  A properly configured shovel config file shown below:

[{sources,[]},
{destinations,
[{destination,
[{name,"MainCollector3"},
{queue,"MainCollector3"},
{addresses,["2.2.2.2","3.3.3.3","4.4.4.4"]},
{collections,["file","syslog","windows"]},
{reconnect_delay,30},
{prefetch_count,3},
{ack_mode,"on_confirm"},
{mandatory_flag,1}]}]}].

4. After performing these steps you will be able to add the collector back into the UI.