RabbitMQ certificates pointing to incorrect certificate files resulting in orphan queues between Local Log Collector and Log Decoder in RSA Security Analytics

[#1]

Article Number

000034484

Applies To

RSA Product Set: RSA Security Analytics
RSA Product/Service Type: RSA Log Collector, RSA Log Decoder
RSA Version/Condition: 10.5.x, 10.6.x
Platform: Centos
O/S Version: EL6

Issue

The RabbitMQ config file points to incorrect certificates files, this can be noticed by any of the below symptoms: 

Symptom 1: 

Queues on the Local Log Collector are missing consumers this can be confirmed by running the below command:

SSH to the Log Decoder:  
 

rabbitmqctl list_queues -p logcollection name consumers | column -t

If there are no consumers, the output should look like the below, where 0 means that there are no consumers to the corresponding queues: 

[root@logdecoder /]# rabbitmqctl list_queues -p logcollection name consumers | column -t
Listing                              queues  ...
LogDecoder.logdecoder.checkpoint     0
LogDecoder.logdecoder.cmdscript      0
LogDecoder.logdecoder.file           0
LogDecoder.logdecoder.netflow        0
LogDecoder.logdecoder.odbc           0
LogDecoder.logdecoder.sdee           0
LogDecoder.logdecoder.snmptrap       0
LogDecoder.logdecoder.syslog         0
LogDecoder.logdecoder.vmware         0
LogDecoder.logdecoder.windows        0
LogDecoder.logdecoder.windowslegacy  0
rabbitmq.log                         0

 

Symptom 2: 
 

Errors similar to the below are noticed in the RabbitMQ log file under :
 

/var/log/rabbitmq/sa\@localhost.log 

 

=ERROR REPORT==== 31-Oct-2016::13:09:09 ===
SSL: certify: ssl_handshake.erl:1331:Fatal error: bad certificate

=ERROR REPORT==== 31-Oct-2016::13:09:09 ===
SSL: certify: ssl_handshake.erl:1331:Fatal error: bad certificate

=ERROR REPORT==== 31-Oct-2016::13:09:09 ===
SSL: certify: ssl_handshake.erl:1331:Fatal error: bad certificate

=ERROR REPORT==== 31-Oct-2016::13:09:11 ===
Error on AMQP connection <0.12631.0>:
{ssl_upgrade_error,{tls_alert,"bad certificate"}}

=ERROR REPORT==== 31-Oct-2016::13:09:11 ===
Error on AMQP connection <0.12635.0>:
{ssl_upgrade_error,{tls_alert,"bad certificate"}}

Symptom 3:

Errors similar to the below are noticed Log Collector Logs under:

/var/log/messages

Oct 31 11:26:19 NwLogCollector[19565]: [AMQPClientBase] [failure] An error occurred creating an AMQP channel: Error in opening SSL/TLS connection for socket
Oct 31 11:26:19 NwLogCollector[19565]: [LogdecoderProcessor] [failure] [queue.windowslegacy] [idle] Failed during getWork: Error in opening SSL/TLS connection for socket
Oct 31 11:26:19 NwLogCollector[19565]: [AMQPClientBase] [failure] An error occurred creating an AMQP channel: Error in opening SSL/TLS connection for socket
Oct 31 11:26:19 NwLogCollector[19565]: [LogdecoderProcessor] [failure] [queue.windows] [idle] Failed during getWork: Error in opening SSL/TLS connection for socket
Oct 31 11:26:19 NwLogCollector[19565]: [AMQPClientBase] [failure] An error occurred creating an AMQP channel: Error in opening SSL/TLS connection for socket
Oct 31 11:26:19 NwLogCollector[19565]: [LogdecoderProcessor] [failure] [queue.vmware] [idle] Failed during getWork: Error in opening SSL/TLS connection for socket

 

Cause

RabbitMQ certificates are not pointing to the correct files. 

The truststore.pem certificate located under:
 

/etc/netwitness/ng/rabbitmq/ssl/

should be pointing to the truststore.pem certificate file under:

/etc/rabbitmq/ssl

The cert.pem and privkey.pem certificates under: 

/etc/netwitness/ng/rabbitmq/ssl/keys

should be pointing to cert.pem and key.pem certificates respectively under: 

/etc/rabbitmq/ssl/server/

Resolution

SSH to the Log Decoder and run the below commands: 

cd /etc/netwitness/ng/rabbitmq/ssl/

ln -s /etc/rabbitmq/ssl/truststore.pem truststore.pem

cd keys/

ln -s  /etc/rabbitmq/ssl/server/cert.pem cert.pem

ln -s /etc/rabbitmq/ssl/server/key.pem privkey.pem

The output should be something like this: 
 

[root@logdecoder ssl]# pwd
/etc/netwitness/ng/rabbitmq/ssl
[root@logdecoder ssl]# ll
total 4
drwxr-xr-x. 2 rabbitmq root 4096 Jun 25  2015 keys
lrwxrwxrwx. 1 rabbitmq root   32 Jun 25  2015 truststore.pem -> /etc/rabbitmq/ssl/truststore.pem
[root@logdecoder ssl]# cd keys/
[root@logdecoder keys]# ll
total 0
lrwxrwxrwx. 1 rabbitmq root 33 Jun 25  2015 cert.pem -> /etc/rabbitmq/ssl/server/cert.pem
lrwxrwxrwx. 1 rabbitmq root 32 Jun 25  2015 privkey.pem -> /etc/rabbitmq/ssl/server/key.pem