RabbitMQ certificates pointing to incorrect certificate files resulting in orphan queues between Local Log Collector and Log Decoder in RSA Security Analytics

[#1]

Article Number

000002363

Applies To

NetWitness Product Set: NetWitness Platform
NetWitness Product/Service Type: Log Collector
NetWitness Version/Condition: 12.x
Platform: CentOS/Alma Linux

Issue

The RabbitMQ config file points to incorrect certificates files, this can be noticed by any of the below symptoms: 

Symptom 1: 

Queues on the Local Log Collector are missing consumers this can be confirmed by running the below command:

SSH to the Log Decoder:  
 

rabbitmqctl list_queues -p logcollection name consumers | column -t

If there are no consumers, the output should look like the below, where 0 means that there are no consumers to the corresponding queues: 

[root@logdecoder /]# rabbitmqctl list_queues -p logcollection name consumers | column -t
Listing                              queues  ...
LogDecoder.logdecoder.checkpoint     0
LogDecoder.logdecoder.cmdscript      0
LogDecoder.logdecoder.file           0
LogDecoder.logdecoder.netflow        0
LogDecoder.logdecoder.odbc           0
LogDecoder.logdecoder.sdee           0
LogDecoder.logdecoder.snmptrap       0
LogDecoder.logdecoder.syslog         0
LogDecoder.logdecoder.vmware         0
LogDecoder.logdecoder.windows        0
LogDecoder.logdecoder.windowslegacy  0
rabbitmq.log                         0

Symptom 2: 
 

Errors similar to the below are noticed in the RabbitMQ log file under :
 

/var/log/rabbitmq/rabbit\@<uuid>.log

 

=ERROR REPORT==== 31-Oct-2016::13:09:09 ===
SSL: certify: ssl_handshake.erl:1331:Fatal error: bad certificate

=ERROR REPORT==== 31-Oct-2016::13:09:09 ===
SSL: certify: ssl_handshake.erl:1331:Fatal error: bad certificate

=ERROR REPORT==== 31-Oct-2016::13:09:09 ===
SSL: certify: ssl_handshake.erl:1331:Fatal error: bad certificate

=ERROR REPORT==== 31-Oct-2016::13:09:11 ===
Error on AMQP connection <0.12631.0>:
{ssl_upgrade_error,{tls_alert,"bad certificate"}}

=ERROR REPORT==== 31-Oct-2016::13:09:11 ===
Error on AMQP connection <0.12635.0>:
{ssl_upgrade_error,{tls_alert,"bad certificate"}}

Symptom 3:

Errors similar to the below are noticed Log Collector Logs under:

/var/log/messages

Oct 31 11:26:19 NwLogCollector[19565]: [AMQPClientBase] [failure] An error occurred creating an AMQP channel: Error in opening SSL/TLS connection for socket
Oct 31 11:26:19 NwLogCollector[19565]: [LogdecoderProcessor] [failure] [queue.windowslegacy] [idle] Failed during getWork: Error in opening SSL/TLS connection for socket
Oct 31 11:26:19 NwLogCollector[19565]: [AMQPClientBase] [failure] An error occurred creating an AMQP channel: Error in opening SSL/TLS connection for socket
Oct 31 11:26:19 NwLogCollector[19565]: [LogdecoderProcessor] [failure] [queue.windows] [idle] Failed during getWork: Error in opening SSL/TLS connection for socket
Oct 31 11:26:19 NwLogCollector[19565]: [AMQPClientBase] [failure] An error occurred creating an AMQP channel: Error in opening SSL/TLS connection for socket
Oct 31 11:26:19 NwLogCollector[19565]: [LogdecoderProcessor] [failure] [queue.vmware] [idle] Failed during getWork: Error in opening SSL/TLS connection for socket

 

Cause

RabbitMQ certificates are not pointing to the correct files. 

The truststore.pem certificate located under:
 

/etc/netwitness/ng/rabbitmq/ssl/

should be pointing to the truststore.pem certificate file under:

/etc/rabbitmq/ssl

The cert.pem and privkey.pem certificates under: 

/etc/netwitness/ng/rabbitmq/ssl/keys

should be pointing to cert.pem and key.pem certificates respectively under: 

/etc/rabbitmq/ssl/server/

Resolution

SSH to the Log Decoder and run the below commands: 

cd /etc/netwitness/ng/rabbitmq/ssl/

ln -s /etc/rabbitmq/ssl/truststore.pem truststore.pem

cd keys/

ln -s  /etc/rabbitmq/ssl/server/cert.pem cert.pem

ln -s /etc/rabbitmq/ssl/server/key.pem privkey.pem

The output should be something like this: 
 

[root@NW11-LOG-HYBRID ~]# ls -l /etc/netwitness/ng/rabbitmq/ssl/
total 0
drwxr-xr-x. 2 rabbitmq rabbitmq 41 Jul 23  2020 keys
lrwxrwxrwx. 1 rabbitmq rabbitmq 32 Jul 23  2020 truststore.pem -> /etc/rabbitmq/ssl/truststore.pem
[root@NW11-LOG-HYBRID ~]# cd keys
-bash: cd: keys: No such file or directory
[root@NW11-LOG-HYBRID ~]# ls -l /etc/netwitness/ng/rabbitmq/ssl/keys
total 0
lrwxrwxrwx. 1 rabbitmq rabbitmq 33 Jul 23  2020 cert.pem -> /etc/rabbitmq/ssl/server/cert.pem
lrwxrwxrwx. 1 rabbitmq rabbitmq 32 Jul 23  2020 privkey.pem -> /etc/rabbitmq/ssl/server/key.pem