python: Error with mco ping. Please check configuration

[JosephWest1]

The network connection is established, and firewall turned off for testing. When I enable the Netwitness agent, the Security Analytics Master / Puppet Master states in /var/log/messages:

puppet-master: Compiled catalog for <key> in envorionment production 0.27 seconds

python: Adding {'node': '<certificate name> , classes " {[base': ''} to ENC database

python: Adding <certificate name> user to /rsa/system

python: signing puppet Cert

python: Pinging host <certificate name> with a 40 second timeout

python: Error with mco ping. Please check configuration.

When I run $mco ping <IP Address> I get a return.

When I run $mco ping <cert name> it returns the other certificate names in the inventory.txt, but not the one I'm trying to add.

The instructions from this site to establish a new puppet certificate has been followed from this site. I also am having an issue with RabbitMQ connecting to the puppet master.

Thank you for your time!

[JosephWest1]

Update,

   I did quite a bit of research, and found that I just need to redeploy rabbitmq/mcollective via puppet, so I'm changing direction and I am going to spend the rest of the time troubleshooting why the puppet-master isn't sending a cert to the agent.

Thanks again for the help on this!

[JosephWest1]

Update on cert issue:

When I run

$puppet agent -t

The service starts to load, then I receive the error:

Error opening Certificates /var/lib/puppet/ssl/certs/<cert_name>

13970843655592:error:02001002:system library:fopen:No such file or directory:bss_file.c:398:fopen('/var/lib/puppet/ssl/certs/<cert_name>' , 'r')

13970843655592:error:20074002:BIO routines:FILE_CTRL:system lib:bss_file.c:400:

unable to load certificate

sh: line 1: .pem: command not found

Info: Caching catalog for <cert_name>

I traced this to an openssl issue. I tested the cert by running the following command from /var/lib/puppet/ssl/certs directory:

$openssl req -x509 -in /var/lib/puppet/ssl/certs/<cert_name.pem> -noout -subject

(output)  Subject= /CN=<cert_name>

Then when I ran

$openssl req -noout -text -in <cert_name.pem>

I received the output:

<cert_name.pem>: No such file or directory

13970843655592:error:02001002:system library:fopen:No such file or directory:bss_file.c:398:fopen('/var/lib/puppet/ssl/certs/<cert_name>' , 'r')

13970843655592:error:20074002:BIO routines:FILE_CTRL:system lib:bss_file.c:400:

unable to load certificate

So we can clearly see it is related to an issue with the SSL cert. I have revoked, cleared, and removed the ssl certs on the puppet master and the client multiple times, however nothing seems to issue a new cert.

[DavidPoirier]

Hi Joseph,

As a test. Did the agent ever send a cert to the puppetmaster?

1. Puppet cert clean node_id of the agent.

2. Also check, /var/lib/puppet/ssl/ca/inventory.txt “ look for the node_id of the agent”

3. Stop rabbitmq-server on the agent, make sure that everything is down, ps-eaf | grep rabbit, The only thing you should see if your grep.

4. Open a secondary putty window. This windows will be used to force start rabbitmq-server if it’s not currently running

5. On the first putty session perform the puppet agent –t –waitforcert=30

6. Having a windows on the SA Head, use this to view via tailf /var/log/messages and look for the cert coming across.

If you have no cert request, I would wonder if iptables is messed up, you can test by service iptables stop.

David

[DavidPoirier]

Hi Joseph,

If you would like to open an RSA case, I would be happy to work with you.

This would allow us to perform a WebEx.

David

[JosephWest1]

Thank you for your time David,

         My team opened a ticket for this yesterday. They are working on it now. Thank you again!

[RamisKayim]

Joseph How solve you thıs problem?  We face to same issues. Could you please share your solution?

sh: line 1: .pem: command not found

[JosephWest1]

There were multiple issues as follows:

The NTP needed to be sync'd.

The MCollective wasn't working, because the facts.yaml wouldn't fill itself in.

RabbitMQ would not start on the device because it could not pull the propper certificates from the Puppet Master.

To fix this, we did the following:

Manually ran yum updates, as the connection between the head unit and the device was to low for the GUI to update.

Sync the NTP to the head unit, and ensure the firewall on the device was not blocking it.

RabbitMQ started immediately, once the firewall, NTP, and the system was updated. We had another device., that was not fixed by these actions, so we coppied some of the configuration files (facts.yaml) to the device, and that also fixed it.

Hope this helps!