The command "orchestration-cli-client --update-admin-node" fails while trying to import certificates in RSA Platform

[#1]

Article Number

000001788

Applies To

RSA Product Set: RSA NetWitness Logs & Network
RSA Product/Service Type: RSA NetWitness Host
RSA Version/Condition: 11.x
Platform: CentOS
O/S Version: EL7

Issue

When attempting to integrate the RSA Archer Platform or RSA NetWitness Endpoint with the RSA NetWitness Respond Server, you may encounter an error when trying to import certificates into the RSA NetWitness Admin Server RabbitMQ keystore. After placing your .pem file(s) into the /etc/pki/nw/trust/import folder and running orchestration-cli-client --update-admin-node, you may encounter an error like the following:

2018-06-13 16:51:35,313 [                          main] INFO             Bootstrap|Service logs will be written to /var/log/netwitness/orchestration-client
2018-06-13 16:51:35,319 [                          main] INFO             Bootstrap|Service configuration will be read from /etc/netwitness/orchestration-client
2018-06-13 16:51:35,696 [                          main] INFO             Bootstrap|Starting orchestration-client.87d919c0-20d3-4397-8d50-728bda6ae8ff (v0.0.0.0)
2018-06-13 16:51:36,191 [                          main] INFO             Bootstrap|Initialized service cryptography with 4 providers (BSAFE=CRYPTOJ 6.2.2 20161215 0745, FIPS-140=true).
2018-06-13 16:51:36,973 [                          main] INFO             Bootstrap|A version change was detected and an upgrade is not required.
2018-06-13 16:51:37,133 [                          main] INFO  c.r.n.i.o.c.OrchestrationApplication|Starting OrchestrationApplication on netwitnesssa with PID 190674 (/usr/bin/orchestration-cli-client.jar started by root in /etc/pki/nw/trust/import)
2018-06-13 16:51:37,133 [                          main] INFO  c.r.n.i.o.c.OrchestrationApplication|The following profiles are active: standard
2018-06-13 16:51:37,212 [                          main] INFO             Bootstrap|Service will accept AMQP requests at broker localhost:5672/rsa/system
2018-06-13 16:51:37,215 [                          main] INFO             Bootstrap|Service will use the deployment security-server
2018-06-13 16:51:38,483 [    Notify Handshake Completed] INFO              Security|Accepted new connection with CN=d4ee2ca7-b16a-48c0-8f14-7f14d1c4d6cf,OU=NetWitness Platform,O=RSA,L=Reston,ST=VA,C=US from 127.0.0.1 using TLS_DHE_RSA_WITH_AES_128_GCM_SHA2
56
2018-06-13 16:51:39,275 [                          main] INFO  c.r.n.i.o.c.LaunchHelper|Task [Refresh Host] running (polling 599 more times)...
2018-06-13 16:51:45,289 [                          main] INFO  c.r.n.i.o.c.LaunchHelper|Task [Refresh Host] running (polling 598 more times)...
2018-06-13 16:51:51,302 [                          main] INFO  c.r.n.i.o.c.LaunchHelper|Task [Refresh Host] running (polling 597 more times)...
2018-06-13 16:51:57,313 [                          main] INFO  c.r.n.i.o.c.LaunchHelper|Task [Refresh Host] running (polling 596 more times)...
2018-06-13 16:52:03,322 [                          main] INFO  c.r.n.i.o.c.LaunchHelper|Task [Refresh Host] running (polling 595 more times)...
2018-06-13 16:52:09,335 [                          main] INFO  c.r.n.i.o.c.LaunchHelper|Task [Refresh Host] running (polling 594 more times)...
2018-06-13 16:52:15,347 [                          main] INFO  c.r.n.i.o.c.LaunchHelper|Task [Refresh Host] running (polling 593 more times)...
2018-06-13 16:52:21,367 [                          main] ERROR c.r.n.i.o.c.OrchestrationClient|Task [Refresh Host] stopped with errors!
2018-06-13 16:52:21,368 [                          main] ERROR c.r.n.i.o.c.OrchestrationApplication|Requested operation failed, aborting...

Cause

A number of issues could cause this particular operation to fail. For instance, the files that exist in /etc/pki/nw/trust/import MUST be in Base64 encoding format. Anything else will cause the process to fail. You can rename any .pem file .cer file as long as it is in Base64 Encoding (As required by the Endpoint Integration). You can review the following logs files on the NetWitness Admin Server to see which file(s) could be causing the problem:

• /var/log/netwitness/config-management/chef-solo.log
• /var/lib/netwitness/config-management/cache/chef-stacktrace.out   (if present)

Here is an example from the chef-stacktrace.out file:

>>>> Caused by Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of openssl pkcs12 -in /etc/pki/nw/trust/truststore.p12 -out /tmp/openssl-trust.daf3e46f-38b9-4fc4-8d93-73e7ba45644c -passin pass:changeit && openssl pkcs12 -keypbe PBE-SHA1-3DES -certpbe PBE-SHA1-3DES -export -nokeys -name rootcastore.crt -in /tmp/openssl-trust.daf3e46f-38b9-4fc4-8d93-73e7ba45644c -certfile /etc/pki/nw/trust/import/rootcastore.crt.pem -out /etc/pki/nw/trust/truststore.p12 -passout pass:changeit && rm -f /tmp/openssl-trust.daf3e46f-38b9-4fc4-8d93-73e7ba45644c && chown netwitness:nwpki /etc/pki/nw/trust/truststore.p12 && chmod 640 /etc/pki/nw/trust/truststore.p12 && echo '/etc/pki/nw/trust/import/rootcastore.crt.pem' >> /etc/pki/nw/trust/truststore.p12.idx ----
STDOUT: 
STDERR: 139904017254304:error:0D07207B:asn1 encoding routines:ASN1_get_object:header too long:asn1_lib.c:157:
---- End output of openssl pkcs12 -in /etc/pki/nw/trust/truststore.p12 -out /tmp/openssl-trust.daf3e46f-38b9-4fc4-8d93-73e7ba45644c -passin pass:changeit && openssl pkcs12 -keypbe PBE-SHA1-3DES -certpbe PBE-SHA1-3DES -export -nokeys -name rootcastore.crt -in /tmp/openssl-trust.daf3e46f-38b9-4fc4-8d93-73e7ba45644c -certfile /etc/pki/nw/trust/import/rootcastore.crt.pem -out /etc/pki/nw/trust/truststore.p12 -passout pass:changeit && rm -f /tmp/openssl-trust.daf3e46f-38b9-4fc4-8d93-73e7ba45644c && chown netwitness:nwpki /etc/pki/nw/trust/truststore.p12 && chmod 640 /etc/pki/nw/trust/truststore.p12 && echo '/etc/pki/nw/trust/import/rootcastore.crt.pem' >> /etc/pki/nw/trust/truststore.p12.idx ----
Ran openssl pkcs12 -in /etc/pki/nw/trust/truststore.p12 -out /tmp/openssl-trust.daf3e46f-38b9-4fc4-8d93-73e7ba45644c -passin pass:changeit && openssl pkcs12 -keypbe PBE-SHA1-3DES -certpbe PBE-SHA1-3DES -export -nokeys -name rootcastore.crt -in /tmp/openssl-trust.daf3e46f-38b9-4fc4-8d93-73e7ba45644c -certfile /etc/pki/nw/trust/import/rootcastore.crt.pem -out /etc/pki/nw/trust/truststore.p12 -passout pass:changeit && rm -f /tmp/openssl-trust.daf3e46f-38b9-4fc4-8d93-73e7ba45644c && chown netwitness:nwpki /etc/pki/nw/trust/truststore.p12 && chmod 640 /etc/pki/nw/trust/truststore.p12 && echo '/etc/pki/nw/trust/import/rootcastore.crt.pem' >> /etc/pki/nw/trust/truststore.p12.idx returned 1

From the above, we can conclude that it is having a problem with the /etc/pki/nw/trust/import/rootcastore.crt.pem file. 

Generally, this is a result of badly formatted .pem files for one reason or another. Here is an example of one such file:

-----BEGIN CERTIFICATE-----
MIID/jCCAuagAwIBAgIQdHbFBN/MNoXVPSOC2q8NrzANBgkqhkiG9w0BAQsFADB9MR8wHQYDVQQM^M
ExZETFAgRW50ZXJwcmlzZSBNYW5hZ2VyMSowKAYDVQQKFCFSU0EsIFRoZSBTZWN1cml0eSBEaXZp^M
c2lvbiBvZiBFTUMxHTAbBgNVBAsUFERhdGEgTG9zcyBQcmV2ZW50aW9uMQ8wDQYDVQQDFAZzZWNv^M
cHMwHhcNMTgwMTA1MDQzMTEwWhcNMjAwMTA1MDQzMTEwWjB9MR8wHQYDVQQMExZETFAgRW50ZXJw^M
cmlzZSBNYW5hZ2VyMSowKAYDVQQKFCFSU0EsIFRoZSBTZWN1cml0eSBEaXZpc2lvbiBvZiBFTUMx^M
HTAbBgNVBAsUFERhdGEgTG9zcyBQcmV2ZW50aW9uMQ8wDQYDVQQDFAZhcmNoZXIwggEiMA0GCSqG^M
SIb3DQEBAQUAA4IBDwAwggEKAoIBAQCaBYac6dT/lLmwQqIHrROcARxuEdxly8H/6/gxnG2M9uNc^M
Lw5xlKaqXVwglIjEdERmrdoctf7s6oGg3etD55xsF9MpR/4OVI0VqN0Kj0X0R8VHI8BNVLW8r+Mh^M
8phfrJJ/lOdeGmDOztn+XraOtkRTg3ffQQx00mQJnrGDV8uoUki/ceFcRkYIrSKWlX6mpBYYdzEH^M
I1Yfesafth4PYk3Y7WA2FFYsD8Hpcu7QRWMZhH9VBSXEvukNhnAr7IhrKpz4JnzA448yIqJeyKNB^M
SQZD5jrHgRSodieiLuE5A3tzxye9AJJ23IEBx4TvpkG8Tm0Z0bZir0O3fXlrjdBI22hRAgMBAAGj^M
ejB4MA4GA1UdDwEB/wQEAwIDuDAgBgNVHSUBAf8EFjAUBggrBgEFBQcDAgYIKwYBBQUHAwEwDAYD^M
VR0TAQH/BAIwADAZBgNVHQ4EEgQQHuBXnY0khPz5HHryIvUcvTAbBgNVHSMEFDASgBDJKEpJIFaZ^M
tPMGBOP8t/81MA0GCSqGSIb3DQEBCwUAA4IBAQA6BhNaGTeDsQ/Y3lVDTGcXiGFKerTq1mYxQ0In^M
8yVHYvelZhWSCyJ4//5jh8H9WPjMqpakb4FgHVTKlnOFtrphgca53jYLguVJOgzd7kt5qBXdxxsI^M
EEWGAm8sQmzeAY1MNAwIt+fHe1hS8G3WF6oF5fsi9XkFGmEwZgto45tkMLm5Obhy4qSRvo2oZuTb^M
0YjO7bQerEg53jyWfbbkZn1jTiRa4AIbqmPWldspcXYFXxs+QiBT4U8QX5uZLj5PO9D1hSPQtSi/^M
+O3pTczC3wYau167ADb0lGppzHWTD1UN8S8SHduj0gHfrtTJKptsoWdyPYFEkeTo+bBSWyyez+WX-----END CERTIFICATE-----

The above file has two problems. If these files originated from a Microsoft Windows device, they may be filled with Windows endline characters ("^M") which violate the structure of the .pem file. This can also happen if you open a file in Windows before moving it over to the NetWitness Admin Server. These Windows endlines must be removed before the import can be completed successfully. In addition, the "------END CERTIFICATE-----" must be on its own line. You may need to open the file in vi to see them if they are not visible to you.
 

Resolution

You can now open the file in vi to check/modify its content

vi /etc/pki/nw/trust/import/rootcastore.crt.pem

Delete the  Windows Carriage Return (CR) characters (denoted as '^M') characters from the end of the lines. I would also suggest you run the following in vi:

:%s/\r//g

Also ensure that the "------END CERTIFICATE-----" is on its own line.

Your finished file will look similar to the following:

-----BEGIN CERTIFICATE-----
MIID/jCCAuagAwIBAgIQdHbFBN/MNoXVPSOC2q8NrzANBgkqhkiG9w0BAQsFADB9MR8wHQYDVQQM
ExZETFAgRW50ZXJwcmlzZSBNYW5hZ2VyMSowKAYDVQQKFCFSU0EsIFRoZSBTZWN1cml0eSBEaXZp
c2lvbiBvZiBFTUMxHTAbBgNVBAsUFERhdGEgTG9zcyBQcmV2ZW50aW9uMQ8wDQYDVQQDFAZzZWNv
cHMwHhcNMTgwMTA1MDQzMTEwWhcNMjAwMTA1MDQzMTEwWjB9MR8wHQYDVQQMExZETFAgRW50ZXJw
cmlzZSBNYW5hZ2VyMSowKAYDVQQKFCFSU0EsIFRoZSBTZWN1cml0eSBEaXZpc2lvbiBvZiBFTUMx
HTAbBgNVBAsUFERhdGEgTG9zcyBQcmV2ZW50aW9uMQ8wDQYDVQQDFAZhcmNoZXIwggEiMA0GCSqG
SIb3DQEBAQUAA4IBDwAwggEKAoIBAQCaBYac6dT/lLmwQqIHrROcARxuEdxly8H/6/gxnG2M9uNc
Lw5xlKaqXVwglIjEdERmrdoctf7s6oGg3etD55xsF9MpR/4OVI0VqN0Kj0X0R8VHI8BNVLW8r+Mh
8phfrJJ/lOdeGmDOztn+XraOtkRTg3ffQQx00mQJnrGDV8uoUki/ceFcRkYIrSKWlX6mpBYYdzEH
I1Yfesafth4PYk3Y7WA2FFYsD8Hpcu7QRWMZhH9VBSXEvukNhnAr7IhrKpz4JnzA448yIqJeyKNB
SQZD5jrHgRSodieiLuE5A3tzxye9AJJ23IEBx4TvpkG8Tm0Z0bZir0O3fXlrjdBI22hRAgMBAAGj
ejB4MA4GA1UdDwEB/wQEAwIDuDAgBgNVHSUBAf8EFjAUBggrBgEFBQcDAgYIKwYBBQUHAwEwDAYD
VR0TAQH/BAIwADAZBgNVHQ4EEgQQHuBXnY0khPz5HHryIvUcvTAbBgNVHSMEFDASgBDJKEpJIFaZ
tPMGBOP8t/81MA0GCSqGSIb3DQEBCwUAA4IBAQA6BhNaGTeDsQ/Y3lVDTGcXiGFKerTq1mYxQ0In
8yVHYvelZhWSCyJ4//5jh8H9WPjMqpakb4FgHVTKlnOFtrphgca53jYLguVJOgzd7kt5qBXdxxsI
EEWGAm8sQmzeAY1MNAwIt+fHe1hS8G3WF6oF5fsi9XkFGmEwZgto45tkMLm5Obhy4qSRvo2oZuTb
0YjO7bQerEg53jyWfbbkZn1jTiRa4AIbqmPWldspcXYFXxs+QiBT4U8QX5uZLj5PO9D1hSPQtSi/
+O3pTczC3wYau167ADb0lGppzHWTD1UN8S8SHduj0gHfrtTJKptsoWdyPYFEkeTo+bBSWyyez+WX
-----END CERTIFICATE-----

When you are done, we can set all endlines to be Unix just for sanity checking. Then, you can type ":wq" to exit vi and save the file.

:set ff=unix
:wq

Workaround

IMPORTANT: Once you are finished applying the changes to the files, you MUST do the following to allow for orchestration-cli-client --update-admin-node to run again. When you look inside of /etc/pki/nw/trust after your failed run, you will see something like the following:

[root@nwadmin1 trust]# ls /etc/pki/nw/trust -alh
total 32K
drwxr-x---.  3 netwitness nwpki 4.0K Apr 16 03:18 .
drwxr-x---. 16 netwitness nwpki 4.0K Mar 26 22:56 ..
drwxr-x---.  2 netwitness nwpki    6 Feb  1 22:55 import
-rw-r-----.  1 netwitness nwpki 3.9K Feb  1 22:54 truststore.jks
-rw-r--r--.  1 root       root    99 Feb  1 22:54 truststore.jks.idx
-rw-r-----.  1 netwitness nwpki    0 Apr 16 03:18 truststore.p12
-rw-r--r--.  1 root       root    58 Apr 16 03:18 truststore.p12.idx
-rw-r-----.  1 netwitness nwpki 4.0K Nov  1  2017 truststore.pem
-rw-r--r--.  1 root       root    58 Nov  1  2017 truststore.pem.idx

You will notice that the truststore.p12 file is of size 0. This happens when a failed run occurs. You will need to move this and it's sister file out of this directory to allow for them to be recreated with the appropriate certs. No data is loss when you make this move:

mv /etc/pki/nw/trust/truststore.p12* /root

After this is done and you are sure you have made the necessary changes for the .pem files to be accepted, you may run the orchestration process again:

orchestration-cli-client --update-admin-node

If these steps did not help you and you are still running into the same error, please contact RSA Customer Support and reference this article for further assistance. When creating a case revolving around this issue, it would speed up the troubleshooting process if you also attach the following files to the case:

• /var/log/netwitness/config-management/chef-solo.log
• /var/lib/netwitness/config-management/cache/chef-stacktrace.out   (if present)