By default, NetWitness Endpoint 11.x creates a self-signed Certificate Authority during its initial installation, and uses this CA to generate certificates for the endpoint agent and the local reverse proxy that handles all incoming agent communications. Because all these certificates are generated from the same CA chain, they automatically trust each other and enable seamless, easy, and secure communications between agents and the endpoint server.
But what if this self-signed CA cannot be used within your organization? For a number of very valid reasons, many orgs might not allow software using a self-signed certificate, and may instead be required to use their own trusted CAs. If this is the case, we have a couple options - an easy way, and a hard way.
This blog covers the hard way.
Everything that we do in the hard way must occur after the Endpoint Log Hybrid host has been fully installed and provisioned. This means you'll need to complete the entire host installation before moving on to this process.
There are 2 primary requirements for the hard way:
- you must be able to create a server certificate and private key capable Server Authentication
- you must be able to create a client certificate and private key capable of Client Authentication
- this client certificate must have Common Name (CN) value of rsa-nw-endpoint-agent
I won't be going into details on how to generate these certificates and keys - your org should have some kind of process in place for this. And since the certificates and keys generated from that process can output in a number of different formats, I won't be going into details on how to convert or reformat them. There are numerous guides, documents, and instructions online to help with that.
Once we have our server and client certificates and keys, make sure to also grab the CA chain used to generate them (at the very least, both certs need to have a common Root or Intermediate CA to be part of the same trusted chain). This should hopefully be available through the same process used to create the certs and keys. If not, we can also export CA chains from websites - if you do this, make sure it is the same chain used to create your certificates and keys.
The endstate format that we'll need for everything will be PEM. The single server and/or client cert should look like this:
-----BEGIN CERTIFICATE-----
MIIFODCCAyCgAwIBAgICEAEwDQYJKoZIhvcNAQELBQAwEjEQMA4GA1UEAwwHUm9v
dC1jYTAeFw0yMDA4MDUyMDE0MTdaFw0zMDA4MDMyMDE0MTdaMCUxIzAhBgNVBAMM
....snip....
-----END CERTIFICATE-----
The private key should look like this:
-----BEGIN PRIVATE KEY-----
MIIJQwIBADANBgkqhkiG9w0BAQEFAASCCS0wggkpAgEAAoICAQCuUtxhFPb+FtWD
mQyIELpYVW7isU2KA7ur6ZhWDnKI6pD1POYHfyftO6MhxYsaRrwQ+XxhRJhyT/Ht
....snip....
-----END PRIVATE KEY-----
And the Certificate Chain should look this (one BEGIN-END block per CA certificate in the chain...also, it will help to simplify the rest of the process if this chain only includes CA certificates):
-----BEGIN CERTIFICATE-----
MIIFODCCAyCgAwIBAgICEAEwDQYJKoZIhvcNAQELBQAwEjEQMA4GA1UEAwwHUm9v
dC1jYTAeFw0yMDA4MDUyMDE0MTdaFw0zMDA4MDMyMDE0MTdaMCUxIzAhBgNVBAMM
....snip....
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIFBzCCAu+gAwIBAgIJAK5iXOLV5WZQMA0GCSqGSIb3DQEBCwUAMBIxEDAOBgNV
BAMMB1Jvb3QtY2EwHhcNMjAwODA1MTk1MTMxWhcNMzAwODAzMTk1MTMxWjASMRAw
....snip....
-----END CERTIFICATE-----
We want to make sure we have each of these PEM files for both the server and client certs and key we generated. Once we have these, we can proceed to the next set of steps.
The rest of this process will assume that all of these certificates, keys, and chains are staged on the Endpoint Log Hybrid host.
Every command we run from this point forward occurs on the Endpoint Log Hybrid.
We end up replacing a number of different files on this host, so you should also consider backup all the affected files before running the following commands.
For the server certificates:
- # cp /path/to/server/certificate.pem /etc/pki/nw/web/endpoint-web-server-cert.pem
- # cp /path/to/server/key.pem /etc/pki/nw/web/endpoint-web-server-key.pem
- # cat /path/to/server/certificate.pem > /etc/pki/nw/web/endpoint-web-server-cert.chain
- # cat /path/to/ca/chain.pem >> /etc/pki/nw/web/endpoint-web-server-cert.chain
- # openssl crl2pkcs7 -nocrl -certfile /path/to/server/certificate.pem -certfile /path/to/ca/chain.pem -out /etc/pki/nw/web/endpoint-web-server-cert.p7b
- # cp /path/to/ca/chain.pem /etc/pki/nw/nwe-trust/truststore.pem
- # cp /path/to/ca/chain.pem /etc/pki/nw/nwe-ca/customrootca-cert.pem
- # echo "/etc/pki/nw/nwe-ca/customrootca-cert.pem" > /etc/pki/nw/nwe-trust/truststore.p12.idx
- # echo "/etc/pki/nw/nwe-ca/customrootca-cert.pem" > /etc/pki/nw/nwe-trust/truststore.pem.idx
The end results, with all the files we modified and replaced, should be:
Once we're confident we've completed these steps, run:
- # systemctl restart nginx
We can verify that everything so far has worked by browsing to https://<endpoint_server_IP_or_FQDN> and checking the certificate presented by the server:
If this matches our server certificate and chain, then we can move on to the client certificates. If not, then we need to go back and figure out which step we did wrong.
For the client certificates:
- # openssl pkcs12 -export -out client.p12 -in /path/to/client/certificate.pem -inkey /path/to/client/key.pem -certfile /path/to/ca/chain.pem
...enter a password for the certificate bundle, and then SCP this client.p12 bundle onto a windows host. We'll come back to it in just a moment.
In the NetWitness UI, browse to Admin/Services --> Endpoint-Server --> Config --> Agent Packager tab. Change or validate any of the configurations you need, and then click "Generate Agent Packager." The Certificate Password field here is required to download the packager, but we won't be using the OOTB client certificate at all so don't stress about the password.
Unzip this packager onto the same windows host that has the client.p12 bundle we generated previously. Next, browse to the AgentPackager\config directory, replace the OOTB client.p12 file with the our custom-made client.p12 bundle, move back up up one directory, and run the AgentPackager.exe.
If our client.p12 bundle has been created correctly, then in the window that opens, we will be prompted for a password. This is the password we used when we ran the openssl pkcs12 command above, not the password we used in the UI to generate the packager. If they happen to be the same, fantastic....
We'll want to verify that the Client certificate and Root CA certificate thumbprints here match with our custom generated certificates.
With our newly generated agent installers, it is now time to test them. Pick a host in your org, run the appropriate agent installer, and then verify that you see the agent showing up in your UI at Investigate/Hosts.
If it does appear, congratulations! Make sure to record all these changes, and be ready to repeat them when certificates expire and agent installers need upgrading/updating.
If it doesn't, a couple things to check:
- first, give it a couple minutes...it's not going to show up instantly
- go back through all these steps and double-check that everything is correct
- check the c:\windows\temp directory for a log file with the same name as your endpoint agent; e.g.: NWEAgent.log....if there are communication errors between the agent/host and the endpoint server, this log will likely have relevant troubleshooting details
- if the agent log file has entries showing both "AgentCert" and "KnownServerCert" values, check that these thumbprints match the Client and Root CA certificate thumbprints from the AgentPackager output
- ...I was not able to consistently reproduce this issue, but it is related to how the certs and keys are bundled together in the client.p12
- ...when this happened to me, I imported my custom p12 bundle into the Windows MMC Certificates snap-in, and then exported it (make sure that the private key gets both imported and exported, as well as all the CAs in the chain), then re-ran my AgentPackger with this exported client.p12, and it fixed the error
- ... ¯\_(ツ)_/¯
- from a cmd prompt on the host, run c:\windows\system32\<service name of the agent>.exe /testnet
- check the NGINX access log on the Endpoint Log Hybrid; along with the agent log file on the endpoint, this can show whether the agent and/or server are communication properly
# tail -f /var/log/nginx/access.log
