NetWItness nwbroker.service is unable to start in NetWitness Platform

[#1]

Article Number

000003215

Applies To

NetWitness Product Set: NetWitness Platform
NetWitness Product/Service Type: Concentrator
NetWitness Version/Condition: 11.x /12.X
Platform: CentOS 7 / Alma

Issue

NwBroker service was failing to start as shown below in NW

# systemctl status nwbroker 
   nwbroker.service - Netwitness Broker
   Loaded: loaded (/usr/lib/systemd/system/nwbroker.service; enabled; vendor preset: disabled)
   Active: failed (Result: start-limit) since Wed  15:20:36 KST; 24h ago
 Main PID: 259920 (code=exited, status=1/FAILURE)

Aug 21 15:20:35 mss-broker1 systemd[1]: Unit nwbroker.service entered failed state.
Aug 21 15:20:35 mss-broker1 systemd[1]: nwbroker.service failed.
Aug 21 15:20:35 mss-broker1 systemd[1]: nwbroker.service holdoff time over, scheduling restart.
Aug 21 15:20:35 mss-broker1 systemd[1]: start request repeated too quickly for nwbroker.service
Aug 21 15:20:35 mss-broker1 systemd[1]: Failed to start Netwitness Broker.
Aug 21 15:20:35 mss-broker1 systemd[1]: Unit nwbroker.service entered failed state.
Aug 21 15:20:35 mss-broker1 systemd[1]: nwbroker.service failed.
Warning: nwbroker.service changed on disk. Run 'systemctl daemon-reload' to reload units.

 

# /usr/sbin/NwBroker
(i) 2019-Aug-22 15:49:45 [Engine]  RSA NetWitness Service Copyright 2001-2019, RSA Security Inc. All Rights Reserved.
(i) 2019-Aug-22 15:49:45 [Engine]  Running broker in console
(d) 2019-Aug-22 15:49:45 [Engine]  [broker](7f1318d5d940): Entering ServiceBase::Initialize()
(d) 2019-Aug-22 15:49:45 [Engine]  [broker](7f1318d5d940): ServiceBase::SetStatus(Stopped, Start Pending)
(a) 2019-Aug-22 15:49:45 [Engine]  RSA NetWitness Service, Broker 11.3.1.0 (Jun 14 2019) 64 bit Starting
(F) 2019-Aug-22 15:49:45 [Engine]  Failed to start engine because of exception: Throw in function X509* nw::{anonymous}::getX509FromPEM(const boost::filesystem::path&)
Dynamic exception type: boost::exception_detail::clone_impl<nw::Exception>
std::exception::what: error loading trusted certificate file
[nw::ssl_error_tag*] = error:0E06D06C:configuration file routines:NCONF_get_string:no value error:02001002:system library:fopen:No such file or directory error:2006D080:BIO routines:BIO_new_file:no such file
[boost::errinfo_at_line_*] = 45
[boost::errinfo_file_name_*] = /etc/netwitness/ng/broker/trustpeers/c5al34bl.0
[boost::errinfo_api_function_*] = BIO_new_file

 

#ls -al /etc/netwitness/ng/broker/trustpeers/ <-- You need to note this information for restoration after pem file regeneration
total 0
drwxr-x---. 2 netwitness netwitness 78 May  2 05:11 .
drwxr-x---. 6 netwitness netwitness 95 May  2 05:10 ..
lrwxrwxrwx. 1 root       root       72 May  2 05:11 1386a7d5.0 -> /etc/pki/nw/peer/respond-server/12b060c9-6f99-4356-8db5-e00313277604.pem
lrwxrwxrwx. 1 root       root       40 May  2 05:10 67342faa.0 -> /etc/pki/nw/nextgen/nwappliance-cert.pem
lrwxrwxrwx. 1 root       root       30 May  2 05:10 90204291.0 -> /etc/pki/nw/node/node-cert.pem
lrwxrwxrwx. 1 root       root       67 May  2 05:10 cf280d67.0 -> /etc/pki/nw/peer/sa-server/b311eddf-8142-46bd-b801-9b80afda3dfe.pem
 

Cause

For some reason, the pem certificate file for the sa-server service id was broken or missing in /etc/pki/nw/peer/sa-server.
In this case, nwbroker service is unable to start.

Resolution

You need to re-generate the pem file for the sa-server service id in this case.

Please follow the steps below.

1. Get ssl certificate information and save it to file('root.out')

   #openssl s_client -connect localhost:7000 -tls1_2 > root.out
   depth=2 C = US, ST = VA, L = Reston, O = RSA, OU = NetWitness Platform, CN = NetWitness Root CA
   verify return:1
   depth=1 CN = NetWitness Intermediate CA, OU = NetWitness, O = RSA, L = Reston, ST = VA, C = US
   verify return:1
   depth=0 C = US, ST = VA, L = Reston, O = RSA, OU = NetWitness, CN = e431bf77-a43c-4a8a-b9cf-4ecd1b34a7c4
   verify return:1
    

   
    
2. Edit it using vi, then extract content and save it as a b311eddf-8142-46bd-b801-9b80afda3dfe.pem file.
   Note: certificate is the copy of the section:

   -----BEGIN CERTIFICATE-----
   to
   -----END CERTIFICATE-----

   #vi b311eddf-8142-46bd-b801-9b80afda3dfe.pem
    

3. Locate pem file into /etc/pki/nw/peer/sa-server/ and link it same as before.
    

   #ls -al /etc/netwitness/ng/broker/trustpeers
   total 0
   drwxr-x---. 2 netwitness netwitness 78 May  2 05:11 .
   drwxr-x---. 6 netwitness netwitness 95 May  2 05:10 ..
   lrwxrwxrwx. 1 root       root       72 May  2 05:11 1386a7d5.0 -> /etc/pki/nw/peer/respond-server/12b060c9-6f99-4356-8db5-e00313277604.pem
   lrwxrwxrwx. 1 root       root       40 May  2 05:10 67342faa.0 -> /etc/pki/nw/nextgen/nwappliance-cert.pem
   lrwxrwxrwx. 1 root       root       30 May  2 05:10 90204291.0 -> /etc/pki/nw/node/node-cert.pem
   lrwxrwxrwx. 1 root       root       67 May  2 05:10 cf280d67.0 -> /etc/pki/nw/peer/sa-server/b311eddf-8142-46bd-b801-9b80afda3dfe.pem
    

   
   Once completed, you are able to start nwbroker.service without any issue.