RSA NetWitness 11.x Orchestration-cli-client failed to remove key

[#1]

Article Number

000002057

Applies To

RSA Product Set: NetWitness Platform
RSA Product/Service Type:  RSA NetWitness Platform
RSA Version/Condition: 11.6.x
Platform: CentOS
O/S Version: 7

Issue

Any changes to the nwsaltuser on the OS level, such as changing the user password or locking the user, orchestartion-cli-client fails to run commands related to the salt keys (using the --list-keys , --accept-key , --remove-key options).
 

Cause

The following error is displayed in the command line and also in /var/log/netwitness/orchestration-client/orchestration-client.log :

2021-09-09 09:20:34,669 [                          main] ERROR c.r.n.i.o.c.OrchestrationApplication|Exception processing request
com.rsa.asoc.launch.api.transport.client.TransportClientException: 401 Unauthorized
	at com.rsa.asoc.launch.api.transport.client.ClientResponseUtils.handleError(ClientResponseUtils.java:99)
	at com.rsa.asoc.launch.api.transport.client.AmqpTransportClient.doSendAndReceive(AmqpTransportClient.java:118)
	at com.rsa.asoc.launch.api.transport.client.AmqpTransportClient.send(AmqpTransportClient.java:69)
	at com.rsa.asoc.launch.api.transport.client.TransportClientInvocationHandler.makeRemoteCall(TransportClientInvocationHandler.java:68)
	at com.rsa.asoc.launch.api.transport.client.TransportClientInvocationHandler.invoke(TransportClientInvocationHandler.java:50)
	at com.sun.proxy.$Proxy48.remove(Unknown Source)
	at com.rsa.netwitness.infrastructure.orchestration.client.commands.RemoveKey.execute(RemoveKey.java:29)
	at com.rsa.netwitness.infrastructure.orchestration.client.commands.OrchestrationCommand.call(OrchestrationCommand.java:178)
	at com.rsa.netwitness.infrastructure.orchestration.client.commands.OrchestrationCommand.call(OrchestrationCommand.java:38)
	at picocli.CommandLine.executeUserObject(CommandLine.java:1743)
	at picocli.CommandLine.access$900(CommandLine.java:145)
	at picocli.CommandLine$RunAll.handle(CommandLine.java:2157)
	at picocli.CommandLine$RunAll.handle(CommandLine.java:2116)
	at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:1935)
	at picocli.CommandLine.execute(CommandLine.java:1864)
	at com.rsa.netwitness.infrastructure.orchestration.client.OrchestrationApplication.run(OrchestrationApplication.java:110)
	at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:781)
	at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:765)
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:319)
	at org.springframework.boot.builder.SpringApplicationBuilder.run(SpringApplicationBuilder.java:140)
	at com.rsa.netwitness.infrastructure.orchestration.client.OrchestrationApplication.main(OrchestrationApplication.java:82)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48)
	at org.springframework.boot.loader.Launcher.launch(Launcher.java:87)
	at org.springframework.boot.loader.Launcher.launch(Launcher.java:51)
	at org.springframework.boot.loader.PropertiesLauncher.main(PropertiesLauncher.java:578)

 

Resolution

Restore the nwsaltuser on the Admin node using the following command:
1. Set the password of nwsaltuser.

security-cli-client --get-config-prop --prop-hierarchy nw.orchestration-server --prop-name rsa.orchestration.engine.salt.client.password -q | passwd --stdin nwsaltuser

2. Unlock the nwsaltuser.

faillock --user nwsaltuser –reset passwd -u nwsaltuser

3. Set the user to never expire.

chage -M -1 nwsaltuser

4. Restart the salt master and salt api.

systemctl restart salt-master salt-api

 

Notes

Note: The above issue will prevent the hosts from being discovered on the UI and it might also cause a cert-reissue to fail while verifying hosts.