Article Number
000002057
Applies To
NetWitness Product Set: NetWitness Platform
NetWitness Product/Service Type: AdminServer
NetWitness Version/Condition: 11.x, 12.x
Platform: CentOS 7 / Alma
Issue
Any changes to the nwsaltuser on the OS level, such as changing the user password or locking the user, orchestartion-cli-client fails to run commands related to the salt keys (using the --list-keys , --accept-key , --remove-key options).
Cause
The following error is displayed in the command line and also in /var/log/netwitness/orchestration-client/orchestration-client.log :
11.x Exception
2021-09-09 09:20:34,669 [ main] ERROR c.r.n.i.o.c.OrchestrationApplication|Exception processing request
com.rsa.asoc.launch.api.transport.client.TransportClientException: 401 Unauthorized
at com.rsa.asoc.launch.api.transport.client.ClientResponseUtils.handleError(ClientResponseUtils.java:99)
at com.rsa.asoc.launch.api.transport.client.AmqpTransportClient.doSendAndReceive(AmqpTransportClient.java:118)
at com.rsa.asoc.launch.api.transport.client.AmqpTransportClient.send(AmqpTransportClient.java:69)
at com.rsa.asoc.launch.api.transport.client.TransportClientInvocationHandler.makeRemoteCall(TransportClientInvocationHandler.java:68)
at com.rsa.asoc.launch.api.transport.client.TransportClientInvocationHandler.invoke(TransportClientInvocationHandler.java:50)
at com.sun.proxy.$Proxy48.remove(Unknown Source)
at com.rsa.netwitness.infrastructure.orchestration.client.commands.RemoveKey.execute(RemoveKey.java:29)
at com.rsa.netwitness.infrastructure.orchestration.client.commands.OrchestrationCommand.call(OrchestrationCommand.java:178)
at com.rsa.netwitness.infrastructure.orchestration.client.commands.OrchestrationCommand.call(OrchestrationCommand.java:38)
at picocli.CommandLine.executeUserObject(CommandLine.java:1743)
at picocli.CommandLine.access$900(CommandLine.java:145)
at picocli.CommandLine$RunAll.handle(CommandLine.java:2157)
at picocli.CommandLine$RunAll.handle(CommandLine.java:2116)
at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:1935)
at picocli.CommandLine.execute(CommandLine.java:1864)
at com.rsa.netwitness.infrastructure.orchestration.client.OrchestrationApplication.run(OrchestrationApplication.java:110)
at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:781)
at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:765)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:319)
at org.springframework.boot.builder.SpringApplicationBuilder.run(SpringApplicationBuilder.java:140)
at com.rsa.netwitness.infrastructure.orchestration.client.OrchestrationApplication.main(OrchestrationApplication.java:82)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:87)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:51)
at org.springframework.boot.loader.PropertiesLauncher.main(PropertiesLauncher.java:578)
12.x Exception:
2024-04-24 20:34:21.672 ERROR 15681 --- [ main] c.r.n.i.o.c.OrchestrationApplication : Exception processing request
com.rsa.asoc.launch.api.transport.client.TransportClientException: 401 Unauthorized: "<!DOCTYPE html PUBLIC<EOL>"-//W3C//DTD XHTML 1.0 Transitional//EN"<EOL>"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"><EOL><html><EOL><head><EOL> <meta http-equiv="Content-Type" content="text/html; charset=utf-8"></meta><EOL> <title>401 Unauthorized</title><EOL> <style type="text/css"><EOL> #powered_by {<EOL> margin-top: 20px;<EOL> border-top: 2px solid black;<EOL> font-style: italic;<EOL> }<EOL><EOL> #traceback {<EOL> color: red;<EOL> }<EOL> </style><EOL></head><EOL> <body><EOL> <h2>401 Unauthorized</h2><EOL> <p>No permission -- see authorization schemes</p><EOL> <pre id="traceback"></pre><EOL> <div id="powered_by"><EOL> <span><EOL> Powered by <a href="http://www.cherrypy.org">CherryPy 5.6.0</a><EOL> </span><EOL> </div><EOL> </body><EOL></html><EOL>"
at com.rsa.asoc.launch.api.transport.client.ClientResponseUtils.handleError(ClientResponseUtils.java:101)
at com.rsa.asoc.launch.api.transport.client.AmqpTransportClient.doSendAndReceive(AmqpTransportClient.java:120)
at com.rsa.asoc.launch.api.transport.client.AmqpTransportClient.send(AmqpTransportClient.java:90)
at com.rsa.asoc.launch.api.transport.client.TransportClientInvocationHandler.makeRemoteCall(TransportClientInvocationHandler.java:71)
at com.rsa.asoc.launch.api.transport.client.TransportClientInvocationHandler.invoke(TransportClientInvocationHandler.java:52)
at com.sun.proxy.$Proxy50.getAll(Unknown Source)
at com.rsa.netwitness.infrastructure.orchestration.client.commands.ListKeys.execute(ListKeys.java:28)
at com.rsa.netwitness.infrastructure.orchestration.client.commands.OrchestrationCommand.call(OrchestrationCommand.java:180)
at com.rsa.netwitness.infrastructure.orchestration.client.commands.OrchestrationCommand.call(OrchestrationCommand.java:40)
at picocli.CommandLine.executeUserObject(CommandLine.java:1933)
at picocli.CommandLine.access$1100(CommandLine.java:145)
at picocli.CommandLine$RunAll.recursivelyExecuteUserObject(CommandLine.java:2402)
at picocli.CommandLine$RunAll.recursivelyExecuteUserObject(CommandLine.java:2404)
at picocli.CommandLine$RunAll.handle(CommandLine.java:2399)
at picocli.CommandLine$RunAll.handle(CommandLine.java:2363)
at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2159)
at picocli.CommandLine.execute(CommandLine.java:2058)
at com.rsa.netwitness.infrastructure.orchestration.client.OrchestrationApplication.run(OrchestrationApplication.java:112)
at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:768)
at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:752)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:314)
at org.springframework.boot.builder.SpringApplicationBuilder.run(SpringApplicationBuilder.java:164)
at com.rsa.netwitness.infrastructure.orchestration.client.OrchestrationApplication.main(OrchestrationApplication.java:84)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:108)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:58)
at org.springframework.boot.loader.PropertiesLauncher.main(PropertiesLauncher.java:467)
Resolution
Restore the nwsaltuser on the Admin node using the following command:
1. Set the password of nwsaltuser.
security-cli-client --get-config-prop --prop-hierarchy nw.orchestration-server --prop-name rsa.orchestration.engine.salt.client.password -q | passwd --stdin nwsaltuser
2. Unlock the nwsaltuser.
faillock --user nwsaltuser --reset
passwd -u nwsaltuser
3. Set the user to never expire.
chage -M -1 nwsaltuser
4. Restart the salt master and salt api.
systemctl restart salt-master salt-api
Notes
Note: The above issue will prevent the hosts from being discovered on the UI and it might also cause a cert-reissue to fail while verifying hosts.