RSA NetWitness Host Fails to Upgrade but no new Chef Logs are being generated

[#1]

Article Number

000001493

Applies To

RSA Product Set: NetWitness Logs & Network
RSA Product/Service Type: Admin Server, Component Host
RSA Version/Condition: 11.X
Platform: CentOS 7

Issue

When reattempting to upgrade a NetWitness 11 host, you may see no new log messages being generated in /var/log/netwitness/config-management/chef-solo.log. Therefore, the upgrade does not seem like it's being started.

Cause

EXAMPLE 1:
If your upgrade doesn't get to the point that chef logs are being generated on the component host, generally, you can find a clue in the Orchestration Server logs. These are located in /var/log/netwitness/orchestration-server/orchestration-server.log on the NetWitness Admin Server. A very common situation includes a message like the following:

018-01-30 19:59:02,629 [      ServiceInstallJobHandler] WARN  ConfigurationManagement|Failed to check isJobReady
org.springframework.http.converter.HttpMessageNotReadableException: Could not read document: Can not deserialize instance of java.util.LinkedHashMap out of VALUE_FALSE token
 at [Source: java.io.PushbackInputStream@702de5ea; line: 1, column: 54] (through reference chain: com.rsa.asoc.orchestration.salt.client.domain.LocalResponse["return"]->
 java.util.ArrayList[0]->java.util.LinkedHashMap["992dcb26-39c2-4c29-b9c9-7f5e98f3c542"]); 
 nested exception is com.fasterxml.jackson.databind.JsonMappingException: Can not deserialize instance of java.util.LinkedHashMap out of VALUE_FALSE token
 at [Source: java.io.PushbackInputStream@702de5ea; line: 1, column: 54] (through reference chain: com.rsa.asoc.orchestration.salt.client.domain.LocalResponse["return"]->
java.util.ArrayList[0]->java.util.LinkedHashMap["992dcb26-39c2-4c29-b9c9-7f5e98f3c542"])
        at org.springframework.http.converter.json.AbstractJackson2HttpMessageConverter.readJavaType(AbstractJackson2HttpMessageConverter.java:240)
        at org.springframework.http.converter.json.AbstractJackson2HttpMessageConverter.read(AbstractJackson2HttpMessageConverter.java:225)
        at org.springframework.web.client.HttpMessageConverterExtractor.extractData(HttpMessageConverterExtractor.java:95)
        at org.springframework.web.client.RestTemplate$ResponseEntityResponseExtractor.extractData(RestTemplate.java:917)
        at org.springframework.web.client.RestTemplate$ResponseEntityResponseExtractor.extractData(RestTemplate.java:901)
        at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:655)
        at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:628)
        at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:602)
        at com.rsa.asoc.orchestration.salt.client.DefaultSaltClient.run(DefaultSaltClient.java:118)
        at com.rsa.asoc.orchestration.task.AsynchronousTaskMonitor.isJobReady(AsynchronousTaskMonitor.java:115)
        at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:174)
        at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
        at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1374)
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
        at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
        at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
        at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
        at com.rsa.asoc.orchestration.task.AsynchronousTaskMonitor.pollTasks(AsynchronousTaskMonitor.java:88)
        at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1626)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: com.fasterxml.jackson.databind.JsonMappingException: Can not deserialize instance of java.util.LinkedHashMap out of VALUE_FALSE token
 at [Source: java.io.PushbackInputStream@702de5ea; line: 1, column: 54] (through reference chain: com.rsa.asoc.orchestration.salt.client.domain.LocalResponse["return"]->
java.util.ArrayList[0]->java.util.LinkedHashMap["992dcb26-39c2-4c29-b9c9-7f5e98f3c542"])
        at com.fasterxml.jackson.databind.JsonMappingException.from(JsonMappingException.java:270)
        at com.fasterxml.jackson.databind.DeserializationContext.reportMappingException(DeserializationContext.java:1234)
        at com.fasterxml.jackson.databind.DeserializationContext.handleUnexpectedToken(DeserializationContext.java:1122)
        at com.fasterxml.jackson.databind.DeserializationContext.handleUnexpectedToken(DeserializationContext.java:1075)
        at com.fasterxml.jackson.databind.deser.std.StdDeserializer._deserializeFromEmpty(StdDeserializer.java:892)
        at com.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(MapDeserializer.java:358)
        at com.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(MapDeserializer.java:27)
        at com.fasterxml.jackson.databind.deser.std.MapDeserializer._readAndBindStringKeyMap(MapDeserializer.java:517)
        at com.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(MapDeserializer.java:362)
        at com.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(MapDeserializer.java:27)
        at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:287)
        at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:259)
        at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:26)
        at com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:499)
        at com.fasterxml.jackson.databind.deser.impl.MethodProperty.deserializeAndSet(MethodProperty.java:101)
        at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:357)
        at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:148)
        at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3798)
        at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2922)
        at org.springframework.http.converter.json.AbstractJackson2HttpMessageConverter.readJavaType(AbstractJackson2HttpMessageConverter.java:237)
        ... 23 common frames omitted

This could indicate that there is a problem with the salt service on the host being upgraded that prevents the orchestration server from communicating with it properly. As part of the chef run, we mask the salt-minion service to prevent users from interacting with it until the chef process is done. In the case of a previous failure, we may never get to the step that we unmask the salt service, thus the above error is displayed.

EXAMPLE 2:

Another situation that can occur is a message like the following:

2018-12-17 23:44:10,004 [                    cfg-mgmt-0] ERROR       TaskManagement|Execution of '5c1834c991c71f054873335b' failed
org.springframework.web.client.ResourceAccessException: I/O error on POST request for "https://localhost:8000/run": Connect to localhost:8000 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: 
Connection refused (Connection refused); nested exception is org.apache.http.conn.HttpHostConnectException: 
Connect to localhost:8000 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused)
	at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:674)
	at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:636)
	at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:610)
	at com.rsa.asoc.orchestration.salt.client.DefaultSaltClient.run(DefaultSaltClient.java:118)
	at com.rsa.asoc.orchestration.salt.handler.HostVerifyTaskHandler.execute(HostVerifyTaskHandler.java:52)
	at com.rsa.asoc.orchestration.task.TaskExecutionService.execute(TaskExecutionService.java:163)
	at com.rsa.asoc.orchestration.task.TaskExecutionService.lambda$null$3(TaskExecutionService.java:147)
	at java.util.ArrayList.forEach(ArrayList.java:1257)
	at com.rsa.asoc.orchestration.task.TaskExecutionService.lambda$submit$4(TaskExecutionService.java:147)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.http.conn.HttpHostConnectException: Connect to localhost:8000 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused)
	at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:159)
	at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:373)
	at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
	at org.springframework.http.client.HttpComponentsClientHttpRequest.executeInternal(HttpComponentsClientHttpRequest.java:89)
	at org.springframework.http.client.AbstractBufferingClientHttpRequest.executeInternal(AbstractBufferingClientHttpRequest.java:48)
	at org.springframework.http.client.AbstractClientHttpRequest.execute(AbstractClientHttpRequest.java:53)
	at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:660)
	... 13 common frames omitted

This is also related to another salt issue but you are more likely to see this just on the Admin Server itself rather than another host.

Resolution

If you find yourself fitting into Example 1, then do the following:
Run the following on the target host to unmask and restart the service to allow the upgrade to, properly.

systemctl unmask salt-minion
systemctl restart salt-minion

If you are finding yourself fitting into Example 2, then do the above AND also the following on the Admin Server:

systemctl unmask salt-master && systemctl unmask salt-api
systemctl restart salt-master && systemctl restart salt-api

Then, you may start your upgrade once more on your target host.
Be sure to tail the /var/log/netwitness/config-management/chef-solo.log files to watch the upgrade in action.

tailf /var/log/netwitness/config-management/chef-solo.log

 

Notes

If you are still having an issue, please submit a case to RSA Technical Support and quote this KB article. It would also help us if you can attach the /var/log/netwitness/orchestration-server/orchestration-server.log file from your NetWitness Admin Server and your /var/log/netwitness/config-management/chef-solo.log from the host that you are attempting to upgrade.