Article Number
000001583
Applies To
RSA Product Set: NetWitness Logs and Packets
RSA Product/Service Type: Security Analytics Server
RSA Version/Condition: 10.6.3.0
Platform: CentOS
O/S Version: 6
Issue
If rabbitmq process is suddenly stuck or hang, you are not able to login Web UI after restart jetty service.
You may see following errors during restart in jetty log(stderrout.log) under /opt/rsa/jetty9/logs.
2018-02-22 14:24:21,524 [org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-1] WARN com.rsa.smc.sa.live.job.quartz.SubscriptionJob - Resource Subscription with name [ High Risk File ] is Discontinued and hence will not be Synched
2018-02-22 14:24:42,810 [main] ERROR com.rsa.smc.sa.core.appliance.ApplianceMigrationService - Failed to connect to SMS service, make sure all services are running and restart SA web server.
com.rsa.smc.sa.core.util.carlos.ASGConnectionException: Failed to load hosts
at com.rsa.smc.sa.core.appliance.service.DefaultApplianceService$ApplianceCache.getAllAppliances(DefaultApplianceService.java:573)
at com.rsa.smc.sa.core.appliance.service.DefaultApplianceService.getAllAppliances(DefaultApplianceService.java:177)
at com.rsa.smc.sa.core.appliance.service.DefaultApplianceService.reloadAppliances(DefaultApplianceService.java:335)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:96)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:260)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:94)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at com.sun.proxy.$Proxy135.reloadAppliances(Unknown Source)
at com.rsa.smc.sa.core.appliance.ApplianceMigrationService.testSMSConnection(ApplianceMigrationService.java:171)
at com.rsa.smc.sa.core.appliance.ApplianceMigrationService.migrateApplianceInfo(ApplianceMigrationService.java:160)
at com.rsa.smc.sa.core.appliance.ApplianceMigrationService.onApplicationEvent(ApplianceMigrationService.java:123)
at com.rsa.smc.sa.core.appliance.ApplianceMigrationService.onApplicationEvent(ApplianceMigrationService.java:55)
at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:96)
at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:334)
at org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:950)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:482)
at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:410)
at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:306)
at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:112)
at com.netwitness.platform.server.common.context.SafeModeContextListener.contextInitialized(SafeModeContextListener.java:37)
at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:788)
at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:434)
at org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:780)
at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:284)
at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1322)
at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:732)
at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:490)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:118)
at org.eclipse.jetty.util.component.ContainerLifeCycle.addBean(ContainerLifeCycle.java:282)
at org.eclipse.jetty.util.component.ContainerLifeCycle.addBean(ContainerLifeCycle.java:214)
at org.eclipse.jetty.util.component.ContainerLifeCycle.updateBeans(ContainerLifeCycle.java:764)
at org.eclipse.jetty.server.handler.HandlerCollection.setHandlers(HandlerCollection.java:89)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.setHandlers(ContextHandlerCollection.java:145)
at org.eclipse.jetty.server.handler.HandlerCollection.addHandler(HandlerCollection.java:155)
at org.eclipse.jetty.deploy.bindings.StandardDeployer.processBinding(StandardDeployer.java:41)
at org.eclipse.jetty.deploy.AppLifeCycle.runBindings(AppLifeCycle.java:186)
at org.eclipse.jetty.deploy.DeploymentManager.requestAppGoal(DeploymentManager.java:495)
at org.eclipse.jetty.deploy.DeploymentManager.addApp(DeploymentManager.java:146)
at org.eclipse.jetty.deploy.providers.ScanningAppProvider.fileAdded(ScanningAppProvider.java:175)
at org.eclipse.jetty.deploy.providers.ScanningAppProvider$1.fileAdded(ScanningAppProvider.java:64)
at org.eclipse.jetty.util.Scanner.reportAddition(Scanner.java:605)
at org.eclipse.jetty.util.Scanner.reportDifferences(Scanner.java:528)
at org.eclipse.jetty.util.Scanner.scan(Scanner.java:391)
at org.eclipse.jetty.util.Scanner.doStart(Scanner.java:313)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)
at org.eclipse.jetty.deploy.providers.ScanningAppProvider.doStart(ScanningAppProvider.java:145)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)
at org.eclipse.jetty.deploy.DeploymentManager.startAppProvider(DeploymentManager.java:557)
at org.eclipse.jetty.deploy.DeploymentManager.doStart(DeploymentManager.java:232)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:118)
at org.eclipse.jetty.server.Server.start(Server.java:342)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:100)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:60)
at org.eclipse.jetty.server.Server.doStart(Server.java:290)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)
at org.eclipse.jetty.xml.XmlConfiguration$1.run(XmlConfiguration.java:1250)
at java.security.AccessController.doPrivileged(Native Method)
at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1174)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.eclipse.jetty.start.Main.invokeMain(Main.java:509)
at org.eclipse.jetty.start.Main.start(Main.java:651)
at org.eclipse.jetty.start.Main.main(Main.java:99)
2018-02-22 14:24:42,812 [main] ERROR com.rsa.smc.sa.core.appliance.ApplianceMigrationService - Failed to migrate appliances, services and groups
com.rsa.smc.sa.core.util.carlos.ASGConnectionException: Failed to load hosts
at com.rsa.smc.sa.core.appliance.service.DefaultApplianceService$ApplianceCache.getAllAppliances(DefaultApplianceService.java:573)
at com.rsa.smc.sa.core.appliance.service.DefaultApplianceService.getAllAppliances(DefaultApplianceService.java:177)
at com.rsa.smc.sa.core.appliance.service.DefaultApplianceService.reloadAppliances(DefaultApplianceService.java:335)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:96)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:260)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:94)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at com.sun.proxy.$Proxy135.reloadAppliances(Unknown Source)
at com.rsa.smc.sa.core.appliance.ApplianceMigrationService.testSMSConnection(ApplianceMigrationService.java:171)
at com.rsa.smc.sa.core.appliance.ApplianceMigrationService.migrateApplianceInfo(ApplianceMigrationService.java:160)
at com.rsa.smc.sa.core.appliance.ApplianceMigrationService.onApplicationEvent(ApplianceMigrationService.java:123)
at com.rsa.smc.sa.core.appliance.ApplianceMigrationService.onApplicationEvent(ApplianceMigrationService.java:55)
at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:96)
at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:334)
at org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:950)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:482)
at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:410)
at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:306)
at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:112)
at com.netwitness.platform.server.common.context.SafeModeContextListener.contextInitialized(SafeModeContextListener.java:37)
at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:788)
at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:434)
at org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:780)
at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:284)
at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1322)
at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:732)
at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:490)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:118)
at org.eclipse.jetty.util.component.ContainerLifeCycle.addBean(ContainerLifeCycle.java:282)
at org.eclipse.jetty.util.component.ContainerLifeCycle.addBean(ContainerLifeCycle.java:214)
at org.eclipse.jetty.util.component.ContainerLifeCycle.updateBeans(ContainerLifeCycle.java:764)
at org.eclipse.jetty.server.handler.HandlerCollection.setHandlers(HandlerCollection.java:89)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.setHandlers(ContextHandlerCollection.java:145)
at org.eclipse.jetty.server.handler.HandlerCollection.addHandler(HandlerCollection.java:155)
at org.eclipse.jetty.deploy.bindings.StandardDeployer.processBinding(StandardDeployer.java:41)
at org.eclipse.jetty.deploy.AppLifeCycle.runBindings(AppLifeCycle.java:186)
at org.eclipse.jetty.deploy.DeploymentManager.requestAppGoal(DeploymentManager.java:495)
at org.eclipse.jetty.deploy.DeploymentManager.addApp(DeploymentManager.java:146)
at org.eclipse.jetty.deploy.providers.ScanningAppProvider.fileAdded(ScanningAppProvider.java:175)
at org.eclipse.jetty.deploy.providers.ScanningAppProvider$1.fileAdded(ScanningAppProvider.java:64)
at org.eclipse.jetty.util.Scanner.reportAddition(Scanner.java:605)
at org.eclipse.jetty.util.Scanner.reportDifferences(Scanner.java:528)
at org.eclipse.jetty.util.Scanner.scan(Scanner.java:391)
at org.eclipse.jetty.util.Scanner.doStart(Scanner.java:313)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)
at org.eclipse.jetty.deploy.providers.ScanningAppProvider.doStart(ScanningAppProvider.java:145)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)
at org.eclipse.jetty.deploy.DeploymentManager.startAppProvider(DeploymentManager.java:557)
at org.eclipse.jetty.deploy.DeploymentManager.doStart(DeploymentManager.java:232)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:118)
at org.eclipse.jetty.server.Server.start(Server.java:342)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:100)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:60)
at org.eclipse.jetty.server.Server.doStart(Server.java:290)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)
at org.eclipse.jetty.xml.XmlConfiguration$1.run(XmlConfiguration.java:1250)
at java.security.AccessController.doPrivileged(Native Method)
at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1174)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.eclipse.jetty.start.Main.invokeMain(Main.java:509)
at org.eclipse.jetty.start.Main.start(Main.java:651)
at org.eclipse.jetty.start.Main.main(Main.java:99)
2018-02-22 14:24:42,813 [main] ERROR com.rsa.smc.sa.core.appliance.ApplianceMigrationService - Migration failed.
2018-02-22 14:24:42,822 [main] ERROR org.springframework.web.context.ContextLoader - Context initialization failed
com.rsa.smc.sa.core.util.carlos.ASGConnectionException: Failed to load hosts
at com.rsa.smc.sa.core.appliance.service.DefaultApplianceService$ApplianceCache.getAllAppliances(DefaultApplianceService.java:573)
at com.rsa.smc.sa.core.appliance.service.DefaultApplianceService.getAllAppliances(DefaultApplianceService.java:177)
at com.rsa.smc.sa.core.appliance.service.DefaultApplianceService.reloadAppliances(DefaultApplianceService.java:335)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:96)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:260)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:94)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at com.sun.proxy.$Proxy135.reloadAppliances(Unknown Source)
at com.rsa.smc.sa.core.appliance.ApplianceMigrationService.testSMSConnection(ApplianceMigrationService.java:171)
at com.rsa.smc.sa.core.appliance.ApplianceMigrationService.migrateApplianceInfo(ApplianceMigrationService.java:160)
at com.rsa.smc.sa.core.appliance.ApplianceMigrationService.onApplicationEvent(ApplianceMigrationService.java:123)
at com.rsa.smc.sa.core.appliance.ApplianceMigrationService.onApplicationEvent(ApplianceMigrationService.java:55)
at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:96)
at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:334)
at org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:950)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:482)
at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:410)
at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:306)
at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:112)
at com.netwitness.platform.server.common.context.SafeModeContextListener.contextInitialized(SafeModeContextListener.java:37)
at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:788)
at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:434)
at org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:780)
at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:284)
at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1322)
at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:732)
at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:490)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:118)
at org.eclipse.jetty.util.component.ContainerLifeCycle.addBean(ContainerLifeCycle.java:282)
at org.eclipse.jetty.util.component.ContainerLifeCycle.addBean(ContainerLifeCycle.java:214)
at org.eclipse.jetty.util.component.ContainerLifeCycle.updateBeans(ContainerLifeCycle.java:764)
at org.eclipse.jetty.server.handler.HandlerCollection.setHandlers(HandlerCollection.java:89)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.setHandlers(ContextHandlerCollection.java:145)
at org.eclipse.jetty.server.handler.HandlerCollection.addHandler(HandlerCollection.java:155)
at org.eclipse.jetty.deploy.bindings.StandardDeployer.processBinding(StandardDeployer.java:41)
at org.eclipse.jetty.deploy.AppLifeCycle.runBindings(AppLifeCycle.java:186)
at org.eclipse.jetty.deploy.DeploymentManager.requestAppGoal(DeploymentManager.java:495)
at org.eclipse.jetty.deploy.DeploymentManager.addApp(DeploymentManager.java:146)
at org.eclipse.jetty.deploy.providers.ScanningAppProvider.fileAdded(ScanningAppProvider.java:175)
at org.eclipse.jetty.deploy.providers.ScanningAppProvider$1.fileAdded(ScanningAppProvider.java:64)
at org.eclipse.jetty.util.Scanner.reportAddition(Scanner.java:605)
at org.eclipse.jetty.util.Scanner.reportDifferences(Scanner.java:528)
at org.eclipse.jetty.util.Scanner.scan(Scanner.java:391)
at org.eclipse.jetty.util.Scanner.doStart(Scanner.java:313)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)
at org.eclipse.jetty.deploy.providers.ScanningAppProvider.doStart(ScanningAppProvider.java:145)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)
at org.eclipse.jetty.deploy.DeploymentManager.startAppProvider(DeploymentManager.java:557)
at org.eclipse.jetty.deploy.DeploymentManager.doStart(DeploymentManager.java:232)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:118)
at org.eclipse.jetty.server.Server.start(Server.java:342)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:100)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:60)
at org.eclipse.jetty.server.Server.doStart(Server.java:290)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)
at org.eclipse.jetty.xml.XmlConfiguration$1.run(XmlConfiguration.java:1250)
at java.security.AccessController.doPrivileged(Native Method)
at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1174)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.eclipse.jetty.start.Main.invokeMain(Main.java:509)
at org.eclipse.jetty.start.Main.start(Main.java:651)
at org.eclipse.jetty.start.Main.main(Main.java:99)
2018-02-22 14:24:42.824:INFO:/:main: Closing Spring root WebApplicationContext
2018-02-22 14:24:42,833 [main] INFO org.quartz.core.QuartzScheduler - Scheduler org.springframework.scheduling.quartz.SchedulerFactoryBean#0_$_NON_CLUSTERED paused.
2018-02-22 14:24:50,228 [main] INFO com.rsa.netwitness.carlos.jmx.DefaultConnectorServerFactoryBean - Stopping JMX connector server: javax.management.remote.rmi.RMIConnectorServer@28e4680
2018-02-22 14:25:05,210 [Puppet Service Daemon 1936322081] ERROR com.rsa.smc.sa.admin.service.DefaultPuppetService - Failed to discover new hosts
com.rsa.smc.sa.admin.exception.PuppetException: Failed to discover new hosts
at com.rsa.smc.sa.admin.service.DefaultPuppetService.refreshCertificateList(DefaultPuppetService.java:148)
at com.rsa.smc.sa.admin.service.DefaultPuppetService$2.run(DefaultPuppetService.java:240)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: com.rsa.smc.sa.admin.exception.MCOAgentException:
at com.rsa.smc.sa.admin.management.mcollective.AbstractMCOAgent.execute(AbstractMCOAgent.java:72)
at com.rsa.smc.sa.admin.management.mcollective.AbstractMCOAgent.executeAndReturnSingle(AbstractMCOAgent.java:89)
at com.rsa.smc.sa.admin.service.DefaultPuppetService.refreshCertificateList(DefaultPuppetService.java:132)
... 6 more
Caused by: org.apache.commons.exec.ExecuteException: Process exited with an error: 4 (Exit value: 4)
at org.apache.commons.exec.DefaultExecutor.executeInternal(DefaultExecutor.java:404)
at org.apache.commons.exec.DefaultExecutor.access$200(DefaultExecutor.java:48)
at org.apache.commons.exec.DefaultExecutor$1.run(DefaultExecutor.java:200)
... 1 more
2018-02-22 14:25:42,817 [System Monitoring Service ApplianceConfig 1562356232] ERROR com.rsa.smc.sa.admin.service.DefaultSystemMonitoringService - Failed to start system monitoring task
com.rsa.smc.sa.core.util.carlos.ASGConnectionException: Failed to load hosts
at com.rsa.smc.sa.core.appliance.service.DefaultApplianceService$ApplianceCache.getAllAppliances(DefaultApplianceService.java:573)
at com.rsa.smc.sa.core.appliance.service.DefaultApplianceService.getAllAppliances(DefaultApplianceService.java:177)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:96)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:260)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:94)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at com.sun.proxy.$Proxy135.getAllAppliances(Unknown Source)
at com.rsa.smc.sa.admin.service.DefaultSystemMonitoringService$4.run(DefaultSystemMonitoringService.java:267)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2018-02-22 14:26:24,097 [Provision Daemon 1684357947] ERROR com.rsa.smc.sa.admin.management.mcollective.MCOUpstartProcessEngine - com.rsa.smc.sa.admin.exception.MCOAgentException:
2018-02-22 14:26:24,098 [Provision Daemon 1684357947] ERROR com.rsa.smc.sa.admin.management.provision.DefaultApplianceProvision - Failed to add services on c8006394-3995-4e89-b53d-e57674a46625 with error
com.rsa.smc.sa.admin.exception.PuppetException: Failed to add services on c8006394-3995-4e89-b53d-e57674a46625 with error
at com.rsa.smc.sa.admin.management.provision.ApplianceAddServicesState.provision(ApplianceAddServicesState.java:94)
at com.rsa.smc.sa.admin.management.provision.DefaultApplianceProvision$3.run(DefaultApplianceProvision.java:478)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: com.rsa.smc.sa.admin.exception.ProcessEngineException: Failed to get available services for c8006394-3995-4e89-b53d-e57674a46625
at com.rsa.smc.sa.admin.management.mcollective.MCOUpstartProcessEngine.getAvailableProcesses(MCOUpstartProcessEngine.java:132)
at com.rsa.smc.sa.admin.service.DefaultProcessService.getAvailableServices(DefaultProcessService.java:173)
at com.rsa.smc.sa.admin.management.provision.ApplianceAddServicesState.getEndpointTypes(ApplianceAddServicesState.java:235)
at com.rsa.smc.sa.admin.management.provision.ApplianceAddServicesState.provision(ApplianceAddServicesState.java:80)
... 6 more
Caused by: com.rsa.smc.sa.admin.exception.MCOAgentException:
at com.rsa.smc.sa.admin.management.mcollective.AbstractMCOAgent.execute(AbstractMCOAgent.java:72)
at com.rsa.smc.sa.admin.management.mcollective.AbstractMCOAgent.executeAndReturnSingle(AbstractMCOAgent.java:89)
at com.rsa.smc.sa.admin.management.mcollective.MCOUpstartProcessEngine.getAvailableProcesses(MCOUpstartProcessEngine.java:122)
... 9 more
Caused by: org.apache.commons.exec.ExecuteException: Process exited with an error: 4 (Exit value: 4)
at org.apache.commons.exec.DefaultExecutor.executeInternal(DefaultExecutor.java:404)
at org.apache.commons.exec.DefaultExecutor.access$200(DefaultExecutor.java:48)
at org.apache.commons.exec.DefaultExecutor$1.run(DefaultExecutor.java:200)
... 1 more
Cause
jetty web server port 443 is not listening up due to rabbitmq(beam.smp) process port issue. (CLOSE_WAIT)
# service rabbitmq-server status
Status of node sa@localhost ...
Error: unable to connect to node sa@localhost: nodedown
DIAGNOSTICS
===========
attempted to contact: [sa@localhost]
sa@localhost:
* connected to epmd (port 4369) on localhost
* node sa@localhost up, 'rabbit' application running
current node details:
- node name: 'rabbitmq-cli-27475@XXXXXXSAHAED'
- home dir: /var/lib/rabbitmq
- cookie hash: AMaoZBL/zIUG+HwcbpJAQw==
# netstat -anop|grep beam
tcp 0 0 127.0.0.1:5672 0.0.0.0:* LISTEN 6290/beam.smp off (0.00/0/0)
tcp 0 0 0.0.0.0:25672 0.0.0.0:* LISTEN 6290/beam.smp off (0.00/0/0)
tcp 0 0 0.0.0.0:37876 0.0.0.0:* LISTEN 28606/beam.smp off (0.00/0/0)
tcp 107 0 xxx.xxx.xx.145:47908 xxx.xxx.xx.147:5671 CLOSE_WAIT 6290/beam.smp off (0.00/0/0)
tcp 107 0 xxx.xxx.xx.145:56314 xxx.xx.xx.102:5671 CLOSE_WAIT 6290/beam.smp off (0.00/0/0)
tcp 107 0 xxx.xxx.xx.145:58090 xxx.xxx.xx.138:5671 CLOSE_WAIT 6290/beam.smp off (0.00/0/0)
tcp 107 0 xxx.xxx.xx.145:44281 xxx.xxx.xx.144:5671 CLOSE_WAIT 6290/beam.smp off (0.00/0/0)
tcp 107 0 xxx.xxx.xx.145:33612 xxx.xxx.xx.36:5671 CLOSE_WAIT 6290/beam.smp off (0.00/0/0)
tcp 107 0 xxx.xxx.xx.145:57301 xxx.xxx.xx.143:5671 CLOSE_WAIT 6290/beam.smp off (0.00/0/0)
tcp 0 0 127.0.0.1:5672 127.0.0.1:56702 ESTABLISHED 6290/beam.smp off (0.00/0/0)
tcp 0 0 127.0.0.1:5672 127.0.0.1:38918 ESTABLISHED 6290/beam.smp off (0.00/0/0)
tcp 160 0 xxx.xxx.xx.145:51634 xxx.xxx.xx.135:5671 CLOSE_WAIT 6290/beam.smp off (0.00/0/0)
tcp 160 0 xxx.xxx.xx.145:57449 xxx.xxx.xx.136:5671 CLOSE_WAIT 6290/beam.smp off (0.00/0/0)
tcp 8 0 127.0.0.1:5672 127.0.0.1:56324 ESTABLISHED 6290/beam.smp off (0.00/0/0)
tcp 0 0 127.0.0.1:55293 127.0.0.1:4369 ESTABLISHED 28606/beam.smp off (0.00/0/0)
tcp 107 0 xxx.xxx.xx.145:50101 xxx.xxx.xx.133:5671 CLOSE_WAIT 6290/beam.smp off (0.00/0/0)
tcp 160 0 xxx.xxx.xx.145:39181 xxx.xxx.xx.33:5671 CLOSE_WAIT 6290/beam.smp off (0.00/0/0)
tcp 160 0 xxx.xxx.xx.145:56010 xxx.xxx.xx.146:5671 CLOSE_WAIT 6290/beam.smp off (0.00/0/0)
tcp 107 0 xxx.xxx.xx.145:58717 xxx.xxx.xx.146:5671 CLOSE_WAIT 6290/beam.smp off (0.00/0/0)
tcp 107 0 xxx.xxx.xx.145:47859 xx.x.xx.101:5671 CLOSE_WAIT 6290/beam.smp off (0.00/0/0)
tcp 107 0 xxx.xxx.xx.145:40090 xxx.xxx.xx.140:5671 CLOSE_WAIT 6290/beam.smp off (0.00/0/0)
tcp 107 0 xxx.xxx.xx.145:43488 xxx.xxx.xx.135:5671 CLOSE_WAIT 6290/beam.smp off (0.00/0/0)
tcp 160 0 xxx.xxx.xx.145:43597 xxx.xxx.xx.144:5671 CLOSE_WAIT 6290/beam.smp off (0.00/0/0)
tcp 8 0 127.0.0.1:5672 127.0.0.1:56630 ESTABLISHED 6290/beam.smp off (0.00/0/0)
tcp 107 0 xxx.xxx.xx.145:50561 xxx.xxx.xx.138:5671 CLOSE_WAIT 6290/beam.smp off (0.00/0/0)
tcp 107 0 xxx.xxx.xx.145:54692 xxx.xxx.xx.141:5671 CLOSE_WAIT 6290/beam.smp off (0.00/0/0)
tcp 160 0 xxx.xxx.xx.145:41234 xxx.xxx.xx.34:5671 CLOSE_WAIT 6290/beam.smp off (0.00/0/0)
.
.
Resolution
On the command line(SA Server)
To resolve rabbitmq port issue, follow the steps below.
Pre-steps: stop puppet agent
# /etc/init.d/puppet stop
1. stop jetty service
# stop jettysrv
2. stop rabbitmq service
# service rabbitmq-server stop
3. find beam.smp {PID} and kill
# netstat -anop|grep beam
# kill -9 {PID}
4. restart rsa-sms service
# service rsa-sms restart
5. start rabbitmq service
# service rabbitmq-server start
6. start jetty service
# start jettysrv
7. Start the puppet agent
# /etc/init.d/puppet start
After above steps, port 443 will be listening up and you are able to access Netwitness GUI.