NetWitness platform Source-server Service Crashes Following Upgrade to Version 12.4.0.0

[#1]

Article Number

000003221

Applies To

NetWitness Product Set: NetWitness Platform
NetWitness Product/Service Type: Source Server, Log decoder
NetWitness Version/Condition: 12.4 or later
Platform: CentOS/ Alma Linux

Issue

The Source-server service may crash after upgrading to RSA NetWitness Platform version 12.4.0.0

The following error logs are observed in the /var/log/netwitness/source-server/source-server.log :
 

2024-04-29 11:32:23,361 [                          main] ERROR              Upgrade|Failed to Update contentName state should be: writes is not an empty list
2024-04-29 11:32:23,768 [                          main] INFO               Upgrade|Start of separating custom xml parser from existing log devices in content library
2024-04-29 11:32:23,823 [                          main] ERROR o.s.b.SpringApplication|Application run failed
java.lang.reflect.UndeclaredThrowableException: Failed to invoke event listener method
HandlerMethod details: 
Bean [com.rsa.asoc.launch.engine.lifecycle.UpgradeComponentVisitor]
Method [public void com.rsa.asoc.launch.engine.lifecycle.UpgradeComponentVisitor.onApplicationEvent(org.springframework.boot.context.event.ApplicationReadyEvent) throws com.rsa.asoc.launch.engine.lifecycle.UpgradeFailedException]
Resolved arguments: 
[0] [type=org.springframework.boot.context.event.ApplicationReadyEvent] [value=org.springframework.boot.context.event.ApplicationReadyEvent[source=org.springframework.boot.SpringApplication@7db791df]]

	at org.springframework.context.event.ApplicationListenerMethodAdapter.doInvoke(ApplicationListenerMethodAdapter.java:365)
	at org.springframework.context.event.ApplicationListenerMethodAdapter.processEvent(ApplicationListenerMethodAdapter.java:233)
	at org.springframework.context.event.ApplicationListenerMethodAdapter.onApplicationEvent(ApplicationListenerMethodAdapter.java:165)
	at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:174)
	at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:167)
	at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:145)
	at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:445)
	at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:378)
	at org.springframework.boot.context.event.EventPublishingRunListener.ready(EventPublishingRunListener.java:109)
	at org.springframework.boot.SpringApplicationRunListeners.lambda$ready$6(SpringApplicationRunListeners.java:80)
	at java.base/java.lang.Iterable.forEach(Iterable.java:75)
	at org.springframework.boot.SpringApplicationRunListeners.doWithListeners(SpringApplicationRunListeners.java:118)
	at org.springframework.boot.SpringApplicationRunListeners.doWithListeners(SpringApplicationRunListeners.java:112)
	at org.springframework.boot.SpringApplicationRunListeners.ready(SpringApplicationRunListeners.java:80)
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:343)
	at com.rsa.asoc.launch.engine.Launch.<init>(Launch.java:33)
	at com.rsa.asoc.launch.engine.Launch.main(Launch.java:43)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49)
	at org.springframework.boot.loader.Launcher.launch(Launcher.java:108)
	at org.springframework.boot.loader.Launcher.launch(Launcher.java:58)
	at org.springframework.boot.loader.PropertiesLauncher.main(PropertiesLauncher.java:467)
Caused by: com.rsa.asoc.launch.engine.lifecycle.UpgradeFailedException: java.lang.NullPointerException
	at com.rsa.asoc.launch.engine.lifecycle.UpgradeComponentVisitor.performUpgrade(UpgradeComponentVisitor.java:100)
	at com.rsa.asoc.launch.engine.lifecycle.UpgradeComponentVisitor.onApplicationEvent(UpgradeComponentVisitor.java:59)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at org.springframework.context.event.ApplicationListenerMethodAdapter.doInvoke(ApplicationListenerMethodAdapter.java:348)
	... 24 common frames omitted
Caused by: java.lang.NullPointerException: null
	at java.base/java.util.Objects.requireNonNull(Objects.java:209)
	at com.rsa.netwitness.usm.server.central.utils.ContentUtil.extractBaseParserFile(ContentUtil.java:1323)
	at com.rsa.netwitness.usm.server.upgrade.central.CentralMgmtUpgradeComponent.extractCustomLogDeviceContent(CentralMgmtUpgradeComponent.java:360)
	at com.rsa.netwitness.usm.server.upgrade.central.CentralMgmtUpgradeComponent.performUpgrade(CentralMgmtUpgradeComponent.java:214)
	at com.rsa.asoc.launch.engine.lifecycle.UpgradeComponentVisitor.lambda$performUpgrade$0(UpgradeComponentVisitor.java:94)
	at java.base/java.util.Optional.ifPresent(Optional.java:178)
	at com.rsa.asoc.launch.engine.lifecycle.UpgradeComponentVisitor.performUpgrade(UpgradeComponentVisitor.java:87)
	... 30 common frames omitted

PreUpgrade check tool can indicate below error for this issue:

Image description
 

Cause

If custom Log Devices uploaded using the CCM-UI page in a previous version. This issue occurs because the upgrade process expects specific file formats (.xml and -custom.xml files) for these custom Log Devices, and any deviations can lead to crashes.

Resolution

The users have to recreate the .envision file with the proper structure:
 envision File Structure:

/etc/netwitness/ng/envision/etc/devices/<LogDevice Name>/<LogDevice>.xml

For example:

/etc/netwitness/ng/envision/etc/devices/accurev/accurevmsg.xml

Please refer below document for additional details: Import Content to Content Library  

Screenshot when upgrade is successful:

Image description