Reporting Engine restarts After upgrade to RSA NetWitness Platform 11.4

[#1]

Article Number

000001934

Applies To

RSA Product Set: NetWitness Platform
RSA Product/Service Type: Reporting Engine
RSA Version/Condition: 11.4

Issue

In some cases, after you upgrade to RSA NetWitness 11.4 from earlier versions, such as 11.2 or 11.3, the Reporting Engine service attempts to restart continuously without success.
 

Cause

The database files for live charts, alert status, or report status may not be loaded successfully as the files may be corrupted.
 

Resolution

To resolve the issue, do the following:

1. Check which database files are corrupted by checking the Reporting Engine log (/var/netwitness/re-server/rsa/soc/reporting-engine/logs/reporting-engine.log) and looking for the log blocks: 

• If the live charts db file is corrupted, the following logs are displayed.

Attempt Failed!!! Clearing pending acquires. While trying to acquire a needed new resource, we failed to succeed 
more than the maximum number of allowed acquisition attempts (30). Last acquisition attempt exception:

org.h2.jdbc.JdbcSQLException: File corrupted while reading record: null. Possible solution: use the recovery tool
[90030-196]

     at org.h2.message.DbException.getJdbcSQLException(DbException.java:345)

     at org.h2.message.DbException.get(DbException.jva:168)

org.springframework.beans.factory.BeanCreationException: Error creating bean with name
'chartSummaryDAOImpl': Invocation of init method failed; nested exception is
com.rsa.soc.re.exception. ReportingException: java.sql.SQLException: Connections could not be acquired from the
underlying database!

• If the alert status db file is corrupted, the following logs are displayed:

Attempt Failed!!! Clearing pending acquires. While trying to acquire a needed new resource, we failed to succeed 
more than the maximum number of allowed acquisition attempts (30). Last acquisition attempt exception:

org.h2.jdbc.JdbcSQLException: File corrupted while reading record: null. Possible solution: use the recovery tool
[90030-196]

     at org.h2.message.DbException.getJdbcSQLException(DbException.java:345)

     at org.h2.message.DbException.get(DbException.jva:168)

org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name
'alertStatusHandler': Unsatisfied dependency expressed through field 'alertExecutionStatsDAO'; nested exception
is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name
'alertExecutionStatsDAOImpl': Unsatisfied dependency expressed through field
'presustedAlertExecutionStatsDAO'; nested exception is
org.springframework.beans.factory.BeanCreationException: Error creating bean with name
'presistedAlertExecutionStatsDAOImpl'

• If the report status db file is corrupted, the following logs are displayed:

org.h2.jdbc.JdbcSQLException: File corrupted while reading record: null.
Possible solution: use the recovery tool [90030-196]

2. To resolve the live charts database file corruption, perform the following steps:

Warning: Some live charts data may be lost after performing these steps.

1. Stop the Reporting Engine service.
2. Move the livechart.mv.db file from /var/netwitness/re-server/rsa/soc/reporting-engine/livecharts folder to a temporary location.
3. Restart the Reporting Engine service.

3. To resolve the alert status or report status database file corruption, perform the following steps:

1. Stop the Reporting Engine service.
2. Replace the corrupted db file with the latest alertstatusmanager.mv.db or reportstatusmanager.mv.db from the /var/netwitness/re-server/rsa/soc/reporting-engine/archives folder.
3. Restart the Reporting Engine service.

Notes

This specific issue can be found on page 47 in Appendix C: Troubleshooting Version Installations and Upgrades of the Upgrade Guide for RSA NetWitness® Platform 11.4 under Reporting Engine Restarts After Upgrade.