Article Number
000032518
Applies To
RSA Product Set: RSA Security Analytics
RSA Version/Condition: 10.5.x
RSA Version/Condition: 10.5.x
Issue
Incidents page on the User Interface (UI) shows as blank.
Even though rsa-im service is running, the Administration -> Services Page shows it as down
Image description
Even though rsa-im service is running, the Administration -> Services Page shows it as down
Image descriptionCause
This can happen when the Incident Management (IM) database grows to an extend that it crashes the IM service.
The database size may be huge because of the number of alerts that are coming in are too high.
The database size may be huge because of the number of alerts that are coming in are too high.
Resolution
Stop the Incident Management service with the following command:
Check the current size of the IM databse with the following command:
Remove old alerts from database with the following command by changing the date accordingly:
Once removed, re-index the database to free up space:
Once the database is cleared, check the rabbitmq queue for im.alert_queue.
Access the RabbitMQ web interface following the instructions in the article entitled How to access the RabbitMQ web interface to monitor and troubleshoot the event broker on an RSA Security Analytics Log Collector.
In the Web Interface, click on the im.alert.queue.
Image description
Check the number of queues messages.
If the number is above 100,000 , the IM service may crash again after the restart.
To purge the queue, scroll down and click on the "Purge" button.
NOTE: Any purged messages cannot be retrieved
Once the queue is purged, start the IM service on the SA head using the following command:
# service rsa-im stop
Check the current size of the IM databse with the following command:
# mongo im -u im -p im > show collections system.indexes 3.48KB (uncompressed), 32.00KB (compressed) system.users NaNundefined (uncompressed), NaNundefined (compressed) categories 16.61KB (uncompressed), 32.00KB (compressed) aggregation_rule 14.05KB (uncompressed), 32.00KB (compressed) alert 1.17TB (uncompressed), 123.91GB (compressed) incident 509.28KB (uncompressed), 1.16MB (compressed) remediation_task 0.00B (uncompressed), 48.50KB (compressed) tracking_id_sequence 182.00B (uncompressed), 32.00KB (compressed) fs.files 0.00B (uncompressed), 48.50KB (compressed) fs.chunks 0.00B (uncompressed), 48.50KB (compressed)
Remove old alerts from database with the following command by changing the date accordingly:
> db.alert.remove({receivedTime : { $gte: ISODate('2016-01-01'), $lt: ISODate('2016-02-02')}})
Once removed, re-index the database to free up space:
> db.alert.reIndex()
Once the database is cleared, check the rabbitmq queue for im.alert_queue.
Access the RabbitMQ web interface following the instructions in the article entitled How to access the RabbitMQ web interface to monitor and troubleshoot the event broker on an RSA Security Analytics Log Collector.
In the Web Interface, click on the im.alert.queue.
Image descriptionCheck the number of queues messages.
If the number is above 100,000 , the IM service may crash again after the restart.
To purge the queue, scroll down and click on the "Purge" button.
NOTE: Any purged messages cannot be retrieved
Once the queue is purged, start the IM service on the SA head using the following command:
# service rsa-im start
Notes
The RabbitMQ web interface should be opened from the SA server.
In this article
