Host stuck on 'Rebooting'

[Anonymous]

I just updated to 11.3.2.0 from 11.3.1.1. One of my hosts is stuck on 'rebooting'.

It will time out after a while

When trying to reboot from the GUI nothing happens on the host and the status of 'rebooting' will just start over again.

[GyeonghwanHong]

Hi,

You can update it manually by mongo command

>db.appliance_update.update({ "_id" : "<Node_ID>" }, { $set: { status : "REBOOTED", acceptWarning : false}})

Please check this document.

000037371 - After upgrading to RSA NetWitness Logs and Network 11.x, the host's status is stuck at 'Rebooting' 

Regards,

Gyeonghwan.

[Anonymous]

Thanks, that's worked and removed the stuck error.

The current problem is that I can't reboot the host from the Admin UI. I click 'Reboot Host' and nothing happens.

[JoshRandall]

This might be due to rabbitMQ on that host not running (or not running properly).

On that host, you could try:

> systemctl status rabbitmq-server

…to check that its running.  And start (or restart, depending on its state) it with

> systemctl <start|restart> rabbitmq-server

If that fails, then a restart of the host from CLI might be in order.

[Anonymous]

Hi Josh,

rabbitmq-server is already started, restarting the service has no effect and rebooting the host from the cli also has no effect. The host still can't be restarted from the Admin UI host pane

[JoshRandall]

If its not a rabbitMQ problem, then I suspect the reboot commands are being blocked by previous unsuccessful commands.

We can check for these with another couple mongo commands on Node0.

(not sure if this one will show anything other than the record you updated with the mongo command above, but worth a try just in case)

>use sa

>db.appliance_update.find({"_id":"<Node_ID>"}).pretty()

(for this one, looking for records with anything other than "status”:COMPLETED”, specifically the most recent ones at the bottom of the mongo output)

>use orchestration-server

>db.task.find({"host._id":"<Node_ID>"}).pretty()

[Anonymous]

This was the result from the first command

{
"_id" : "047595fd-5b2d-4579-a622-70e52992baac",
"_class" : "com.rsa.smc.sa.admin.management.domain.bean.ApplianceUpdate",
"status" : "REBOOT_ERROR",
"acceptWarning" : false,
"message" : "Reboot failed.",
"updateMessageInfo" : {
"headlineMsg" : "Reboot failed.",
"message" : "Reboot task [5e0a7ded7a7fa311bc96569f] did not complete in allotted time. Check status of RabbitMQ and host LOG-HYB01(X.X.X.X).<br/>",
"updateMessages" : [ ]
},
"version" : {
"major" : 11,
"minor" : 3,
"servicePack" : 2,
"patch" : 0
}
}

The second command shows a lot of results, but the most recent result is the following.

{
"_id" : ObjectId("5e0a7ded7a7fa311bc96569f"),
"setup" : false,
"host" : {
"_id" : "047595fd-5b2d-4579-a622-70e52992baac",
"thirdParty" : false
},
"queued" : ISODate("2019-12-30T22:45:01.110Z"),
"start" : ISODate("2019-12-30T22:45:01.112Z"),
"end" : ISODate("2019-12-30T22:45:16.535Z"),
"status" : "COMPLETED",
"_class" : "com.rsa.asoc.orchestration.api.HostVerifyTask"
}

[JoshRandall]

Did the second command show any records with anything other than "status”:”COMPLETED” ?

I believe there should be a record in the second command's output with a field matching the failed reboot task, possibly (probably?) in the "_id" field....

"id_":ObjectId("5e0a7ded7a7fa311bc96569f)"

...and IIRC this record should have a "status":"<something other than COMPLETED here>".

[Anonymous]

At the start of the results there are the following results.

{
"_id" : ObjectId("5e014a2c7a7fa3116e198715"),
"version" : "11.3.2.0",
"updateTaskType" : "ConfigManagementUpdate",
"successful" : false,
"host" : {
"_id" : "047595fd-5b2d-4579-a622-70e52992baac",
"thirdParty" : false
},
"queued" : ISODate("2019-12-23T23:13:48.226Z"),
"start" : ISODate("2019-12-23T23:13:48.228Z"),
"end" : ISODate("2019-12-23T23:14:03.629Z"),
"status" : "FAILED",
"_class" : "com.rsa.asoc.orchestration.api.UpdateHostTask"
}

[JoshRandall]

Ok, we'll take note of that one for later.

What i'm expecting to find is a record matching your failed Reboot task.  Do you get any results for this command:

>use orchestration-server

>db.task.find({"_id":ObjectId("5e0a7ded7a7fa311bc96569f")}).pretty()