Node Stuck in Removal Process | Nutanix Community
Skip to main content

Long story short we are swapping over to newer Nutanix chassis and replacing a few nodes. The previous tech had issues imaging the new chassis. However, in the prism hardware tab, the chassis is present (i think the previous tech was able to get it added somehow despite previous issues). I was able to go through foundation and get the new chassis nodes imaged and ready to be added back to the cluster, but the information remains in Prism and we cannot remove it. The new nodes show up when we select the expand cluster option, but the information (host ips, etc.) matches what is already in Prism, so it cannot be added. We are using a 5.x AOS and cannot upgrade.

We need a way to forcibly remove the stale information regarding the nodes that is still present in the cluster. 

So far we have tried the following with support and are hoping someone has possibly seen something like this before. 

We tried removing the nodes through Prism, but the 1st node we tried to remove is showing the meta data drives greyed out and it shows host being removed. It has been this way for 2 weeks now. 

When issuing the “ncli host get-rm-status” command, I still see the node as stuck in “MARKED FOR REMOVAL BUT NOT DETACHABLE”

However, the task list output does not show a task labeled “kEnteringMaintenanceMode” as described in Article #000014161 and the problem host in the host list reflects “maintenance mode = false.”

When entering the ncli host edit id=<host.id> enable-maintenance-mode=true command, we receive the following: “Error node <node id> removal in progress. Cannot set node id <node id> in maintenance mode.”

When we tried to issue the “ncli host delete <host id> force=true” command it says the process of node removal is started, but nothing is reflected in the task list and Prism does not show any progress and the node remains as “MARKED FOR REMOVAL BUT NOT DETACHABLE.”

Please assist if you happen to have any experience with this. 

When you have new nodes with ip-addresses of nodes already in the cluster you will get errors and duplicated ip-addresses. Dont foundation new nodes with ip-addresses already in the cluster 

 

In your case, involve support to help you out. 

Unfortunately, they have to have the ip addresses of the ones whos data is stale in the cluster. We don’t have another option. 

Unfortunately, they have to have the ip addresses of the ones whos data is stale in the cluster. We don’t have another option. 

You can do that. But shut them down and remove to old one first. 

IF the removal got stuck involve support. There is a force switch on the removal command ;) 

Yes the original nodes were removed properly, then the previous person working this somehow added the new nodes in question to the cluster (not sure how as they had some issues on these hosts) so we had to re-image these new nodes again. Before imaging we tried to remove them through the gui thats when the nodes got “stuck” so after some time trying to get them out of the cluster we resorted to the “ncli host delete <host id> force=true” command and it returns that it is initiated, but nothing actually happens and the node remains “MARKED FOR REMOVAL BUT NOT DETACHABLE” when issuing the ncli host get-rm-status command and it also shows this in the ncli host list information for the node. 

Ahhh so the nodes you want to remove are not there anymore? (if so, use the force switch with the cli remove command)

They are there but with new images.

Yes so the nodes to be removed are not (technically) member of the cluster? If so, use the force switch in de remove command from cli.  But best is to create a support case, they will help you and you will be online before eod ;) 

Correct the nodes that need to be removed are not technically members. As stated,

when we issue the “ncli host delete <host id> force=true” command it says the process of node removal is started, but nothing is reflected in the task list and Prism does not show any progress and the node remains as “MARKED FOR REMOVAL BUT NOT DETACHABLE.”

I have a support case and we have been working with them for days with no resolution, hence my reaching to the community to broaden our search for a possible solution. 

Terms & Conditions