Solved

Restoring the RF after a node failure

  • 15 September 2016
  • 3 replies
  • 3510 views

Badge +7
In a Nutanix cluster a node failure is the worst event and implies a restart of the VM on the surviving nodes. The data is not lost since it is replicated but if we have the standard RF2 is exposed to be lost in case of a further failure. So my questions are:

- What is the timing for the Nutanix cluster to start restoring the RF2 using space available on the surviving nodes?

- What happens if after a certain amount of time the lost node comes back on line?

I think this is a relevant question since the time to restore the RF after the failure of a node with 4 x 6 TB disk would not be short.

Of course the problem would be easily solved moving to RF3 since in this case the probability of 2 failures would be much lower. Unfortunately many customer are starting with small clusters of 3 or 4 node where you can not implement RF3 ....

Stefano
icon

Best answer by DonnieBrasco 15 September 2016, 11:47

Yes, we will discard the 3rd copy which is created as part of data rebuilding process.

Also, currently we start rebuilding as soon as node goes down which doesn't differentiates between node down dur to failure or node down for some planned maintenance. But work is in progress to identify situation where node is brought down for maintenance and then we should not start rebuilding data as it is a prority and intense activity for the cluster.

-NP
View original

3 replies

Userlevel 4
Badge +18
Hi Stefano,

- What is the timing for the Nutanix cluster to start restoring the RF2 using space available on the surviving nodes?

> Rebuilding process starts as soon as the node goes down but time to rebuild depends upon the amount of data that was present on the failed node. Prism will continue to show the resiliency status as critical unless entire data is rebuild.

- What happens if after a certain amount of time the lost node comes back on line?

> In case the node comes back online, curator service (which is responsible for rebuilding data) will continue to rebuild the data, also at the same time it will make sure that node data integrity is fine and it will run scans to make sure there are no under replicated data in the cluster, once it is done, curator will cancel all the tasks pertaining to rebuilding of the data due to a node failure.

Hope it helps.

-NP
Badge +7
So the Software correctly takes the "worst case" approach and starts rebuilding the parity for all the data that had a copy on the lost node.

In case the node comes back I assume tha the curator will delete the "3rd copies" already done in order to avoid waste of space... Is it correct?
Userlevel 4
Badge +18
Yes, we will discard the 3rd copy which is created as part of data rebuilding process.

Also, currently we start rebuilding as soon as node goes down which doesn't differentiates between node down dur to failure or node down for some planned maintenance. But work is in progress to identify situation where node is brought down for maintenance and then we should not start rebuilding data as it is a prority and intense activity for the cluster.

-NP

Reply