re-open of topic "Stretch 3-Node Cluster Across 3 Locations" | Nutanix Community
Skip to main content

Due to recurring discussions with customers, here again the question about the technical background. why can't I split Nutanix nodes in a cluster across 3 locations (these are less meters apart)? ESXi can do that, other virtualization solutions too (MS HV, PVE).

I'm interested in a technical explanation, that it's not supported, I know, but why?

In fact can we build stretched Clusters across Locations (No geolocations)?
 

Thanks in advance

 

 

Nutanix is a HCI environment. Meaning all nodes together are the compute and storage power. All writes will go locally on the node (where the vm is running) and the copies are stored on the other nodes. When the nodes are place across different zones the write will take longer having a terrible cluster experience. So this is an absolute nogo. all nodes need to be in the same vlan/subnet in the same location. If you want to use three locations you can create a cluster in each location and do data protection between them. 

Even synchronyous replication is not working when latency is more then 5ms. 


How far are we talking between the nodes from a LATENCY perspective?    You mention “meters” apart.  Are we talking 5 meters, 10 meters, 100 meters?   


Hi Thomas,

When such scenarios are discussed the most important point to discuss is Fault domains, every marketing material says yes we can do that, but ask them what will happen when two sites or links between then are down, if ttl is less than 2ms between three sites/location technically Nutanix can also do that, but consider failure scenarios, how the split brain is going to be managed. is such configuration worth. Ask customer to do POC with these vendors and update here, we will be interested to know.

Don’t bother to test Nutanix in these scenarios as its not supported.

 

F>P


Thanks for all answers,

So the reason is that due to network latencies there is a risk that the copies of the data blocks could not be written to the other nodes quickly enough? It's always complicated to explain the Nutanix solution to people coming from VMware, even though the two infrastructures are not comparable...


I’m still trying to understand how much distance are we talking about here and the physical infrastructure between those locations and what VMware solutions you are referencing.  

If you’re referencing a non HCI based solution it really isn’t comparable, you can do some pretty wacky things with node location as long as they can get a FC or iSCSI connection back to their storage.  

With HCI you have to think of having the storage array inside the cluster.  You wouldn’t put half of your storage array in one room and the other half in another, you’d run into issues if the link between them was severed AND you’d have lower performance due to the distance.  You’d have two smaller arrays that replicate between them so your writes are fast and local, and then the data is available at the second location when needed.   This is the way it’s done with VSAN (stretch cluster where PFTT is local and SFTT is the remote) and how Nutanix would build such a design.   
 

Also, what are you protecting from with 3 nodes in 3 locations?   The loss of an individual data closet?   


Reply