Nutanix Backplane network

All nodes (AHV/ESXi and AOS) must be in the same broadcast domain (vlan./subnet). No routing in between. 

The 192.168.5.x subnet is between CVM and HOST (internal traffic) and when a CVM goes down (LCM or failure) then the HOST will use that 192.168.5.x network to get to the data via another CVM. 

Backplane network will manage you to specify specific interfaces or vlans to separate intra-cluster-traffic from (for example) management traffic. That also is a non routed subnet. 

Hey Jeroen! 

Right so the 192.68.5.x is for the CVM and Host to communicate locally and for the CVM’s to replicated as well correct and if there is like you said maintenance or issue via LCM or Failure the other CVM’s recognize this issue that they can no longer communicate to that CVM and move things like Prism and replicate changes over to other CVM’s (correct depending on the RF of course)? But if the external switch is down and lets say the HBA / Host using that switch went offline including the CVM the other CVM’s can no longer communicate to that CVM correct? There isn't an offline backplane network that they can still communicate to correct? they still need to talk over the external switch right? 

Super helpful answer, appreciate the reply!! 

Thanks 

So Nutanix nodes should ALWAYS connect to two (2) switches. And those uplinks should be connected via active-backup of LACP. In case of an switch outage then the other switch/interface takes over. The preferred network topology for Nutanix is leaf/spine. 

Right, totally agree, I split the Dual Interfaces between multiple switches for redundancy. I used Foundation for initial deployment and I thought it was either not supported or preferred not to do LAG’s / LACP and just have untagged uplinks for the Hypervisor (ESXi) + the CVM’s. Would I use another deployment method if i want to use LACP? 

Thanks  

Thanks so much man this is awesome! 

With enabling backplane traffic (seg) which puts the cluster traffic on the segmented network is this something that can be configured while running ESXI or only an AHV configuration. Could you send over any documentation or articles on this, would like to read up more and dive into this. 

Thanks!!!! 

Oeff are you using ESXi? Then good luck ;) You can only configure LACP in vCenter, so the cluster needs to be up and running before you can connect it to vCenter. (Little chicken and egg story). I would recommend to stick with the active-backup best practices, dont make your life to hard ;) 

If you have multiple dual interfaces (so 4 ports in total) I would take 1 port from each interface for the normal workloads (virtual machine traffic) and 1 port from each interface for the cluster traffic (and use backplane traffic on this as well). In that case a physical interface in the node can go kaput ;) and all will keep on running. If you only have 1 multiple port interface, then you have a spof. 

It is al described here ;) https://www.jeroentielen.nl/nutanix-on-esx-howto-setup-the-network-stack/

Awesome!

Now do you suggest keeping the vmkernels separated from the vm vdportgroups physically if you have two HBA’s per host w/ 2x ports to reduce any crosstalk or congestion / performance impact? For instance keeping a VSS with 2x vmnics / uplinks (with mgt and vmotion service enabled vmk’s) and VDS with 2x vmnics / uplinks for all vm vdportgroups ? This way the traffic for vmks and vms are physically separated, to avoid things like a vmotion burst not impacting vm performance if it tries to take over the entire 10GbE or 25GbE interfaces. Ultimately not migrating the vmks to the VDS to allow for better performance, or would you say it depends? 

Thanks  for the great conversation !! 

Yes, that is what I always do (if there are enough interfaces in the node). So yes please. And it works the other ways around as well, so a user cannot use all bandwidth what the cluster is needed. 

​@JeroenTielen  - Thanks so much for all the technical guidance, much appreciated!! 

Love this kind of discussions ;)