Does live migration traffic use the VM’s network when Backplane LAN is not configured?

Hi ​@jonekool 
VMs don't actually migrate themselves, the hypervisor handles all of that in the background. So your live migration traffic will never touch the guest VM networks

To answer your questions

1. No, the migration traffic won't use the VM's network. Even if your VM is on the 1G DMZ, that 1G link won't see any migration traffic. By default, all host-to-host traffic (which includes storage replication and live migrations) just goes over the default management network on br0. Since your AHV and CVM IPs are on br0 and you're using the 10G uplinks for that, the migration traffic is already moving over your 10G links.

2.Yes. Once you setup backplane network segmentation (like putting it on VLAN 10 on your 10G links), AHV automatically pushes all that host-to-host memory transfer traffic over to the dedicated backplane interface. The VM's actual network connection doesn't matter at all.

3.You can verify this in the Nutanix Security Guide under the Network Segmentation section, or in the AHV Live Migration Best Practices guide.

I strongly recommend segmenting your backplane for a production cluster so you can isolate the heavy storage and migration traffic from your standard management traffic. Just build out the backplane on your 10G links and don't worry about your 1G networks for this.
https://portal.nutanix.com/page/documents/solutions/details?targetId=BP-2029-AHV:nutanix-ahv-live-migration.html#:~:text=By%20default%2C%20live%20migration%20uses,bandwidth_mbps%3DX%20during%20each%20migration.

https://portal.nutanix.com/page/documents/details?targetId=Nutanix-Security-Guide-v7_3:wc-segmented-unsegmented-networks-difference-c.html#:~:text=The%20following%20figure%20Unsegmented%20Network,Unsegmented%20Network%2D%20AHV%20Cluster

Thanks 
Selvamani.S

Hi Selvami! 
 

Thank you for the detailed response and the documentation links, that clarifies a lot!
To give more context on why I was experiencing the bottleneck:
My br0 is not exclusively used for management traffic. I currently have production virtual machines running on br0 as well, all on the same 10G uplinks. On top of that, my VMware vSAN cluster shares the same 10G switch (we are currently getting out of VMware to go to nutanix). 
So when a Nutanix node went into maintenance mode, what I believe happened was:
    •    Live migration traffic hit br0
    •    That same br0 is carrying my production VM traffic
    •    vSAN replication traffic was also competing on the same 10G switch
    •    All of this converged on the same physical uplinks simultaneously
The result was a noticeable bottleneck, migrations were taking a very long time and both my vSAN environment and end users were impacted during the maintenance window.
So even though the migration wasn’t technically using the VM guest networks, it was still competing heavily with everything else sharing br0 and the same physical switch.
This confirms for me that configuring the backplane on a dedicated VLAN 10 on my 10G uplinks is the right move and at minimum it will isolate the migration and storage replication traffic from my production VM traffic on br0.
Does that sound like an accurate assessment of what was happening? And would the backplane segmentation be sufficient to resolve this? 

Hi ​@jonekool 
Your assessment is 100% spot on. 

Putting the backplane on VLAN 10 will not magically solve a physical bandwidth problem.

VLANs only give you logical separation. If you put your backplane on VLAN 10 but push it over the exact same physical 10G uplinks, it is still fighting for the exact same physical bandwidth as your production VMs and your vSAN traffic. Tagging the packets differently doesn't make the 10G any wider.

If your 10G links or physical switch buffers are getting saturated, you are still going to feel the pain during maintenance mode.

You should absolutely still configure the backplane segmentation because it isolates the broadcast domains and it is Nutanix best practice.

But to actually solve the bottleneck right now, you either need to configure QoS on your physical switch so migration traffic doesn't crush production, or just ride it out until you finish migrating off vSAN.

Once vSAN stops flooding your physical switch, your 10G links will have plenty of bandwidth.

Do the backplane segmentation now, but don't expect it to cure physical port congestion.

Hi ​@jonekool 
 

Your understanding is very accurate, and what you observed is exactly the kind of behavior that converged networking can produce during maintenance events.

+A dedicated Backplane VLAN will isolate the traffic logically.

+But it does not automatically isolate it physically if everything still rides the same 10G uplinks and same switch fabric.

So the improvement depends on how you implement it.

Thanks that all makes sense!

One quick switch question before I configure the backplane on VLAN 10, since this traffic is purely node to node and doesn’t need to go anywhere else on the network, do I just need to:
    1.    Create VLAN 10 on the switch
    2.    Allow it on the trunk ports going to each Nutanix node

No IP address on the VLAN, no gateway, no routing  just the VLAN tagged on those ports and Nutanix handles the rest?

Just want to confirm I’m not overcomplicating the switch side of this.

Yes create vlan10 and trunk it over the Nutanix data ports.
 

Thank you for your help! 
 

My pleasure !

Keep in mind that if you do segmentation over the same interfaces as where management, workload traffic etc is sending traffic you can still have the same problems as your initial question. 

(The result was a noticeable bottleneck, migrations were taking a very long time and both my vSAN environment and end users were impacted during the maintenance window)

To mitigate this you need to have additional interfaces in the nodes. Create a seperate virtuwal switch and attach those interfaces and make sure intra cluster traffic is following that path.