In the Nutanix DRaaS Infrastructure, there is a tenant cluster and a production VPC for each customer. Generally, customers will have production VMs running on the On-prem cluster. Customers connect their On-prem environment with the Nutanix DRaaS VPC using an IPSec tunnel, which is used by the DRaaS workflow for replicating On-prem production data. During a Disaster Recovery (DR) situation or while running DR Tests, customers would want to failover their VMs from On-prem to a cluster in the DRaaS service.

When this happens, usually all VMs in one subnet of an On-prem network (e.g., 192.168.10.0/24) are failed over to the DRaaS service. If the customer chooses to preserve the IP, VMs in the DRaaS service come up with the same IPs as On-prem (e.g., 192.168.10.0/24 network). In such DR situations, customers can certainly choose only certain critical VMs to be replicated, but in those cases, on-prem VMs cannot communicate with VMs in the DR target.

WHY, you may wonder?

The reason behind this is that the DRaaS service advertises the subnet 192.168.10.0/24 over eBGP to On-prem while the On-prem VPN Gateway will already have that route learned locally. In the networking world, since local routes have a lesser admin distance, no User VMs from On-prem can communicate over to the DR target. This is a Layer 3 routing conflict. This problem can be solved by using the Layer 2 Stretch feature.

This blog will cover:

What is Layer 2 Stretch?

L2 Stretch enables VxLAN over IPSec on both the Nutanix DRaaS VPN gateway and on-prem VPN gateways to stretch an on-prem subnet to a Nutanix DRaaS DR target. VxLAN technology is a Layer 2 overlay scheme over a Layer 3 network. VxLAN uses MAC address-in-user datagram protocol (MAC-in-UDP) encapsulation to provide a means to extend Layer 2 segments across a Layer 3 segment. This basically means the Layer 2 packet gets a VxLAN header applied, then the frame gets encapsulated into a UDP IP packet and sent over a Layer 3 network.

When the feature is enabled, the VPN controller plugs NICs corresponding to the stretched subnet into both the Nutanix DRaaS VPN gateway and on-prem VPN gateway. As a part of the subnet stretching, VPN gateway configures a VxLAN interface, creates a Linux bridge, and adds the extended subnet interface and the VxLAN interface to the bridge.

NOTE: The Nutanix DRaaS VPN gateway uses VTI (Virtual Tunnel Interfaces) to establish eBGP over IPSec for on-prem cluster connectivity. Thus, this VTI interface serves as a transport link for VxLAN communication so that the VxLAN traffic will be encrypted and tunneled through the IPSec tunnel.

NOTE: Nutanix recommends using Layer 3 routing over Layer 2 stretch implementation. If preserving IP addresses for the UVMs is not a requirement, it is best practice to resort to Layer 3 routing. In that case, an On-prem network (e.g., 192.168.10.0/24) is failed over to the DR target with a different subnet (e.g., 172.16.30.0/24) and thus no routing conflict happens.

Pre-requisites

• The Layer 2 Stretch feature is available for customers since the release of PE AOS:5.19.x ,PC AOS:2021.3.x and all future versions.
• The feature is compatible with only VYOS gateway appliances and the minimum required version is 4.0.x and above.
• IPSec and eBGP must be established between Nutanix DRaaS and On-Prem Nutanix VPN Gateways.
• Need two static IPs from the subnets to be stretched. This will later be assigned to the additional ethernet interfaces of the Nutanix VPN gateways.
• General limitation: The VPN gateway and Layer 2 subnet extensions are only supported on ESXi standard vSwitch port groups. Distributed vSwitch is not supported.

+DRaaS+environment">Life of a VxLAN packet in a Nutanix DRaaS environment

As part of how a VxLAN packet traverses through a stretched network, this blog will cover a few VxLAN terminologies, the MTU and TCP-MSS settings, and an ARP resolution scenario when VM details are not learnt by the VPN gateway. Let’s jump right into the setup after the feature is enabled.

VxLAN Components and Significances

• Significance of the eth1 interface: With the eth1 interface being part of the Linux bridge of the VPN gateway, it’s required for relaying the ARP requests sourced from on-prem to Nutanix DRaaS or vice versa. Typically, once a VM, either on-prem or in Nutanix DRaaS, tries to reach a remote VM (remote will be Nutanix DRaaS if source is on-prem and vice-versa) , this interface picks up the broadcast frame (ARP Request) and relays it through the Linux Bridge br1, VxLAN1 interface, and over the IPSec tunnel.
• VxLAN Interface: The VxLAN interfaces handle the encapsulation and decapsulation, known as VxLAN tunnel endpoints (VTEPs). The VTEPs encapsulate a Layer 2 frame with a VxLAN header, a UDP segment (port number 4789), and then add an IP packet.
• VxLAN Network Identifiers (VNI): The VxLAN header is used to map Layer 2 VLAN numbers to VNI (VxLAN Network Identifier - Range: 4096 to 16,777,215). This is how Layer 2 to Layer 3 mapping is kept intact. For example, 192.168.10.0/24 may belong to VLAN 20 on-prem and VLAN 100 on Nutanix DRaaS; however, encapsulation and decapsulation of the VTEPs happen with a common VNI between the two VPN endpoints.

Assuming we have a subnet 10.41.18.0/23 in which the failed-over subnet will recover on Nutanix DRaaS, here is what happens:

1. When a ping request from the Nutanix DRaaS UVM (10.41.18.100) to On-prem UVM (10.41.18.25) is initiated, the UVM may not have an ARP entry in its cache. Hence, it broadcasts the ARP request that enters the Nutanix DRaaS VPN gateway through the eth1 interface (10.41.19.7).
   • Since the Nutanix DRaaS VPN Gateway has no information about the on-prem UVM, the packet is given to br1 and then to the VxLAN1 interface.
   • At this point, before the packet reaches the VTI interface, the original ethernet frame has been encapsulated with a VxLAN header, a UDP header, and an IP packet. The source and the destination IPs in this point will be the VTI interface IPs.
   • Once the packet reaches the VTI interface, the packets are encapsulated again with ESP header (Encryption) and an IP packet, with source and destination being the Nutanix DRaaS eth0 IP and On-prem eth0 IP.
2. The packet now enters the On-Prem VPN gateway through the IPSec tunnel, gets decrypted (ESP Header removed), and is given to the VxLAN1 interface. The VTEP (VxLAN1) interface decapsulates the UDP header, the VxLAN header, and hands it over to the br1. The br1 Linux Bridge of the On-Prem VPN gateway registers the Nutanix DRaaS UVMs MAC address (Source MAC address). From the br1, it is given to the eth1 interface and sent out to the On-Prem UVM.
3. When the On-Prem UVM receives this ARP request, it stores the Nutanix DRaaS UVM ARP entry in its cache, and the ARP resolution for Nutanix DRaaS UVM is complete. The On-prem UVM now sends the unicast ARP reply with its own source MAC address.
4. The On-Prem UVM now sends the ping reply packets, which reach the Nutanix DRaaS UVM in the same path as before.

The following picture depicts the path flow with possible encapsulation the packet takes with respect to the above points:

MTU and TCP-MSS

After all levels of encapsulation, the packets flowing through the IPsec tunnel between the NX Gateway appliances will generally look like this:

1. Ethernet Frame received from Nutanix DRaaS UVM at Nutanix DRaaS VPN GW’s eth1 will be 1320 bytes:
   • 1320 = 1266(Data) + 20 (TCP Header) + 20 (IPv4 Header) + 14(Ethernet Header)
2. VxLAN interface encapsulated the above frame with VxLAN header, UDP Header and IPv4 Header making the packet size to 1356 bytes:
   • 1356 = 1320 (Original Frame size from (1)) + 8 (VxLAN Header) + 8 (UDP Header) + 20 (IPv4 Header)
3. VTI interface encrypts the packet and adds IPSec overhead with ESP header, UDP for NAT-T and IPv4 Header making the size 1424 bytes:
   • 1424 = 1356 (Original VxLAN packet from (2)) + 40 (ESP Header) + 8 (UDP header for NAT-T) + 20 (IPv4 Header)

Thus, when the VM from either side sends a packet with a size of 1266 bytes, the packet after VxLAN and IPSec encapsulation would look like:

Scenarios to Overcome

Controller VM subnet extension across Clusters

This scenario assumes that Nutanix DRaaS has the same subnet already created. Layer 2 Stretch can be leveraged in a scenario where the on-prem CVM subnet needs to be extended to Nutanix DRaaS. However, to stretch the subnet in this case, a few routes need to be added.

• Log in to your Nutanix DRaaS portal and select Virtual Private Clouds.
• In the top corner, select the Production VPC > Select Routes tab and select Create Static Routes.

• Create a Static route to the PCVM in the on-prem cluster alone.
  • NOTE: This is a requirement to restore the AZ connectivity between the clusters as the subnets are in conflicting state without the L2 feature enabled.
• After the route creation, AZ connectivity will be restored and the Layer 2 extension can be configured for the desired subnet.

+DRaaS">Defining Gateways of Extended Subnet on Nutanix DRaaS

Nutanix recommends that all VMs on a particular site are configured to use the default gateway of the respective sites. For example, VMs running on Site 1 use the gateway of Site 1, and VMs running on Site 2 use Default gateway of Site 2. However, here are two possible choices to choose from:

Nutanix DRaaS subnet has the same on-prem gateway
In a scenario where the Nutanix DRaaS subnet created has the same on-prem subnet gateway, both Nutanix DRaaS UVM can reach the extended subnet on-prem and all other subnets on Nutanix DRaaS without any issue. However, if the Nutanix DRaaS UVM wants to reach a different subnet on-prem, this would require a static route to be added on Nutanix DRaaS to the destination subnet on-prem. Similarly, when an on-prem VM in the extended subnet range tries to reach Nutanix DRaaS UVM on a different subnet, a static route on the on-prem TOR needs to be added.

In case of a planned failover scenario, the failed-over VMs will honor the static route to reach the destination subnet as well. Reachability to other Nutanix DRaaS UVMs in other subnets will happen without any issue.

Nutanix DRaaS subnet has a different gateway compared to on-prem
In a scenario where the Nutanix DRaaS subnet created has a different gateway than on-prem, a Nutanix DRaaS UVM can reach the extended on-prem subnet and all other subnets on Nutanix DRaaS without any issues. If the Nutanix DRaaS UVM wants to reach a different subnet on-prem, a static route to the destination on-prem subnet would be required. Here, the static route added is honored only by the UVMs created on Nutanix DRaaS.

However, post-planned failover, a VM would carry the on-prem gateway information and would be able to reach all subnets on-prem. However, failed-over VMs cannot reach other subnets on Nutanix DRaaS as the gateway points to on-prem. For this VM to reach subnets on Nutanix DRaaS, the TOR switch on-prem should have a static route to the Nutanix DRaaS destination subnet with next-hop of on-prem VPN gateway’s VTI interface. The traffic path here will look like this:

Nutanix DRaaS VM -> VxLAN/IPSec -> on-prem TOR router -> Static route pointing to on-prem VPN vti -> Nutanix DRaaS VPN gateway -> VPC LR -> Nutanix DRaaS dest subnet

The above scenario would be ideal if a customer wants to maintain the traffic flow of failed-over VMs from within the on-prem environment. An example would be if a customer wants to control internet reachability of failed-over VMs from an on-prem firewall.

For configuring the Layer 2 stretch feature, refer to Layer 2 Virtual Subnet Extension (On-Prem and Nutanix® DRaaS Cloud Services).

2022 Nutanix, Inc.  All rights reserved. Nutanix, the Nutanix logo and all Nutanix product, feature and service names mentioned herein are registered trademarks or trademarks of Nutanix, Inc. in the United States and other countries. Other brand names mentioned herein are for identification purposes only and may be the trademarks of their respective holder(s). This post may contain links to external websites that are not part of Nutanix.com. Nutanix does not control these sites and disclaims all responsibility for the content or accuracy of any external site. Our decision to link to an external site should not be considered an endorsement of any content on such a site. Certain information contained in this post may relate to or be based on studies, publications, surveys and other data obtained from third-party sources and our own internal estimates and research. While we believe these third-party studies, publications, surveys and other data are reliable as of the date of this post, they have not independently verified, and we make no representation as to the adequacy, fairness, accuracy, or completeness of any information obtained from third-party sources.

This post may contain express and implied forward-looking statements, which are not historical facts and are instead based on our current expectations, estimates and beliefs. The accuracy of such statements involves risks and uncertainties and depends upon future events, including those that may be beyond our control, and actual results may differ materially and adversely from those anticipated or implied by such statements. Any forward-looking statements included herein speak only as of the date hereof and, except as required by law, we assume no obligation to update or otherwise revise any of such forward-looking statements to reflect subsequent events or circumstances.