How-To: Change VLAN Tag for Nutanix Cluster (HV Host, CVM, IPMI)
I recently needed to change the VLAN tag (aka VLAN ID) used for cluster communication (HV host management, CVM, and IPMI) to a different tag. I did not find one document describing all needed configuration steps in one place, and the Nutanix documentation and knowledge base seems to lack a description of the IPMI change. This forum contains the information about how to change the IPMI VLAN ID, alas the respective threads do not mention HV host and/or CVM configuration. This post is supposed to close this gap. :-)
Changing the VLAN tag includes changing from "using no VLAN" (aka untagged) to "using a VLAN" (aka tagged).
I strongly suggest to change the VLAN tag of the cluster in two parts:
1. Change HV host and CVM VLAN ID
2. Change IPMI VLAN ID
Start the second part only after completing and testing the first part. The reason for this is, that you will (at least shortly) lose network connectivity to the HV host and CVM when changing the VLAN tag, thus it sould be done via IPMI access to the HV host console. After that part is completed, tested, and working, you use the HV host to change the VLAN ID of IPMI using ipmitool, thus (at least shortly) interrupting network connectivity to IPMI.
If possible, you should stop the cluster before changing the VLAN ID. Otherwise you need to proceed one node at a time and continue to the next node only after the VLAN tag change for the current node was successful. If you did not stop the cluster, ensure that the changed node has re-joined the cluster before continuing with the next node. If you did stop the cluster, remember starting it again after verifying a completely successful cluster VLAN tag change. ;-)
Now to the process of changing the VLAN tag:
0. Stop cluster, if applicable
1. Change HV host and CVM VLAN tag
1.a. Log on to IPMI web interface and start a remote console connection (I prefer to use the HTML5 one)
1.b.A. On an ESXi host, use 'esxcli network vswitch standard portgroup set -p "Management Network" -v VLAN_TAG' to change the VLAN tag of the host (i.e. the VMkernel interface vmk0). Then use 'esxcli network vswitch standard portgroup set -p "VM Network" -v VLAN_TAG' to change the CVM's VLAN tag.
1.b.B. On an AHV host, use 'ovs-vsctl set port br0 tag=VLAN_TAG' to change the host's VLAN ID. Then log on to the CVM via the internal network using 'ssh email@example.com' and change the CVM's VLAN ID using 'change_cvm_vlan VLAN_ID'
1.c. Now change the Switch configuration to use the correct VLAN ID and tag frames to the Nutanix nodes
1.d. Repeat steps 1.a. to 1.d. until all Nutanix nodes use the new VLAN tag for HV host and CVM
2. Change IPMI VLAN tag
2.a.A With a vSphere cluster, log on to HV host and use '/ipmitool lan set 1 vlan id VLAN_TAG' to change the IPMI VLAN ID
2.a.B. With an AHV cluster (or for storage only nodes), log on to the HV host and use 'ipmitool lan set 1 vlan id VLAN_TAG' to change the IPMI VLAN ID
2.b. Now change the switch to use the correct VLAN ID and tag the frames sent to the IPMI interface of the Nutanix node
2.c. Repeat steps 2.a. to 2.c. until all Nutanix nodes use the new VLAN tag for IPMI
Re: How-To: Change VLAN Tag for Nutanix Cluster (HV Host, CVM, IPMI)
both "Management Network" and "VM Network" are the portgroup names. These two are the Nutanix defaults for vSphere.
In step 1.b.A you use the specific command given in the post (at least with a standard install before changing the VMware networking setup). The commands change two portgroups on the vSphere host to use the given VLAN_TAG. The portgroup itself does not know about IP addresses, it creates connectivity on the Ethernet level.
The portgroup "Management Network" contains the vmk0 interface used for communication between the vSphere host and CVMs on other nodes in the cluster. It is the default management interface for the vSphere host that can (and probably should) be used for vCenter communication as well.
The portgroup "VM Network" contains the CVM interface used for external communication, i.e. with other CVMs and vSphere hosts in the cluster and the external world (management pc, updates servers, Prism Central, ...).
The actions in step 1.b.A affect a single vSphere host only. Thus you need to implement the changes on all the individual hosts in the cluster. By changing the VLAN tag you disrupt communication inside the cluster until the whole cluster is reconfigured. Therefore you need to stop the cluster (if it is running), and the use of the IPMI functionality to access the nodes in step 1.*.