I recently needed to change the VLAN tag (aka VLAN ID) used for cluster communication (HV host management, CVM, and IPMI) to a different tag. I did not find one document describing all needed configuration steps in one place, and the Nutanix documentation and knowledge base seems to lack a description of the IPMI change. This forum contains the information about how to change the IPMI VLAN ID, alas the respective threads do not mention HV host and/or CVM configuration. This post is supposed to close this gap.
Changing the VLAN tag includes changing from "using no VLAN" (aka untagged) to "using a VLAN" (aka tagged).
I strongly suggest to change the VLAN tag of the cluster in two parts:
1. Change HV host and CVM VLAN ID
2. Change IPMI VLAN ID
Start the second part only after completing and testing the first part. The reason for this is, that you will (at least shortly) lose network connectivity to the HV host and CVM when changing the VLAN tag, thus it sould be done via IPMI access to the HV host console. After that part is completed, tested, and working, you use the HV host to change the VLAN ID of IPMI using ipmitool, thus (at least shortly) interrupting network connectivity to IPMI.
If possible, you should stop the cluster before changing the VLAN ID. Otherwise you need to proceed one node at a time and continue to the next node only after the VLAN tag change for the current node was successful. If you did not stop the cluster, ensure that the changed node has re-joined the cluster before continuing with the next node. If you did stop the cluster, remember starting it again after verifying a completely successful cluster VLAN tag change.
Now to the process of changing the VLAN tag:
0. Stop cluster, if applicable
1. Change HV host and CVM VLAN tag
1.a. Log on to IPMI web interface and start a remote console connection (I prefer to use the HTML5 one)
1.b.A. On an ESXi host, use 'esxcli network vswitch standard portgroup set -p "Management Network" -v VLAN_TAG' to change the VLAN tag of the host (i.e. the VMkernel interface vmk0). Then use 'esxcli network vswitch standard portgroup set -p "VM Network" -v VLAN_TAG' to change the CVM's VLAN tag.
1.b.B. On an AHV host, use 'ovs-vsctl set port br0 tag=VLAN_TAG' to change the host's VLAN ID. Then log on to the CVM via the internal network using 'ssh email@example.com' and change the CVM's VLAN ID using 'change_cvm_vlan VLAN_ID'
1.c. Now change the Switch configuration to use the correct VLAN ID and tag frames to the Nutanix nodes
1.d. Repeat steps 1.a. to 1.d. until all Nutanix nodes use the new VLAN tag for HV host and CVM
2. Change IPMI VLAN tag
2.a.A With a vSphere cluster, log on to HV host and use '/ipmitool lan set 1 vlan id VLAN_TAG' to change the IPMI VLAN ID
2.a.B. With an AHV cluster (or for storage only nodes), log on to the HV host and use 'ipmitool lan set 1 vlan id VLAN_TAG' to change the IPMI VLAN ID
2.b. Now change the switch to use the correct VLAN ID and tag the frames sent to the IPMI interface of the Nutanix node
2.c. Repeat steps 2.a. to 2.c. until all Nutanix nodes use the new VLAN tag for IPMI
3. Start cluster, if applicable