I have a host that needed to come down for maintenance (memory replace). I followed the procedures for cleanly shutting down the CVM and getting the esxi host offline, replaced the memory etc. I had some weirdness with VDS issues coming back up and had to play some games with the hypervisor and the CVM vm to get them reconnected to the network, but they are all pinging agian. However, when the cvm powers on, it does not reconnect to the nuta nix cluster.
Any thoughts or tips i can try before opening a ticket? perhaps some sort of cluster reconnect command i can try first? This is a newish cluster with no workload yet, so i can have the host out for a while, and removing the node entirely and rejoining is an option, but i'd like to try and figure a way to do this without taking that route.
@sandeepmp thanks much! I'll run through these and report back with what I find. for sure it was down for more than 30 minutes because i had a vmware license issue that slowed the cvm from coming back online. I'll start with that.
@sandeepmp It was the metadatastore issue, with an extra twist related to the licensing problem in a round about way.
See, this host is still on temp licensing since the cluster is still being stood up, but its been past 60 days. so, when the host required to be taken down to have the bad DIMM replaced, it would not let me power on the VM until i got the host license issue resolved. in resolving it, i had to remove the host from vcenter and readd it, which messed up all of the distributed switch settings. In the process of resolving that, the vmkernel port vmk-cvm-iscsi-pg got messed up. after resolving that problem, the enable metadatastore option came back in prism. Things are looking much better now!
EDIT: incase anyone follows this later some more detail: to resolve the vmkernel port issue i had to create a vmkernel port on the nutanix vswitch (which has no physical uplinks, its just for host to cvm talk) that had 192.168.5.1 as the ip (netmast 255.255.255.0, just accept the current gateway). once this was recreated, the host could speak to the cvm over their local link once more.