We are running a Nutanix environment with 3 CVM nodes (A,B,C) on 3 separate ESXI hosts (7.0 Update 3) with 3 IPMI configurations using vCLS. Originally, Node A failed during an FSM upgrade and was unable to talk correctly to our Houston Datastore (MCQ-HOU). In doing so, there was no snapshot and we were unable to roll it back. 

We were currently in the midst of attempting to create a new node on the same ESXI host (.204) through Nutanix Prism but when searching for any existing or accessible hosts it does not find any of them and we are unable to create a new one. We attempted to shut down the cluster, unmount the Datastore on the broken node and re-mount it once we were able to clone a functioning node and change the configuration. In doing so, we are now unable to mount the datastore with the following commands and were presented with an error:

esxcli storage nfs remove -v 192.168.5.2:MCQ-Hou Unmount
esxcli storage nfs add -H MCQ-Hou -v MCQ-Hou -s 192.168.5.2 Mount
Please see the VMkernel log file for more details.: Unable to connect to NFS server

Each ESXI host can see the Datastore, can ping it and it appears as “Mounted” and “Accessible” (for the nodes it is currently on. Within the logs we see a variety of I/O Errors.

From each ESXI Web console that is still connected to the datastore, we can create new VM’s pulling ISO’s from the Datastore successfully. 

I was able to pull a PCAP capture from our LAN network and was able to see the requests coming through when trying to mount the datastore, and it shows it is reaching the datastore, being acknowledged and then rejected. I have already hopped on with VMWare support and have gone through all remediation attempts but the issue lies within the Datastore.

If anyone has any information that may help we would greatly appreciate it!