Hi Everyone.
I have a 4-node Nutanix NX demo/POC cluster running AHV. After a power outage at the office, the cluster did not recover properly.
Initial situation:
- 4-node Nutanix NX cluster
- Hypervisor: AHV
- This is a POC environment, so there is no important data to recover
- After the outage, only the CVM on Node A was active
- Nodes B, C and D only booted into AHV
- On Nodes B, C and D,
virsh list --alldid not show any CVM
What I already tried:
- I generated a Phoenix ISO from the working CVM using the AHV ISO and Nutanix/AOS installer package
- The generated Phoenix ISO is around 6 GB
- I used that same Phoenix ISO successfully on Node B
- On Node B, I selected “Wipe data and install CVM + hypervisor” and the node was reinstalled successfully
- However, Node C and Node D fail during the same Phoenix reinstall process
Phoenix error on Node C:
tar: /mnt/local/images/svm/nutanix_installer_package-release-euphrates-5.20.0.1-stable-f5e54bf50b9b92d1e27560e422629e322968a3e0-x86_64.tar: Cannot read: Input/output error
tar: Too many errors, quitting
tar: Error is not recoverable: exiting now
FATAL An exception was raised:
StandardError: Failed command: [tar xvf /mnt/local/images/svm/nutanix_installer_package-release-euphrates-5.20.0.1-stable-f5e54bf50b9b92d1e27560e422629e322968a3e0-x86_64.tar -C /mnt/svm_installer --totals] with reason [None]Additional Foundation issue:
I also tried Nutanix Portable Foundation, but it fails connecting to the IPMI interfaces.
IPMI addresses:
- Node A: 172.16.70.213
- Node B: 172.16.70.214
- Node C: 172.16.70.215
- Node D: 172.16.70.216
Foundation error:
Failed to connect to the following IPMIs:
172.16.70.213, 172.16.70.214, 172.16.70.215, 172.16.70.216Network/IPMI tests:
- Ping to all IPMI addresses works
- I can access the IPMI web interface using HTTP/port 80
- TCP 443 works only on 172.16.70.213
- TCP 443 fails on 172.16.70.214, 172.16.70.215 and 172.16.70.216
- I tested Portable Foundation from a Windows Server
- I also tested Portable Foundation from a laptop connected to the same switch and same VLAN as the IPMI ports, but the same error occurs
USB boot issue:
- I created a bootable USB using BalenaEtcher with the Phoenix ISO
- I connected it to the rear USB ports of the Nutanix node
- In the boot menu, the USB device did not appear as expected
- I only saw a SanDisk option, but selecting it booted the existing AHV installation
- I tried multiple rear USB ports with the same result
Current goal:
Since this is only a POC cluster, I do not need to recover the old data. I just need to reinstall/rebuild Nutanix from scratch, even if the cluster is empty.
Questions:
- Why would Phoenix fail with “Cannot read: Input/output error” on Node C/D while extracting the AOS/SVM installer package, if the same ISO worked successfully on Node B?
- Could this indicate a local disk, SATADOM/boot device, USB/virtual media, RAM, or BMC issue on Nodes C/D?
- What is the recommended way to fully wipe and reinstall all 4 nodes with AHV + CVM?
- For the Foundation IPMI error, should HTTPS/Web SSL be enabled or disabled on older NX IPMI firmware?
- Is there a recommended way to validate IPMI/RMCP+ connectivity from Windows before running Foundation?
Any guidance on how to proceed with a clean rebuild would be appreciated.
