Phoenix fails with “Cannot read: Input/output error” while reinstalling NX node after power outage

Hi Everyone.

I have a 4-node Nutanix NX demo/POC cluster running AHV. After a power outage at the office, the cluster did not recover properly.

Initial situation:

• 4-node Nutanix NX cluster
• Hypervisor: AHV
• This is a POC environment, so there is no important data to recover
• After the outage, only the CVM on Node A was active
• Nodes B, C and D only booted into AHV
• On Nodes B, C and D, virsh list --all did not show any CVM

What I already tried:

• I generated a Phoenix ISO from the working CVM using the AHV ISO and Nutanix/AOS installer package
• The generated Phoenix ISO is around 6 GB
• I used that same Phoenix ISO successfully on Node B
• On Node B, I selected “Wipe data and install CVM + hypervisor” and the node was reinstalled successfully
• However, Node C and Node D fail during the same Phoenix reinstall process

Phoenix error on Node C:

tar: /mnt/local/images/svm/nutanix_installer_package-release-euphrates-5.20.0.1-stable-f5e54bf50b9b92d1e27560e422629e322968a3e0-x86_64.tar: Cannot read: Input/output error
tar: Too many errors, quitting
tar: Error is not recoverable: exiting now
FATAL An exception was raised:
StandardError: Failed command: [tar xvf /mnt/local/images/svm/nutanix_installer_package-release-euphrates-5.20.0.1-stable-f5e54bf50b9b92d1e27560e422629e322968a3e0-x86_64.tar -C /mnt/svm_installer --totals] with reason [None]

Additional Foundation issue:
I also tried Nutanix Portable Foundation, but it fails connecting to the IPMI interfaces.

IPMI addresses:

• Node A: 172.16.70.213
• Node B: 172.16.70.214
• Node C: 172.16.70.215
• Node D: 172.16.70.216

Foundation error:

Failed to connect to the following IPMIs:
172.16.70.213, 172.16.70.214, 172.16.70.215, 172.16.70.216

Network/IPMI tests:

• Ping to all IPMI addresses works
• I can access the IPMI web interface using HTTP/port 80
• TCP 443 works only on 172.16.70.213
• TCP 443 fails on 172.16.70.214, 172.16.70.215 and 172.16.70.216
• I tested Portable Foundation from a Windows Server
• I also tested Portable Foundation from a laptop connected to the same switch and same VLAN as the IPMI ports, but the same error occurs

USB boot issue:

• I created a bootable USB using BalenaEtcher with the Phoenix ISO
• I connected it to the rear USB ports of the Nutanix node
• In the boot menu, the USB device did not appear as expected
• I only saw a SanDisk option, but selecting it booted the existing AHV installation
• I tried multiple rear USB ports with the same result

Current goal:
Since this is only a POC cluster, I do not need to recover the old data. I just need to reinstall/rebuild Nutanix from scratch, even if the cluster is empty.

Questions:

1. Why would Phoenix fail with “Cannot read: Input/output error” on Node C/D while extracting the AOS/SVM installer package, if the same ISO worked successfully on Node B?
2. Could this indicate a local disk, SATADOM/boot device, USB/virtual media, RAM, or BMC issue on Nodes C/D?
3. What is the recommended way to fully wipe and reinstall all 4 nodes with AHV + CVM?
4. For the Foundation IPMI error, should HTTPS/Web SSL be enabled or disabled on older NX IPMI firmware?
5. Is there a recommended way to validate IPMI/RMCP+ connectivity from Windows before running Foundation?

Any guidance on how to proceed with a clean rebuild would be appreciated.