We moved a NX3000 to a new environment but one of the nodes died. So we had to reimage that node. The other two where still up and running. But after a power-failure on the rack, the two nodes rebooted but won't come up anymore. Genesis showing some failures:2016-08-18 14:38:40 INFO node_manager.py:3492 Svm has configured ip 10.160.35.124 and device eth0 has ip 10.160.35.1242016-08-18 14:38:43 INFO node_manager.py:3542 Setting up key based SSH access to host hypervisor for the first time...2016-08-18 14:38:43 INFO hypervisor_ssh.py:32 Trying to access hypervisor with provided key...2016-08-18 14:38:46 INFO hypervisor_ssh.py:40 Failed.2016-08-18 14:38:46 INFO hypervisor_ssh.py:44 Trying to access hypervisor with provided password...2016-08-18 14:38:49 INFO hypervisor_ssh.py:52 Failed2016-08-18 14:38:49 ERROR node_manager.py:3547 Failed to set up key based SSH access to hypervisor, most likely because we do not have the correct password cached. Please run fix_host_ssh command manually to fix this problem.2016-08-18 14:39:19 CRITICAL node_manager.py:3552 RestartingWe tried running that fix_host_ssh but that didn't work. ssh looks all fine:nutanix@NTNX-13AM3K080045-1-CVM:10.160.35.121:~$ ssh zk2FIPS mode initializedNutanix Controller VMLast login: Thu Aug 18 14:35:40 CEST 2016 from 10.160.35.124 on sshLast login: Thu Aug 18 14:43:47 2016 from 10.160.35.121nutanix@NTNX-13AM3K080045-2-CVM:10.160.35.124:~$ exitlogoutConnection to zk2 closed.nutanix@NTNX-13AM3K080045-1-CVM:10.160.35.121:~$ ssh firstname.lastname@example.orgFIPS mode initializedThe time and date of this login have been sent to the system logs.VMware offers supported, powerful system administration tools. Pleasesee www.vmware.com/go/sysadmintools
for details.The ESXi Shell can be disabled by an administrative user. See thevSphere Security documentation for more information.~ # Any idea what the problem might be here?
Best answer by jgrinwis
just found it out. I have no idea why but all my 192.168.5.1 vmk's interfaces where gone on my esxi hosts, WTF.
just recreated them and genesis is starting fine now.