Hi @rockstershiny 

I suppose that you are going to perform a bare metal foundation on a flat switch, with that being said:

If you are using Supermicro hardware (NX models) the answer is yes, nodes come factory configured with IPMI interface in failover, which behaves in the following way. On boot if dedicated IPMI interface is connected that one will be used otherwise the shared IPMI port will be used. On my own experience what worked best for me is plugging one cable to the IPMI and another to one of the BASE-T ports.

Speaking about Dell and HPE, as far as i know the don’t have a shared IPMI port and you have to connect one cable to the IPMI and another to the 10G ports.

Hope this helps

Regards!

Hi @rockstershiny 

I suppose that you are going to perform a bare metal foundation on a flat switch, with that being said:

 

If you are using Supermicro hardware (NX models) the answer is yes, nodes come factory configured with IPMI interface in failover, which behaves in the following way. On boot if dedicated IPMI interface is connected that one will be used otherwise the shared IPMI port will be used. On my own experience what worked best for me is plugging one cable to the IPMI and another to one of the BASE-T ports.

Speaking about Dell and HPE, as far as i know the don’t have a shared IPMI port and you have to connect one cable to the IPMI and another to the 10G ports.

 

Hope this helps

 

Regards!

Hi, thank you for your detailed answer.

My question is about whether I can do this: Connect one cable to the dedicated IPMI, and the other to the Shared IPMI (instead of a 10G data port)?

I’m asking because I tried doing it in a greenfield deployment last week, and Foundation could not proceed (fatal error: waiting for installer to boot up) with the following error in Phoenix kernel:

eth1 did not provide connectivity to (Workstation IP)

I ended up using the 10Gb SFP port instead of the Shared IPMI, and it worked.

Hi @rockstershiny 

Which hardware are you using?

Regards!

NX-8155-G8

I guess Foundation need to see (at least some of) the networks interfaces to setup the cluster. 

what you describe looks more like the phoenix process than the foundation process.

Foundation process is dedicated to setup a whole cluster, Phoenix is dedicated to just image a node, so you can integrated it to an existing cluster, or just for recovery purpose.

Using phoenix you can probably be able to image you nodes independently and after use foundation to create the cluster. 

I recommend you look at this post : https://next.nutanix.com/installation-configuration-23/appendix-imaging-a-node-phoenix-38084

regards :-)

Hello Dominix, 

Thank you for the reply. Actually the situation I meant was imaging a 4 node cluster via Foundation. At some point during the imaging process the nodes boot up on phoenix, which failed because of failed port connectivity to Foundation VM.

Hi guys!

@rockstershiny 

If you use the shared IPMI port on the nodes to perform the foundation it should work but my personal recommendation is to use always with NX hardware one cable to the dedicated IPMI port and another to the shared IPMI port. Sometimes i get foundation errors when using only the shared one. Another thing to bear in mind is to review if the LAN interface comes preconfigured as failover (it should). 

On the other hand when you are performing a foundation nodes reboot into phoenix to perform the installation.

Hope this helps

Regards!

Hi bcaballero,

Thank you for your reply. I still quite don’t understand how a shared IPMI port (1G) or its failover (10G) can be used for imaging, because I encountered the errors in my screenshot above (NIC flap). 

Hi @rockstershiny 

Here’s an extract from the field installation guide available on this link: Field Installation Guide

Nutanix NX Series: Connect the dedicated IPMI port and any one of the data ports to the switch. Ensure that you use the dedicated IPMI port instead of the shared IPMI port. In addition to the dedicated IPMI port, we highly recommend that you use a 10G data port. You can use a 1G port instead of a 10G data port at the cost of increased imaging time or imaging failure. If you use SFP+ 10G NICs and a 1G RJ45 switch for imaging, connect the 10G data port to the switch by using one of our approved GBICs. If the BMC is configured to use it, you can also use the shared IPMI/1G data port in place of the dedicated port. However, the shared IPMI/1G data port is less reliable than the dedicated port. The IPMI LAN interfaces of the nodes must be in failover mode (factory default setting). 

Nutanix is compatible with any IEEE 802.3 compliant Ethernet switches. SFP module compatibility is determined by the switch vendor. Cables shipped with Nutanix appliances will work with most switch vendors. Certain switch vendors will not work with third party SFP modules. Use the switch vendor's preferred SFP.

If you choose to use the shared IPMI port on G4 and later platforms, make sure that the connected switch can auto-negotiate to 100 Mbps. This auto-negotiation capability is required because the shared IPMI port can support 1 Gbps throughput only when the host is online. If the switch cannot auto-negotiate to 100 Mbps when the host goes offline, make sure to use the dedicated IPMI port instead of the shared port (the dedicated IPMI port support 1 Gbps throughput always). Older platforms support only 10/100 Mbps throughput.

The shared IPMI port also works as data port (on newer modes they are 10G Base-T) and maybe your flat switch has not been able to auto-negotiate to 100Mbps and started searching for other interfaces trying to connect.

It can be used for imaging because as said above it works as data port and the requirements for foundation are to connect one cable to IPMI and other to a Data Port. Shared IPMI port it’s a single port doing both roles.

The point here is that usually you don’t have a 10G SFP+ switch available  to perform a foundation, if you have it, perfect, plug a SFP+ to Base-T adapter connect it to the IPMI and use a Twin-AX cable to connect a port from the node that’s the recommended way to go, but, if you only have a 1G switch then use the IPMI port and the shared port for better results.

Hope this clarify things a bit

Regards!