Installation & Configuration

Welcome to the Nutanix NEXT community. To get started please read our short welcome post. Thanks!

Showing results for 
Search instead for 
Did you mean: 

Imaging Failure with layer 2/3 10G switch for NOS 4.7 and above

SOLVED Go to solution

Re: Imaging Failure with layer 2/3 10G switch for NOS 4.7 and above

bezeddin wrote:

If you plan to upgrade to 4.7. Simply perform one click NOS/AOS upgrade via Prism no need reimaging


If you're using Foundation please check your vlan config on ICX6610 and ICX7250. Make sure its only default vlan or if possible backup your switch config and reset your switch to factory default


I'm using ICX 6650 BASE-T no issue during foundation process






in general, you should not use an enterprise switch with the default configuration for Foundation.


Many enterprise switches have STP enabled in the default configuration and will thus take some time to enable access ports, that is the ports the Nutanix nodes are connected to. Since during the Foundation process the nodes are rebooted several times, each reboot resets the port to wait for STP BPDUs, or what amounts to a timeout for access ports, before allowing any traffic flowing across. Foundation uses remote media via IPMI which times out before the switch allows traffic through the port, thus foundation fails in this situation.


You should configure the switch ports that are used to connect Nutanix nodes for Foundation as STP edge ports (known as "portfast" in e.g. Cisco IOS and NX-OS) to avoid the problems described above.


Switches from Cisco (both IOS and NX-OS) and Arista do not work for Foundation in the default configuration. They need edge port configuration using the "portfast" keyword.


You may have more userfriendly enterprise switches, e.g. from Enterasys (now Extreme) Networks, that use MSTP and auto-edge by default, therefore remembering the automatically detected edge-port status across link-flaps and just working for Foundation in the default configuration.


The default configuration of Extreme Networks EXOS based switches works as well for Foundation.


Dell OS 9 (formerly known as FTOS or (Dell) Force10 switches) cannot be used for Foundation in the default configuration, since all ports are disabled (shutdown) and configured as routed ports (no switchport) by default. Of course they can be configured for Foundation use.


To be sure that Foundation will work, always manually configure STP edge ports where appropriate.


The fact that the needed edge port (aka portfast) configuration is not documented by Nutanix, though they should have the needed expertise to do so, results in the strange recommendation to use dumb switches (which nowadays do not provide STP any more) or manageable switches in default configuration (which works for some brands, but not others). That leads to painful experiences during cluster setup when Foundation fails for no obvious reason.


The fact that the IPMI boot from remote media will silently fall back to local boot further complicates troubleshooting this issue.





Re: Imaging Failure with layer 2/3 10G switch for NOS 4.7 and above

another 10GB switch can be used for reimaging with default config is Dell N series of course with BaseT module