Hi @Jose_orozco



Can you try reset the IPMI on all nodes and try the Foundation process again. Also, please let us know additional details about your setup, like if you are using flat network or VLAN based network, the Foundation version, if using Foundation VM or applet and so on.

---

Getting below error while creating cluster on nutanix node



20191009 01:08:17 DEBUG Setting state of ) @ac10> from PENDING to NR

20191009 01:08:17 WARNING Skipping ) @ac10> because dependencies not met, failed tasks: [) @7810>, ) @7250>, ) @7550>, ) @7210>, ) @7590>, ) @7f50>]

20191009 01:08:17 DEBUG Setting state of ) @a1d0> from PENDING to NR

20191009 01:08:17 WARNING Skipping ) @a1d0> because dependencies not met

---

getting error in node logs



+++++++++++++++++++++++++++



9 02:38:59 INFO Running ) @c210>

20191009 02:38:59 DEBUG Cache MISS: key(_()_{'global_config': })

20191009 02:38:59 INFO Validating parameters. This may take few minutes

20191009 02:38:59 INFO Validating parameters. This may take few minutes

20191009 02:38:59 INFO Validating parameters. This may take few minutes

20191009 02:38:59 WARNING skip Haswell, Broadwell mix check for cluster

20191009 02:38:59 WARNING tmap was called with empty args and kwargs, ignoring

20191009 02:38:59 INFO Validating whether all members of cluster YIPLSMHPBCLU001 belong to the same arch

20191009 02:39:02 INFO Node with ip 10.25.37.153 is in phoenix. Generating hardware_config.json

20191009 02:39:04 INFO Node with ip 10.25.37.156 is in phoenix. Generating hardware_config.json

20191009 02:39:07 INFO Node with ip 10.25.37.159 is in phoenix. Generating hardware_config.json

20191009 02:39:09 INFO Running one and two node validations

20191009 02:39:12 DEBUG Setting state of ) @c210> from RUNNING to FINISHED

20191009 02:39:12 INFO Completed ) @c210>

20191009 02:39:12 DEBUG Setting state of ) @dad0> from PENDING to RUNNING

20191009 02:39:12 INFO Running ) @dad0>

20191009 02:39:12 INFO Node IP: CVM(10.25.37.153) HOST(10.25.37.152) IPMI(10.25.37.151)

20191009 02:39:12 DEBUG Cache HIT: key(_(u'/home/nutanix/foundation/nos/nutanix_installer_package-release-euphrates-5.10.7-stable.tar',)_{})

20191009 02:39:12 INFO NOS Version is: 5.10.7

20191009 02:39:12 INFO metadata update: {"nos_version": "5.10.7"}

20191009 02:39:12 DEBUG Setting state of ) @dad0> from RUNNING to FINISHED

20191009 02:39:12 INFO Completed ) @dad0>

20191009 02:39:12 DEBUG Setting state of ) @db50> from PENDING to RUNNING

20191009 02:39:12 INFO Running ) @db50>

20191009 02:39:12 DEBUG Trying to use CVM/Phoenix to start imaging

20191009 02:39:14 ERROR Exception on executing cmd: ['uname', '-m']

Traceback (most recent call last):

File "foundation/foundation_tools.py", line 218, in ssh

File "foundation/foundation_tools.py", line 194, in get_ssh_client

File "/home/nutanix/foundation/bin/../lib/py/paramiko-1.18.4-py2.7.egg/paramiko/client.py", line 382, in connect

look_for_keys, gss_auth, gss_kex, gss_deleg_creds, gss_host)

File "/home/nutanix/foundation/bin/../lib/py/paramiko-1.18.4-py2.7.egg/paramiko/client.py", line 624, in _auth

raise saved_exception

AuthenticationException: Authentication failed.

20191009 02:39:14 INFO CVM/Phoenix detected, using it to continue imaging

20191009 02:39:14 DEBUG Setting state of ) @db50> from RUNNING to FINISHED

20191009 02:39:14 INFO Completed ) @db50>

20191009 02:39:14 DEBUG Setting state of ) @8c10> from PENDING to RUNNING

20191009 02:39:14 INFO Running ) @8c10>

20191009 02:39:14 DEBUG Setting state of ) @8c10> from RUNNING to FINISHED

20191009 02:39:14 INFO Completed ) @8c10>

20191009 02:39:14 DEBUG Setting state of ) @8890> from PENDING to RUNNING

20191009 02:39:14 INFO Running ) @8890>

20191009 02:39:14 DEBUG Setting state of ) @8050> from PENDING to RUNNING

20191009 02:39:14 INFO Running ) @8050>

20191009 02:39:14 DEBUG skipped, this node will be imaged via CVM

20191009 02:39:14 DEBUG Setting state of ) @8050> from RUNNING to FINISHED

20191009 02:39:14 INFO Completed ) @8050>

20191009 02:39:15 ERROR Exception on executing cmd: ['uname', '-m']

Traceback (most recent call last):

File "foundation/foundation_tools.py", line 218, in ssh

File "foundation/foundation_tools.py", line 194, in get_ssh_client

File "/home/nutanix/foundation/bin/../lib/py/paramiko-1.18.4-py2.7.egg/paramiko/client.py", line 382, in connect

look_for_keys, gss_auth, gss_kex, gss_deleg_creds, gss_host)

File "/home/nutanix/foundation/bin/../lib/py/paramiko-1.18.4-py2.7.egg/paramiko/client.py", line 624, in _auth

raise saved_exception

AuthenticationException: Authentication failed.

20191009 02:39:15 INFO Discovered arch x86_64

20191009 02:39:15 INFO Checking if target node is already in Phoenix

20191009 02:39:16 INFO Target node is running Phoenix. Moving to next step.

20191009 02:39:16 INFO foundation version 4.4.3, phoenix version 4.4.3

20191009 02:39:16 DEBUG Skipping phoenix upgrade as foundation and phoenix are running same version 4.4.3

20191009 02:39:16 INFO Phoenix upgrade not required

20191009 02:39:16 DEBUG Setting state of ) @8890> from RUNNING to FINISHED

20191009 02:39:16 INFO Completed ) @8890>

20191009 02:39:17 DEBUG Setting state of ) @8ad0> from PENDING to RUNNING

20191009 02:39:17 INFO Running ) @8ad0>

20191009 02:39:17 INFO Node with ip 10.25.37.153 is in phoenix. Generating hardware_config.json

20191009 02:39:19 DEBUG not a G7/RAID platform node, skipping RAID checks

20191009 02:39:19 DEBUG Setting state of ) @8ad0> from RUNNING to FINISHED

20191009 02:39:19 INFO Completed ) @8ad0>

20191009 02:39:19 DEBUG Setting state of ) @6d50> from PENDING to RUNNING

20191009 02:39:19 INFO Running ) @6d50>

20191009 02:39:19 INFO Rebooting into staging environment

20191009 02:39:20 DEBUG Cache HIT: key(_(u'/home/nutanix/foundation/nos/nutanix_installer_package-release-euphrates-5.10.7-stable.tar',)_{})

20191009 02:39:20 INFO NOS version is 5.10.7

20191009 02:39:20 INFO Preparing NOS package (/home/nutanix/foundation/nos/nutanix_installer_package-release-euphrates-5.10.7-stable.tar)

20191009 02:39:20 DEBUG Cache HIT: key(_(u'/home/nutanix/foundation/nos/nutanix_installer_package-release-euphrates-5.10.7-stable.tar',)_{})

20191009 02:39:21 DEBUG Recorded system information at /home/nutanix/foundation/log/20191009-023856-21/system_information_10.25.37.153.json

20191009 02:39:22 INFO Node with ip 10.25.37.153 is in phoenix. Generating hardware_config.json

20191009 02:39:24 INFO metadata update: {"hardware_config": {"node": {"storage_controllers": [{"subsystem": "15d9:0808", "name": "LSI Logic / Symbios Logic SAS3008 PCI-Express Fusion-MPT SAS-3", "mapping": [{"slot_designation": "1", "hba_address": "0", "slot_id": null, "location": {"access_plane": 1, "cell_x": 0, "width": 5, "cell_y": 1, "height": 2}, "led_address": "sas3ircu:0,1:0"}, {"slot_designation": "2", "hba_address": "1", "slot_id": null, "location": {"access_plane": 1, "cell_x": 5, "width": 5, "cell_y": 1, "height": 2}, "led_address": "sas3ircu:0,1:1"}, {"slot_designation": "3", "hba_address": "2", "slot_id": null, "location": {"access_plane": 1, "cell_x": 10, "width": 5, "cell_y": 1, "height": 2}, "led_address": "sas3ircu:0,1:2"}, {"slot_designation": "4", "hba_address": "3", "slot_id": null, "location": {"access_plane": 1, "cell_x": 15, "width": 5, "cell_y": 1, "height": 2}, "led_address": "sas3ircu:0,1:3"}], "address": "1000:0097:0", "attributes": {}, "class": "MPTSAS"}], "misc_device_classes": [{"model": "Tesla M10", "sensors": [{"address": "nvidia_smi:temperature,gpu_temp", "type": "temperature", "max_temp_c": 90}], "type": "GPU", "make": "NVIDIA"}, {"model": "Tesla P40", "sensors": [{"address": "nvidia_smi:temperature,gpu_temp", "type": "temperature", "max_temp_c": 90}], "type": "GPU", "make": "NVIDIA"}], "hba": {"subsystem": "15d9:0808", "name": "LSI Logic / Symbios Logic SAS3008 PCI-Express Fusion-MPT SAS-3", "mapping": [{"slot_designation": "1", "hba_address": "0", "slot_id": null, "location": {"access_plane": 1, "cell_x": 0, "width": 5, "cell_y": 1, "height": 2}, "led_address": "sas3ircu:0,1:0"}, {"slot_designation": "2", "hba_address": "1", "slot_id": null, "location": {"access_plane": 1, "cell_x": 5, "width": 5, "cell_y": 1, "height": 2}, "led_address": "sas3ircu:0,1:1"}, {"slot_designation": "3", "hba_address": "2", "slot_id": null, "location": {"access_plane": 1, "cell_x": 10, "width": 5, "cell_y": 1, "height": 2}, "led_address": "sas3ircu:0,1:2"}, {"slot_designation": "4", "hba_address": "3", "slot_id": null, "location": {"access_plane": 1, "cell_x": 15, "width": 5, "cell_y": 1, "height": 2}, "led_address": "sas3ircu:0,1:3"}], "address": "1000:0097:0", "attributes": {}, "class": "MPTSAS"}, "cpus": [{"sensors": [{"type": "temperature", "address": "ipmi_sensor:CPU1 Temp"}, {"type": "voltage", "address": "ipmi_sensor:Vcpu1 "}], "description": "cpu 1"}], "model_string": "NX-1175S-G6", "fans": [{"sensors": [{"type": "rpm", "address": "ipmi_sensor:FAN2"}], "description": "fan 2"}, {"sensors": [{"type": "rpm", "address": "ipmi_sensor:FAN3"}], "description": "fan 3"}, {"sensors": [{"type": "rpm", "address": "ipmi_sensor:FAN4"}], "description": "fan 4"}, {"sensors": [{"type": "rpm", "address": "ipmi_sensor:FAN5"}], "description": "fan 5"}, {"sensors": [{"type": "rpm", "address": "ipmi_sensor:FAN6"}], "description": "fan 6"}], "node_number": 1, "location": {"access_plane": 2, "cell_x": 0, "width": 6, "cell_y": 0, "height": 4}, "memory": [{"sensors": [{"type": "temperature", "address": "ipmi_sensor:P1-DIMMA1 Temp"}, {"type": "voltage", "address": "ipmi_sensor:VDIMMP1ABC"}], "description": "dimm p1a1"}, {"sensors": [{"type": "temperature", "address": "ipmi_sensor:P1-DIMMB1 Temp"}, {"type": "voltage", "address": "ipmi_sensor:VDIMMP1ABC"}], "description": "dimm p1b1"}, {"sensors": [{"type": "temperature", "address": "ipmi_sensor:P1-DIMMC1 Temp"}, {"type": "voltage", "address": "ipmi_sensor:VDIMMP1ABC"}], "description": "dimm p1c1"}, {"sensors": [{"type": "temperature", "address": "ipmi_sensor:P1-DIMMD1 Temp"}, {"type": "voltage", "address": "ipmi_sensor:VDIMMP1DEF"}], "description": "dimm p1d1"}, {"sensors": [{"type": "temperature", "address": "ipmi_sensor:P1-DIMME1 Temp"}, {"type": "voltage", "address": "ipmi_sensor:VDIMMP1DEF"}], "description": "dimm p1e1"}, {"sensors": [{"type": "temperature", "address": "ipmi_sensor:P1-DIMMF1 Temp"}, {"type": "voltage", "address": "ipmi_sensor:VDIMMP1DEF"}], "description": "dimm p1f1"}], "sensors": [{"type": "temperature", "address": "ipmi_sensor:System Temp"}], "boot_device": {"controller": "HBA:8086:a1d2:0", "mapping": [{"slot_designation": "0", "slot_id": null, "port": "3", "preferred": true}], "structure": "ONLY_HYPERVISOR"}, "hardware_attributes": {"two_node_cluster": true, "default_workload": "vdi", "one_node_cluster": true, "robo_mixed_hypervisor": true, "no_conventional_cluster": true, "lcm_family": "smc_gen_11", "maybe_1GbE_only": true}}, "version": 1, "chassis": {"height_u": 1, "num_nodes": 1, "access_planes": [{"rows": 4, "type": "chassis_front", "id": 1, "columns": 20, "description": "Front of chassis"}, {"rows": 4, "type": "chassis_rear", "id": 2, "columns": 10, "description": "Rear of chassis"}], "class": "SMIPMI", "power_supplies": [{"sensors": [{"discrete_response_mask": 15, "type": "discrete", "discrete_response_expected": 1, "address": "ipmi_sensor:PS1 Status"}], "description": "power supply 1", "location": {"access_plane": 2, "cell_x": 6, "width": 2, "cell_y": 0, "height": 4}}, {"sensors": [{"discrete_response_mask": 15, "type": "discrete", "discrete_response_expected": 1, "address": "ipmi_sensor:PS2 Status"}], "description": "power supply 2", "location": {"access_plane": 2, "cell_x": 8, "width": 2, "cell_y": 0, "height": 4}}]}}}

20191009 02:39:25 INFO Boot device WWN: 500a075122fcce42

20191009 02:39:26 INFO Boot device serial: 192822FCCE42

20191009 02:39:27 INFO Copied /home/nutanix/foundation/templates/crystal_plat_reference.json to phoenix at ip 10.25.37.153

20191009 02:39:30 ERROR Command '/bin/cat /active_nic/address' returned error code 1

stdout:



stderr:

/bin/cat: /active_nic/address: No such file or directory



20191009 02:39:31 ERROR Command 'cat /tmp/online_interfaces' returned error code 1

stdout:



stderr:

cat: /tmp/online_interfaces: No such file or directory



20191009 02:39:31 INFO This node is not a gold node

20191009 02:39:31 ERROR Command '/usr/bin/ipmicfg-linux.x86_64 -tp info' returned error code 13

stdout:



stderr:

Not TwinPro



20191009 02:39:32 ERROR Command '/usr/bin/ipmicfg-linux.x86_64 -tp nodeid' returned error code 13

stdout:



stderr:

Not TwinPro



20191009 02:39:32 DEBUG Setting state of ) @6d50> from RUNNING to FINISHED

20191009 02:39:32 INFO Completed ) @6d50>

20191009 02:39:32 DEBUG Setting state of ) @6790> from PENDING to RUNNING

20191009 02:39:32 INFO Running ) @6790>

20191009 02:39:32 DEBUG Setting state of ) @6590> from PENDING to RUNNING

20191009 02:39:32 INFO Making node specific Phoenix json. This may take few minutes

20191009 02:39:32 INFO Running ) @6590>

20191009 02:39:32 DEBUG Cache HIT: key(_(u'/home/nutanix/foundation/nos/nutanix_installer_package-release-euphrates-5.10.7-stable.tar',)_{})

20191009 02:39:32 DEBUG Skipping non serializable keys: ['graph', 'imaging_thread', 'tasks', '_session_config', '_events']

20191009 02:39:32 DEBUG Cache HIT: key(_(u'/home/nutanix/foundation/nos/nutanix_installer_package-release-euphrates-5.10.7-stable.tar',)_{})

20191009 02:39:32 DEBUG Cache HIT: key(_('/home/nutanix/foundation/lib/driver/kvm/anaconda.tar.gz',)_{})

20191009 02:39:32 DEBUG Cache HIT: key(_('/home/nutanix/foundation/lib/driver/kvm/rpms/OM-iSM-Dell-Web-LX-340-1471_A00.tar.gz',)_{})

20191009 02:39:32 DEBUG Cache HIT: key(_('/home/nutanix/foundation/lib/driver/kvm/rpms/DellPTAgent-1.9.0-371.x86_64.rpm',)_{})

20191009 02:39:32 DEBUG Cache MISS: key(_('/home/nutanix/foundation/tmp/sessions/20191009-023856-21/files/arizona.node_10.25.37.153.json',)_{})

20191009 02:39:44 INFO A previous Phoenix failure detected. Clearing temp files

20191009 02:39:46 INFO Start downloading resources, this may take several minutes

20191009 02:39:47 INFO Waiting for Phoenix to finish downloading resources

20191009 02:39:48 INFO Model detected: NX-1175S-G6

20191009 02:39:49 INFO vpd_info {'node_serial': 'OM196S020810', 'vpd_method': None, 'cluster_id': 1945744289975860186, 'rackable_unit_serial': '19SM59330008'}

20191009 02:39:49 INFO Using node_serial from FRU

20191009 02:39:49 INFO Using block_id from FRU

20191009 02:39:49 INFO Using cluster_id from FRU

20191009 02:39:49 CRITICAL Node position must be specified

20191009 02:39:49 ERROR Exception in running ) @6590>

Traceback (most recent call last):

File "foundation/imaging_step.py", line 161, in _run

File "foundation/imaging_step_hypervisor.py", line 45, in run

File "foundation/imaging_step.py", line 350, in wait_for_event

StandardError: Received fatal in waiting for event Running CVM Installer: Node position must be specified

20191009 02:39:49 DEBUG Setting state of ) @6590> from RUNNING to FAILED

20191009 02:39:49 WARNING Foundation timed out waiting for phoenix to download resources. Fataling other imaging steps which are waiting

20191009 02:39:49 ERROR Exception in ) @6790>

Traceback (most recent call last):

File "foundation/decorators.py", line 77, in wrap_method

File "foundation/imaging_step_phoenix.py", line 263, in run

File "foundation/imaging_step.py", line 350, in wait_for_event

StandardError: Received fatal in waiting for event Running CVM Installer: Node position must be specified

20191009 02:39:49 ERROR Exception in running ) @6790>

Traceback (most recent call last):

File "foundation/imaging_step.py", line 161, in _run

File "foundation/decorators.py", line 77, in wrap_method

File "foundation/imaging_step_phoenix.py", line 263, in run

File "foundation/imaging_step.py", line 350, in wait_for_event

StandardError: Received fatal in waiting for event Running CVM Installer: Node position must be specified

20191009 02:39:49 DEBUG Setting state of ) @6790> from RUNNING to FAILED

---

Hi nikhilkumar,

 

For the error “A node must be especified” take a look at the second tab in Foundation VM or applet installer (tab NODES). You must need to especify the nodes order (A,B,C,D) at the colunm nodes. I had a similar issue, because I forget to especify the order, and all my nodes were with “A”

I hope this can help you.

 

Br,

 

Thelmo

---