AOS installation failed with fatal error

Question

Hi !Does anyone know how to fix the issue ? 2023-10-04 09:35:23,605Z DEBUG Setting state of ) @4e90> from PENDING to RUNNING2023-10-04 09:35:23,641Z INFO Running ) @4e90>2023-10-04 09:35:23,670Z DEBUG Cache MISS: key(_()_{'global_config': })2023-10-04 09:35:23,677Z INFO Running redundancy factor validations2023-10-04 09:35:23,677Z INFO Validating parameters. This may take few minutes2023-10-04 09:35:23,677Z INFO Validating parameters. This may take few minutes2023-10-04 09:35:23,678Z INFO Validating parameters. This may take few minutes2023-10-04 09:35:23,681Z INFO Running validations to ensure non-duplicacy of hostnames2023-10-04 09:35:23,683Z INFO Running hypervisor hostname pattern validations2023-10-04 09:35:23,687Z INFO Running HyperV hostname validations2023-10-04 09:35:23,687Z INFO Running lacp validations2023-10-04 09:35:23,687Z INFO Running validations on AOS tarball2023-10-04 09:35:23,687Z INFO Running AHV bundled with AOS validations2023-10-04 09:35:23,770Z DEBUG Cache HIT: key(_()_{'nos_package_path': u'C:\Program Files (x86)\Nutanix\Portable Foundation\nos\nutanix_installer_package-release-fraser-6.5.1.5-stable-5189b7ebf56ea74b69369e03ed815427a30e2c15-x86_64.tar'})2023-10-04 09:35:24,673Z INFO Running chassis configuration validations2023-10-04 09:35:24,674Z INFO Ensuring there is no Haswell, Broadwell mix in the same chassis for cluster HKAHVCL012023-10-04 09:35:26,381Z INFO Running RDMA validations2023-10-04 09:35:26,381Z WARNING tmap was called with empty args and kwargs, ignoring2023-10-04 09:35:26,381Z INFO Running one and two node validations2023-10-04 09:36:08,490Z INFO Checking whether hypervisor-specific drivers are present inside Foundation2023-10-04 09:36:08,490Z DEBUG Setting state of ) @4e90> from RUNNING to FINISHED2023-10-04 09:36:08,490Z INFO Completed ) @4e90>2023-10-04 09:36:08,490Z DEBUG Setting state of ) @4670> from PENDING to RUNNING2023-10-04 09:36:08,490Z INFO Running ) @4670>2023-10-04 09:36:08,506Z INFO Node IP: CVM(10.5.86.42) HOST(10.5.86.43) IPMI(10.5.86.41)2023-10-04 09:36:08,506Z DEBUG Cache HIT: key(_()_{'nos_package_path': u'C:\Program Files (x86)\Nutanix\Portable Foundation\nos\nutanix_installer_package-release-fraser-6.5.1.5-stable-5189b7ebf56ea74b69369e03ed815427a30e2c15-x86_64.tar'})2023-10-04 09:36:08,506Z INFO NOS Version is: 6.5.1.52023-10-04 09:36:08,506Z INFO metadata update: {"nos_version": "6.5.1.5"}2023-10-04 09:36:08,506Z DEBUG Setting state of ) @4670> from RUNNING to FINISHED2023-10-04 09:36:08,506Z INFO Completed ) @4670>2023-10-04 09:36:08,506Z DEBUG Setting state of ) @4e10> from PENDING to RUNNING2023-10-04 09:36:08,520Z INFO Running ) @4e10>2023-10-04 09:36:08,520Z INFO Attempting to detect device type on 10.5.86.412023-10-04 09:36:08,520Z INFO Factory mode is False2023-10-04 09:36:08,520Z INFO Loading Tartarus from C:\Program Files (x86)\Nutanix\Portable Foundation\lib\foundation-platforms artarus artarus.base-1.11.1_f07b7b47-py2.7.egg2023-10-04 09:36:08,959Z INFO Registering hypervisor if applicable2023-10-04 09:36:17,436Z WARNING Failed to register internal hypervisor: '") at 0x5765878>2023-10-04 09:36:17,436Z INFO Tartarus initialization complete2023-10-04 09:36:20,976Z DEBUG Failed to load all plugins: Unable to satisfy required capability 'LOCAL_SHELL' for Skipping command##.GenericRedfish_Pyghmi-1.0.0, entity is disabled Unable to satisfy required capability 'LOCAL_SHELL' for Unable to satisfy required capability 'LOCAL_SHELL' for Unable to satisfy required capability 'LOCAL_SHELL' for Unable to satisfy required capability 'LOCAL_SHELL' for Unable to satisfy required capability 'LOCAL_SHELL' for Unable to satisfy required capability 'LOCAL_SHELL' for Unable to satisfy required capability 'LOCAL_SHELL' for Unable to satisfy required capability 'LOCAL_SHELL' for 2023-10-04 09:36:20,976Z INFO Registered new credentials with alias '10.5.86.41-ADMIN'2023-10-04 09:36:20,976Z INFO Registered new host with alias '10.5.86.41'2023-10-04 09:36:21,006Z INFO Fallback to legacy bmc detection2023-10-04 09:36:21,009Z ERROR Could not execute 'ipmitool -I lanplus -H 10.5.86.41 -U ADMIN -P fru'Traceback (most recent call last): File "C:\Program Files (x86)\Nutanix\Portable Foundation\lib\foundation-platforms\foundation_bmc-2.13.2+r.1841.c1fecc5-py2.7.egg\bmc_utils ools.py", line 81, in system creationflags=CREATE_NO_WINDOW) File "site-packages\subprocess32.py", line 617, in __init__ File "site-packages\subprocess32.py", line 922, in _execute_childWindowsError: [Error 2] The system cannot find the file specified2023-10-04 09:36:21,009Z ERROR Command 'ipmitool -I lanplus -H 10.5.86.41 -U ADMIN -P fru' returned error code -1stdout:stderr:2023-10-04 09:36:21,026Z DEBUG SMC Board Model: X12SPW-TF-NI22-P2023-10-04 09:36:21,026Z INFO Detected class smc_cgi for node with IPMI IP 10.5.86.412023-10-04 09:36:21,026Z ERROR Could not execute 'ipmitool -I lanplus -H 10.5.86.41 -U ADMIN -P fru'Traceback (most recent call last): File "C:\Program Files (x86)\Nutanix\Portable Foundation\lib\foundation-platforms\foundation_bmc-2.13.2+r.1841.c1fecc5-py2.7.egg\bmc_utils ools.py", line 81, in system creationflags=CREATE_NO_WINDOW) File "site-packages\subprocess32.py", line 617, in __init__ File "site-packages\subprocess32.py", line 922, in _execute_childWindowsError: [Error 2] The system cannot find the file specified2023-10-04 09:36:21,026Z ERROR Command 'ipmitool -I lanplus -H 10.5.86.41 -U ADMIN -P fru' returned error code -1stdout:stderr:2023-10-04 09:36:21,026Z DEBUG Setting state of ) @4e10> from RUNNING to FINISHED2023-10-04 09:36:21,026Z INFO Completed ) @4e10>2023-10-04 09:36:21,035Z DEBUG Setting state of ) @4f50> from PENDING to RUNNING2023-10-04 09:36:21,035Z INFO Running ) @4f50>2023-10-04 09:36:21,035Z DEBUG Setting state of ) @4f50> from RUNNING to FINISHED2023-10-04 09:36:21,035Z INFO Completed ) @4f50>2023-10-04 09:36:21,035Z DEBUG Setting state of ) @40b0> from PENDING to RUNNING2023-10-04 09:36:21,035Z DEBUG Setting state of ) @48b0> from PENDING to RUNNING2023-10-04 09:36:21,035Z INFO Running ) @40b0>2023-10-04 09:36:21,035Z INFO Running ) @48b0>2023-10-04 09:36:21,035Z INFO Making node specific Phoenix image2023-10-04 09:36:21,035Z DEBUG skipped, this node will be imaged via IPMI2023-10-04 09:36:21,045Z DEBUG Setting state of ) @48b0> from RUNNING to FINISHED2023-10-04 09:36:21,045Z INFO Completed ) @48b0>2023-10-04 09:36:21,671Z INFO Using C:\Program Files (x86)\Nutanix\Portable Foundation\lib/foundation-platforms\aurora-1.4.0-7954202.tar.gz2023-10-04 09:36:21,671Z DEBUG Cache HIT: key(_('C:\Program Files (x86)\Nutanix\Portable Foundation\lib\foundation-platforms\aurora-1.4.0-7954202.tar.gz',)_{})2023-10-04 09:36:21,671Z INFO Using C:\Program Files (x86)\Nutanix\Portable Foundation\lib/foundation-platforms artarus-1.11.1-f07b7b47-py2.7.tar.gz2023-10-04 09:36:21,671Z DEBUG Cache HIT: key(_('C:\Program Files (x86)\Nutanix\Portable Foundation\lib\foundation-platforms\tartarus-1.11.1-f07b7b47-py2.7.tar.gz',)_{})2023-10-04 09:36:21,674Z INFO Using C:\Program Files (x86)\Nutanix\Portable Foundation\lib/foundation-platforms\phorest-phoenix-1.1.1-31a83ca.tgz2023-10-04 09:36:21,674Z DEBUG Cache HIT: key(_('C:\Program Files (x86)\Nutanix\Portable Foundation\lib\foundation-platforms\phorest-phoenix-1.1.1-31a83ca.tgz',)_{})2023-10-04 09:36:21,674Z INFO Using C:\Program Files (x86)\Nutanix\Portable Foundation mp\sessions\20231004-173459-3\files\features_node_10.5.86.42.json2023-10-04 09:36:21,674Z DEBUG Cache MISS: key(_('C:\Program Files (x86)\Nutanix\Portable Foundation\tmp\sessions\20231004-173459-3\files\features_node_10.5.86.42.json',)_{})2023-10-04 09:36:21,674Z INFO Using C:\Program Files (x86)\Nutanix\Portable Foundation\lib/foundation-platforms/hcl.json2023-10-04 09:36:21,674Z DEBUG Cache HIT: key(_('C:\Program Files (x86)\Nutanix\Portable Foundation\lib\foundation-platforms\hcl.json',)_{})2023-10-04 09:36:21,674Z INFO Using C:\Program Files (x86)\Nutanix\Portable Foundation\config\config_parameters.json2023-10-04 09:36:21,674Z DEBUG Cache HIT: key(_('C:\Program Files (x86)\Nutanix\Portable Foundation\config\config_parameters.json',)_{})2023-10-04 09:36:21,674Z INFO Using c:\program files (x86) utanix\portable foundation\lib\foundation-platforms\foundation_layout-2.13.2+r.1841.c1fecc5-py2.7.egg2023-10-04 09:36:21,674Z DEBUG Cache HIT: key(_('c:\program files (x86)\nutanix\portable foundation\lib\foundation-platforms\foundation_layout-2.13.2+r.1841.c1fecc5-py2.7.egg',)_{})2023-10-04 09:36:21,678Z DEBUG Generating updates config json C:\Program Files (x86)\Nutanix\Portable Foundation mp\sessions\20231004-173459-3\files\updates_config_node_10.5.86.42.json2023-10-04 09:36:21,678Z DEBUG Cache MISS: key(_('C:\Program Files (x86)\Nutanix\Portable Foundation\tmp\sessions\20231004-173459-3\files\updates_config_node_10.5.86.42.json',)_{})2023-10-04 09:36:21,684Z DEBUG Cache HIT: key(_('C:\Program Files (x86)\Nutanix\Portable Foundation\lib\phoenix\x86_64\squashfs.img',)_{})2023-10-04 09:36:21,684Z DEBUG Cache HIT: key(_('C:\Program Files (x86)\Nutanix\Portable Foundation\lib\phoenix\x86_64\squashfs.img',)_{})2023-10-04 09:36:21,684Z DEBUG using C:\Program Files (x86)\Nutanix\Portable Foundation\lib\bin\cygwin\mkisofs.exe for mkisofs2023-10-04 09:36:22,005Z INFO Loading Tartarus from C:\Program Files (x86)\Nutanix\Portable Foundation\lib\foundation-platforms artarus artarus.base-1.11.1_f07b7b47-py2.7.egg2023-10-04 09:36:22,401Z INFO Registering hypervisor if applicable2023-10-04 09:36:24,571Z WARNING Failed to register internal hypervisor: '") at 0xa3fb800>2023-10-04 09:36:24,571Z INFO Tartarus initialization complete2023-10-04 09:36:29,124Z DEBUG Failed to load all plugins: Unable to satisfy required capability 'LOCAL_SHELL' for Skipping command##.GenericRedfish_Pyghmi-1.0.0, entity is disabled Unable to satisfy required capability 'LOCAL_SHELL' for Unable to satisfy required capability 'LOCAL_SHELL' for Unable to satisfy required capability 'LOCAL_SHELL' for Unable to satisfy required capability 'LOCAL_SHELL' for Unable to satisfy required capability 'LOCAL_SHELL' for Unable to satisfy required capability 'LOCAL_SHELL' for Unable to satisfy required capability 'LOCAL_SHELL' for Unable to satisfy required capability 'LOCAL_SHELL' for 2023-10-04 09:36:29,124Z INFO Selected legacy remote boot implementation smc_cgi via 2023-10-04 09:36:29,170Z DEBUG Cache MISS: key(_('C:\Program Files (x86)\Nutanix\Portable Foundation\tmp\sessions\20231004-173459-3\phoenix_node_isos\foundation.node_10.5.86.42.iso',)_{})2023-10-04 09:36:29,276Z INFO Setting boot mode2023-10-04 09:36:29,276Z DEBUG No implementation found for setting boot mode to BootMode.LEGACY. Skipping2023-10-04 09:36:29,276Z INFO Powering off node2023-10-04 09:36:29,288Z INFO current power state is on2023-10-04 09:36:29,288Z DEBUG changing power state to off, attempt 1/42023-10-04 09:36:38,427Z INFO changed power state to off2023-10-04 09:36:43,441Z INFO Mounting phoenix iso image2023-10-04 09:36:43,441Z INFO Starting SMCGI Virtual CDROM2023-10-04 09:36:43,441Z INFO Detecting power status2023-10-04 09:36:43,457Z INFO current power state is off2023-10-04 09:36:43,457Z INFO Attaching virtual media: http://10.5.86.100:8000/files/tmp/sessions/20231004-173459-3/phoenix_node_isos/foundation.node_10.5.86.42.iso2023-10-04 09:36:43,457Z DEBUG Starting new HTTPS connection (1): 10.5.86.41:4432023-10-04 09:36:43,566Z DEBUG https://10.5.86.41:443 "GET / HTTP/1.1" 200 104302023-10-04 09:36:43,566Z INFO Encoding credentials with btoa2023-10-04 09:36:43,598Z DEBUG https://10.5.86.41:443 "POST /cgi/login.cgi HTTP/1.1" 200 4832023-10-04 09:36:43,660Z DEBUG https://10.5.86.41:443 "GET /cgi/url_redirect.cgi?url_name=topmenu HTTP/1.1" 200 282342023-10-04 09:36:43,660Z DEBUG Login to BMC with CSRF Token2023-10-04 09:36:43,660Z DEBUG SMCGI Req: cgi=op, op=config_iso, args={'path': u'/files/tmp/sessions/20231004-173459-3/phoenix_node_isos/foundation.node_10.5.86.42.iso', 'host': u'http://10.5.86.100:8000', 'pwd': '', 'user': ''}2023-10-04 09:36:43,707Z DEBUG https://10.5.86.41:443 "POST /cgi/op.cgi HTTP/1.1" 200 None2023-10-04 09:36:43,753Z DEBUG SMCGI Resp: text=ok2023-10-04 09:36:43,753Z DEBUG SMCGI Req: cgi=op, op=mount_iso, args={}2023-10-04 09:36:43,769Z DEBUG https://10.5.86.41:443 "POST /cgi/op.cgi HTTP/1.1" 200 132023-10-04 09:36:43,785Z DEBUG SMCGI Resp: text=VMCOMCODE=0012023-10-04 09:36:43,785Z DEBUG SMCGI Req: cgi=op, op=vm_status, args={}2023-10-04 09:36:43,910Z DEBUG https://10.5.86.41:443 "POST /cgi/op.cgi HTTP/1.1" 200 1552023-10-04 09:36:43,910Z DEBUG SMCGI Resp: text=

2023-10-04 09:36:44,926Z DEBUG SMCGI Req: cgi=op, op=vm_status, args={}2023-10-04 09:36:44,957Z DEBUG https://10.5.86.41:443 "POST /cgi/op.cgi HTTP/1.1" 200 1532023-10-04 09:36:44,957Z DEBUG SMCGI Resp: text=2023-10-04 09:36:44,957Z INFO Setting cdrom as boot device for next boot2023-10-04 09:36:44,987Z INFO Next boot device is set to optical2023-10-04 09:36:45,003Z INFO Power status is off2023-10-04 09:36:45,003Z INFO Powering up node2023-10-04 09:36:51,119Z INFO Sleeping for 40 seconds2023-10-04 09:37:31,154Z INFO Power status is on2023-10-04 09:37:31,154Z INFO BMC should be booting into phoenix2023-10-04 09:37:31,154Z INFO Waiting for remote node to boot into phoenix, this may take 20 minutes2023-10-04 09:57:52,825Z ERROR Exception on executing cmd: ['test', '-f', '/usr/bin/layout_finder.py']Traceback (most recent call last):  File "foundation\foundation_tools.py", line 239, in ssh  File "foundation\foundation_tools.py", line 215, in get_ssh_client  File "site-packages\paramiko\client.py", line 349, in connect  File "site-packages\paramiko\util.py", line 283, in retry_on_signal  File "site-packages\paramiko\client.py", line 349, in   File "socket.py", line 228, in metherror: [Errno 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond2023-10-04 09:57:52,825Z WARNING Unable to reach 10.5.86.42 over SSH or it is not in phoenix2023-10-04 09:57:52,825Z INFO Retrying booting into phoenix2023-10-04 09:57:52,825Z INFO Resetting BMC2023-10-04 09:58:02,851Z DEBUG [0/300s] Waiting for BMC2023-10-04 09:58:07,900Z DEBUG [5/300s] Waiting for BMC2023-10-04 09:58:12,947Z DEBUG [10/300s] Waiting for BMC2023-10-04 09:58:17,980Z DEBUG [15/300s] Waiting for BMC2023-10-04 09:58:23,032Z DEBUG [20/300s] Waiting for BMC2023-10-04 09:58:28,078Z DEBUG [25/300s] Waiting for BMC2023-10-04 09:58:33,125Z DEBUG [30/300s] Waiting for BMC2023-10-04 09:58:38,178Z DEBUG [35/300s] Waiting for BMC2023-10-04 09:58:43,237Z DEBUG [40/300s] Waiting for BMC2023-10-04 09:58:48,308Z DEBUG [45/300s] Waiting for BMC2023-10-04 09:58:53,391Z DEBUG [50/300s] Waiting for BMC2023-10-04 09:58:58,473Z DEBUG [55/300s] Waiting for BMC2023-10-04 09:59:03,522Z DEBUG [60/300s] Waiting for BMC2023-10-04 09:59:08,555Z DEBUG [65/300s] Waiting for BMC2023-10-04 09:59:13,604Z DEBUG [70/300s] Waiting for BMC2023-10-04 09:59:18,665Z DEBUG [75/300s] Waiting for BMC2023-10-04 09:59:23,717Z DEBUG [80/300s] Waiting for BMC2023-10-04 09:59:28,767Z DEBUG [85/300s] Waiting for BMC2023-10-04 09:59:33,813Z DEBUG [90/300s] Waiting for BMC2023-10-04 09:59:38,880Z DEBUG [96/300s] Waiting for BMC2023-10-04 09:59:43,927Z DEBUG [101/300s] Waiting for BMC2023-10-04 09:59:48,976Z DEBUG [106/300s] Waiting for BMC2023-10-04 09:59:54,025Z DEBUG [111/300s] Waiting for BMC2023-10-04 09:59:59,059Z DEBUG [116/300s] Waiting for BMC2023-10-04 10:00:04,108Z DEBUG [121/300s] Waiting for BMC2023-10-04 10:00:09,155Z DEBUG [126/300s] Waiting for BMC2023-10-04 10:00:14,203Z DEBUG [131/300s] Waiting for BMC2023-10-04 10:00:19,253Z DEBUG [136/300s] Waiting for BMC2023-10-04 10:00:24,302Z DEBUG [141/300s] Waiting for BMC2023-10-04 10:00:29,351Z DEBUG [146/300s] Waiting for BMC2023-10-04 10:00:34,384Z DEBUG [151/300s] Waiting for BMC2023-10-04 10:00:39,433Z DEBUG [156/300s] Waiting for BMC2023-10-04 10:00:44,480Z DEBUG [161/300s] Waiting for BMC2023-10-04 10:00:49,515Z DEBUG [166/300s] Waiting for BMC2023-10-04 10:00:54,563Z DEBUG [171/300s] Waiting for BMC2023-10-04 10:00:59,612Z DEBUG [176/300s] Waiting for BMC2023-10-04 10:01:04,645Z DEBUG [181/300s] Waiting for BMC2023-10-04 10:01:09,678Z DEBUG [186/300s] Waiting for BMC2023-10-04 10:01:14,727Z DEBUG [191/300s] Waiting for BMC2023-10-04 10:01:19,775Z DEBUG [196/300s] Waiting for BMC2023-10-04 10:01:24,825Z DEBUG [201/300s] Waiting for BMC2023-10-04 10:01:29,874Z DEBUG [207/300s] Waiting for BMC2023-10-04 10:01:34,921Z DEBUG [212/300s] Waiting for BMC2023-10-04 10:01:39,970Z DEBUG [217/300s] Waiting for BMC2023-10-04 10:01:45,017Z DEBUG [222/300s] Waiting for BMC2023-10-04 10:01:50,065Z DEBUG [227/300s] Waiting for BMC2023-10-04 10:01:55,114Z DEBUG [232/300s] Waiting for BMC2023-10-04 10:02:00,164Z DEBUG [237/300s] Waiting for BMC2023-10-04 10:02:05,211Z DEBUG [242/300s] Waiting for BMC2023-10-04 10:02:10,260Z DEBUG [247/300s] Waiting for BMC2023-10-04 10:02:15,309Z DEBUG [252/300s] Waiting for BMC2023-10-04 10:02:20,372Z DEBUG [257/300s] Waiting for BMC2023-10-04 10:02:25,421Z DEBUG [262/300s] Waiting for BMC2023-10-04 10:02:30,469Z DEBUG [267/300s] Waiting for BMC2023-10-04 10:02:35,500Z DEBUG [272/300s] Waiting for BMC2023-10-04 10:02:40,548Z DEBUG [277/300s] Waiting for BMC2023-10-04 10:02:45,596Z DEBUG [282/300s] Waiting for BMC2023-10-04 10:02:50,644Z DEBUG [287/300s] Waiting for BMC2023-10-04 10:02:55,694Z DEBUG [292/300s] Waiting for BMC2023-10-04 10:03:00,743Z DEBUG [297/300s] Waiting for BMC2023-10-04 10:03:05,790Z INFO Umounting Virtual CDROM using SMCGI2023-10-04 10:03:05,790Z DEBUG SMCGI Req: cgi=op, op=umount_iso, args={}2023-10-04 10:03:05,790Z DEBUG Resetting dropped connection: 10.5.86.412023-10-04 10:03:05,931Z DEBUG https://10.5.86.41:443 "POST /cgi/op.cgi HTTP/1.1" 200 8932023-10-04 10:03:05,931Z DEBUG SMCGI Resp: text=

ahmad.jamali · Answer

I know the post is from last one year.you have “2023-10-04 09:36:21,009Z ERROR Command 'ipmitool -I lanplus -H 10.5.86.41 -U ADMIN -P <obfuscated> fru' returned error code -1”which I think is due to wrong password on IPMI password.

Reply

Sign up

Login to the community

Scanning file for viruses.

This file cannot be downloaded