Prism Central Pre-Upgrade check stuck at 30%

  • 18 March 2024
  • 2 replies

Badge +1

Prism Central Pre-Upgrade check stuck at 30%

Followed the given Steps to kill the running, hung, stuck task(s) 

Step 1: SSH to any Nutanix CVM of the Nutanix cluster ( Nutanix CVM default credentials )

Step 2: Get the running / stuck / hung task(s) list

cvm$ progress_monitor_cli --fetchall

Output would be like this:

    ================== Proto Start =========================    logical_timestamp: 10    progress_info_id {      operation: kDownload      entity_type: kPackage      entity_id: "xx.xx.tar.gz"    }    title_message: "Software Upload" ----> This is My Stuck / Hung task    start_time_secs: 1087597349    progress_task_list {      component: kPrism      task_tag: "xx.xx.tar.gz"      start_time_secs: 1087597349      percentage_complete: 10      progress_status: kRunning    }    =================== Proto End ==========================

now you have to collect following info from above command output.

  1. operation
  2. entity_type
  3. entity_id

Step 3: Now kill the stuck / hung task using following command

cvm$ progress_monitor_cli --entity_id="<Entity_ID>" --entity_type=<Package_Name> --operation=<Operation> -delete


cvm$ progress_monitor_cli --entity_id="xx.xx.tar.gz" --entity_type=package --operation=download -delete


-PCVM:~$ progress_monitor_cli --entity_id="3996104131303755246" --entity_type=kCluster --operation=kUpgrade -delete
2024-03-18 08:02:23,345Z:200457(0x7f7937b7c380):ZOO_INFO@zookeeper_init@994: Initiating client connection, host=zk3:9876,zk2:9876,zk1:9876 se           ssionTimeout=20000 watcher=0x7f7946c75b80 sessionId=0 sessionPasswd=<null> context=0x557cee490040 flags=0
2024-03-18 08:02:23,349Z:200457(0x7f79371ff700):ZOO_INFO@zookeeper_interest@1940: Connecting to server
2024-03-18 08:02:23,349Z:200457(0x7f79371ff700):ZOO_INFO@zookeeper_interest@1977: Zookeeper handle state changed to ZOO_CONNECTING_STATE for            socket []
2024-03-18 08:02:23,350Z:200457(0x7f79371ff700):ZOO_INFO@check_events@2185: initiated connection to server []
2024-03-18 08:02:23,353Z:200457(0x7f79371ff700):ZOO_INFO@check_events@2233: session establishment complete on server [], sess           ionId=0x38dcaf4421b8c91, negotiated timeout=20000
F20240318 08:02:23.354409Z 200457] Check failed: false Unknown operation: kUpgrade
*** Check failure stack trace: ***
First FATAL tid: 200457
Stack traces are generated at /home/nutanix/data/cores/progress_monito.200457.20240318-010224.stack_trace.txt





2 replies

Badge +1

Restarted Prism Central VMs, the pre upgrade task was killed successfully, however another task got initialized “Enabling microservices infrastructure” but not getting completed since 3 days. doesn't show anything in progress_monitor_cli --fetchall or ecli task.list. 


Badge +1


I am not seeing any response on this thread.

Update: -- I was able to ABORT the task successfully, but now there is a new error.

Let me know if anyone has a feedback.