Prism Central Pre-Upgrade check stuck at 30%
Followed the given Steps to kill the running, hung, stuck task(s)
Step 1: SSH to any Nutanix CVM of the Nutanix cluster ( Nutanix CVM default credentials )
Step 2: Get the running / stuck / hung task(s) list
cvm$ progress_monitor_cli --fetchall
Output would be like this:
================== Proto Start ========================= logical_timestamp: 10 progress_info_id { operation: kDownload entity_type: kPackage entity_id: "xx.xx.tar.gz" } title_message: "Software Upload" ----> This is My Stuck / Hung task start_time_secs: 1087597349 progress_task_list { component: kPrism task_tag: "xx.xx.tar.gz" start_time_secs: 1087597349 percentage_complete: 10 progress_status: kRunning } =================== Proto End ==========================now you have to collect following info from above command output.
- operation
- entity_type
- entity_id
Step 3: Now kill the stuck / hung task using following command
cvm$ progress_monitor_cli --entity_id="<Entity_ID>" --entity_type=<Package_Name> --operation=<Operation> -delete
Example:
cvm$ progress_monitor_cli --entity_id="xx.xx.tar.gz" --entity_type=package --operation=download -delete
RESULT.
-PCVM:~$ progress_monitor_cli --entity_id="3996104131303755246" --entity_type=kCluster --operation=kUpgrade -delete
2024-03-18 08:02:23,345Z:200457(0x7f7937b7c380):ZOO_INFO@zookeeper_init@994: Initiating client connection, host=zk3:9876,zk2:9876,zk1:9876 se ssionTimeout=20000 watcher=0x7f7946c75b80 sessionId=0 sessionPasswd=<null> context=0x557cee490040 flags=0
2024-03-18 08:02:23,349Z:200457(0x7f79371ff700):ZOO_INFO@zookeeper_interest@1940: Connecting to server 10.1.62.226:9876
2024-03-18 08:02:23,349Z:200457(0x7f79371ff700):ZOO_INFO@zookeeper_interest@1977: Zookeeper handle state changed to ZOO_CONNECTING_STATE for socket [10.1.62.226:9876]
2024-03-18 08:02:23,350Z:200457(0x7f79371ff700):ZOO_INFO@check_events@2185: initiated connection to server [10.1.62.226:9876]
2024-03-18 08:02:23,353Z:200457(0x7f79371ff700):ZOO_INFO@check_events@2233: session establishment complete on server [10.1.62.226:9876], sess ionId=0x38dcaf4421b8c91, negotiated timeout=20000
F20240318 08:02:23.354409Z 200457 progress_monitor_cli.cc:353] Check failed: false Unknown operation: kUpgrade
*** Check failure stack trace: ***
First FATAL tid: 200457
Stack traces are generated at /home/nutanix/data/cores/progress_monito.200457.20240318-010224.stack_trace.txt
