Hello Nutanix Community,
I am running a 3-node Nutanix Community Edition lab cluster with AOS 6.8.1 and I am currently troubleshooting an LCM Framework AutoUpdate that appears to have been interrupted on one CVM.
Prism Element continuously reports:
LCM Framework Update in Progress. Please check back when the update process is completed.
An LCM Inventory cannot be scheduled because Prism reports:
Changing configuration is not allowed while LCM operations in progress
However, I have verified that there is currently no active LCM Ergon root task and no active LCM inventory/framework-update worker process.
The current LCM framework versions are:
CVM 10.44.0.110 : 3.0.48302 <-- affected CVM
CVM 10.44.0.120 : 3.3.74044
CVM 10.44.0.130 : 3.3.74044
ZooKeeper shows the cluster LCM update intent as:
/appliance/logical/lcm/update
3.3.74044
Per-node LCM state:
2fd57079-00b8-41b5-bac7-259f949e766b -> 3.0.48302
1e342ac1-ade6-4937-a50b-20ff947026d2 -> 3.3.74044
830f3d4a-d063-4ddd-bc67-629aca33b3a3 -> 3.3.74044
The current LCM remote version is already newer:
/appliance/logical/lcm/remote_version
3.4.0.1.90933
Therefore, my intention is not to upgrade to 3.4.0.1.90933 yet. I first want to safely complete/recover the interrupted framework synchronization of the affected CVM from 3.0.48302 to the existing cluster intent 3.3.74044.
genesis.out continuously reports:
Node 2fd57079-00b8-41b5-bac7-259f949e766b not updated yet.
Local version: 3.0.48302, intended version 3.3.74044
Framework update is in progress. Returning Autoupdate
This is still occurring continuously, even though no actual LCM update task is running.
Additional checks already performed:
- LCM leader is CVM 10.44.0.120.
- No active LCM Ergon root task.
- No active LCM framework/inventory worker.
- Restarting Genesis on the affected CVM did not resolve the version mismatch.
- Mercury configuration exists across all three CVMs.
- The affected CVM can see the native LCM catalog entries for both:
- nutanix.lcm.lcm_update
- nutanix.lcm.lcm_3.3.74044
- The affected CVM still physically contains the older framework/SDK:
nutanix_lcm_framework-3.0.48302
nutanix_lcm_dep_lcm_sdk-2.1.0.1287
while the other two CVMs are running framework 3.3.74044.
I reviewed KB18248 – “LCM auto framework upgrade stuck on a node within the cluster”, which appears very similar to this condition. The KB recommends contacting Nutanix Support to update the framework on the affected CVM.
I also reviewed KB17820, but that KB concerns a Dark Site framework downgrade scenario, which does not apply here.
I have not manually modified any ZooKeeper nodes, framework version files, LCM wheels, schema/IDF data, or manually forced the internal FrameworkUpdater.
Before making any unsupported changes, I would like to ask:
Is there a recomm
Hello Nutanix Community,
I am running a 3-node Nutanix Community Edition lab cluster with AOS 6.8.1 and I am currently troubleshooting an LCM Framework AutoUpdate that appears to have been interrupted on one CVM.
Prism Element continuously reports:
LCM Framework Update in Progress. Please check back when the update process is completed.
An LCM Inventory cannot be scheduled because Prism reports:
Changing configuration is not allowed while LCM operations in progress
However, I have verified that there is currently no active LCM Ergon root task and no active LCM inventory/framework-update worker process.
The current LCM framework versions are:
CVM 10.44.0.110 : 3.0.48302 <-- affected CVM
CVM 10.44.0.120 : 3.3.74044
CVM 10.44.0.130 : 3.3.74044
ZooKeeper shows the cluster LCM update intent as:
/appliance/logical/lcm/update
3.3.74044
Per-node LCM state:
2fd57079-00b8-41b5-bac7-259f949e766b -> 3.0.48302
1e342ac1-ade6-4937-a50b-20ff947026d2 -> 3.3.74044
830f3d4a-d063-4ddd-bc67-629aca33b3a3 -> 3.3.74044
The current LCM remote version is already newer:
/appliance/logical/lcm/remote_version
3.4.0.1.90933
Therefore, my intention is not to upgrade to 3.4.0.1.90933 yet. I first want to safely complete/recover the interrupted framework synchronization of the affected CVM from 3.0.48302 to the existing cluster intent 3.3.74044.
genesis.out continuously reports:
Node 2fd57079-00b8-41b5-bac7-259f949e766b not updated yet.
Local version: 3.0.48302, intended version 3.3.74044
Framework update is in progress. Returning Autoupdate
This is still occurring continuously, even though no actual LCM update task is running.
Additional checks already performed:
- LCM leader is CVM 10.44.0.120.
- No active LCM Ergon root task.
- No active LCM framework/inventory worker.
- Restarting Genesis on the affected CVM did not resolve the version mismatch.
- Mercury configuration exists across all three CVMs.
- The affected CVM can see the native LCM catalog entries for both:
- nutanix.lcm.lcm_update
- nutanix.lcm.lcm_3.3.74044
- The affected CVM still physically contains the older framework/SDK:
nutanix_lcm_framework-3.0.48302
nutanix_lcm_dep_lcm_sdk-2.1.0.1287
while the other two CVMs are running framework 3.3.74044.
I reviewed KB18248 – “LCM auto framework upgrade stuck on a node within the cluster”, which appears very similar to this condition. The KB recommends contacting Nutanix Support to update the framework on the affected CVM.
I also reviewed KB17820, but that KB concerns a Dark Site framework downgrade scenario, which does not apply here.
I have not manually modified any ZooKeeper nodes, framework version files, LCM wheels, schema/IDF data, or manually forced the internal FrameworkUpdater.
Before making any unsupported changes, I would like to ask:
Is there a recommended recovery procedure for Nutanix CE to safely bring only the affected CVM from LCM framework 3.0.48302 to the existing cluster intent 3.3.74044?
Specifically, is there a supported/native method to resume or re-trigger the incomplete framework AutoUpdate for that CVM without manually modifying ZooKeeper or framework files?
Any guidance from someone familiar with KB18248 or LCM FrameworkUpdater recovery would be greatly appreciated.
Thank you.
ended recovery procedure for Nutanix CE to safely bring only the affected CVM from LCM framework 3.0.48302 to the existing cluster intent 3.3.74044?
Specifically, is there a supported/native method to resume or re-trigger the incomplete framework AutoUpdate for that CVM without manually modifying ZooKeeper or framework files?
Any guidance from someone familiar with KB18248 or LCM FrameworkUpdater recovery would be greatly appreciated.
Thank you.