Question

Nutanix CE – LCM Framework AutoUpdate stuck on one CVM with mixed framework versions

Forum|Forum|3 days ago
September 15, 2026
9 replies
193 views

FrankoRojas
Adventurer

Hello Nutanix Community,

I am running a 3-node Nutanix Community Edition lab cluster with AOS 6.8.1 and I am currently troubleshooting an LCM Framework AutoUpdate that appears to have been interrupted on one CVM.

Prism Element continuously reports:

LCM Framework Update in Progress. Please check back when the update process is completed.

An LCM Inventory cannot be scheduled because Prism reports:

Changing configuration is not allowed while LCM operations in progress

However, I have verified that there is currently no active LCM Ergon root task and no active LCM inventory/framework-update worker process.

The current LCM framework versions are:

CVM 10.44.0.110 : 3.0.48302 <-- affected CVM

CVM 10.44.0.120 : 3.3.74044

CVM 10.44.0.130 : 3.3.74044

ZooKeeper shows the cluster LCM update intent as:

/appliance/logical/lcm/update

3.3.74044

Per-node LCM state:

2fd57079-00b8-41b5-bac7-259f949e766b -> 3.0.48302

1e342ac1-ade6-4937-a50b-20ff947026d2 -> 3.3.74044

830f3d4a-d063-4ddd-bc67-629aca33b3a3 -> 3.3.74044

The current LCM remote version is already newer:

/appliance/logical/lcm/remote_version

3.4.0.1.90933

Therefore, my intention is not to upgrade to 3.4.0.1.90933 yet. I first want to safely complete/recover the interrupted framework synchronization of the affected CVM from 3.0.48302 to the existing cluster intent 3.3.74044.

genesis.out continuously reports:

Node 2fd57079-00b8-41b5-bac7-259f949e766b not updated yet.

Local version: 3.0.48302, intended version 3.3.74044

Framework update is in progress. Returning Autoupdate

This is still occurring continuously, even though no actual LCM update task is running.

Additional checks already performed:

LCM leader is CVM 10.44.0.120.
No active LCM Ergon root task.
No active LCM framework/inventory worker.
Restarting Genesis on the affected CVM did not resolve the version mismatch.
Mercury configuration exists across all three CVMs.
The affected CVM can see the native LCM catalog entries for both:
- nutanix.lcm.lcm_update
- nutanix.lcm.lcm_3.3.74044
The affected CVM still physically contains the older framework/SDK:

nutanix_lcm_framework-3.0.48302

nutanix_lcm_dep_lcm_sdk-2.1.0.1287

while the other two CVMs are running framework 3.3.74044.

I reviewed KB18248 – “LCM auto framework upgrade stuck on a node within the cluster”, which appears very similar to this condition. The KB recommends contacting Nutanix Support to update the framework on the affected CVM.

I also reviewed KB17820, but that KB concerns a Dark Site framework downgrade scenario, which does not apply here.

I have not manually modified any ZooKeeper nodes, framework version files, LCM wheels, schema/IDF data, or manually forced the internal FrameworkUpdater.

Before making any unsupported changes, I would like to ask:

Is there a recomm

Hello Nutanix Community,

I am running a 3-node Nutanix Community Edition lab cluster with AOS 6.8.1 and I am currently troubleshooting an LCM Framework AutoUpdate that appears to have been interrupted on one CVM.

Prism Element continuously reports:

LCM Framework Update in Progress. Please check back when the update process is completed.

An LCM Inventory cannot be scheduled because Prism reports:

Changing configuration is not allowed while LCM operations in progress

However, I have verified that there is currently no active LCM Ergon root task and no active LCM inventory/framework-update worker process.

The current LCM framework versions are:

CVM 10.44.0.110 : 3.0.48302 <-- affected CVM

CVM 10.44.0.120 : 3.3.74044

CVM 10.44.0.130 : 3.3.74044

ZooKeeper shows the cluster LCM update intent as:

/appliance/logical/lcm/update

3.3.74044

Per-node LCM state:

2fd57079-00b8-41b5-bac7-259f949e766b -> 3.0.48302

1e342ac1-ade6-4937-a50b-20ff947026d2 -> 3.3.74044

830f3d4a-d063-4ddd-bc67-629aca33b3a3 -> 3.3.74044

The current LCM remote version is already newer:

/appliance/logical/lcm/remote_version

3.4.0.1.90933

genesis.out continuously reports:

Node 2fd57079-00b8-41b5-bac7-259f949e766b not updated yet.

Local version: 3.0.48302, intended version 3.3.74044

Framework update is in progress. Returning Autoupdate

This is still occurring continuously, even though no actual LCM update task is running.

Additional checks already performed:

LCM leader is CVM 10.44.0.120.
No active LCM Ergon root task.
No active LCM framework/inventory worker.
Restarting Genesis on the affected CVM did not resolve the version mismatch.
Mercury configuration exists across all three CVMs.
The affected CVM can see the native LCM catalog entries for both:
- nutanix.lcm.lcm_update
- nutanix.lcm.lcm_3.3.74044
The affected CVM still physically contains the older framework/SDK:

nutanix_lcm_framework-3.0.48302

nutanix_lcm_dep_lcm_sdk-2.1.0.1287

while the other two CVMs are running framework 3.3.74044.

I also reviewed KB17820, but that KB concerns a Dark Site framework downgrade scenario, which does not apply here.

I have not manually modified any ZooKeeper nodes, framework version files, LCM wheels, schema/IDF data, or manually forced the internal FrameworkUpdater.

Before making any unsupported changes, I would like to ask:

Is there a recommended recovery procedure for Nutanix CE to safely bring only the affected CVM from LCM framework 3.0.48302 to the existing cluster intent 3.3.74044?

Specifically, is there a supported/native method to resume or re-trigger the incomplete framework AutoUpdate for that CVM without manually modifying ZooKeeper or framework files?

Any guidance from someone familiar with KB18248 or LCM FrameworkUpdater recovery would be greatly appreciated.

Thank you.

ended recovery procedure for Nutanix CE to safely bring only the affected CVM from LCM framework 3.0.48302 to the existing cluster intent 3.3.74044?

Specifically, is there a supported/native method to resume or re-trigger the incomplete framework AutoUpdate for that CVM without manually modifying ZooKeeper or framework files?

Any guidance from someone familiar with KB18248 or LCM FrameworkUpdater recovery would be greatly appreciated.

Thank you.

JeroenTielen
Vanguard
Forum|Forum|2 days ago
September 16, 2026

Can you have a look at these two topics:

https://next.nutanix.com/how-it-works-22/lcm-framework-stuck-updating-39273

https://next.nutanix.com/how-it-works-22/lcm-stuck-with-framework-update-41247

Nutanix NTC 2022+2023+2024+2025+2026 || Community Excellence Award 2025+2026 || Nutanix NUG Champion || Nutanix Certified Instructor

jamali.ahmad
Vanguard
Forum|Forum|22 hours ago
September 18, 2026

would you go to the leader node and let us know what you see on the genesis.out log ?
I would definitely look at two links which are shared with @JeroenTielen .

FrankoRojas
Author
Adventurer
Forum|Forum|20 hours ago
September 18, 2026

Hi Jamali, thank you for the suggestion.

I checked the current genesis.out on the LCM leader CVM (10.44.0.120).

The leader is repeatedly detecting the affected CVM UUID as still running the old framework version:

I (10.44.0.120) am the LCM leader

Node 2fd57079-00b8-41b5-bac7-259f949e766b not updated yet.
Local version: 3.0.48302, intended version 3.3.74044

Currently updating LCM framework. Waiting for it to finish

Framework update is in progress. Returning Autoupdate

Waiting 10 seconds for completion of LCM autoupdate in progress…

Genesis is also waiting for the schema update:

Schema version not updated to 1.73. Assuming LCM update is in progress
Schema update is in progress. Waiting for schema update to finish before updating IDF table.

The affected UUID 2fd57079-00b8-41b5-bac7-259f949e766b corresponds to CVM 10.44.0.110, which is still running LCM Framework 3.0.48302. The other two CVMs (10.44.0.120 and 10.44.0.130) are already on 3.3.74044.

I also searched the current leader genesis.out for check_my_url_connectivity, but that signature is not present in the current log.

There is currently no active LCM root task/worker that I can identify, while Genesis continuously considers the framework AutoUpdate to be in progress because .110 has not reached the intended version.

I reviewed the two Community cases referenced earlier. In my cluster, the Mercury configuration already exists for all three CVM UUIDs, so I have intentionally not modified ZooKeeper or Ergon state.

Would you recommend a supported/safe way to re-trigger or deploy the intended LCM Framework 3.3.74044 specifically to the affected CVM, without modifying ZooKeeper/Ergon state manually?

I can provide additional sanitized genesis.out excerpts if useful.

jamali.ahmad
Vanguard
Forum|Forum|19 hours ago
September 18, 2026

Have you tried to do :

Cvm$cluster restart_genesis

And then after afew minutes try to do inventory again and see the result?

FrankoRojas
Author
Adventurer
Forum|Forum|18 hours ago
September 18, 2026

Hi @jamali.ahmad ,

Yes, I already performed cluster restart_genesis earlier during the troubleshooting.

After Genesis restarted successfully, the condition persisted: CVM 10.44.0.110 remained on LCM Framework 3.0.48302, while 10.44.0.120 and 10.44.0.130 remained on 3.3.74044. The cluster intended framework version also remained 3.3.74044.

Genesis subsequently continued reporting:

Node 2fd57079-00b8-41b5-bac7-259f949e766b not updated yet.
Local version: 3.0.48302, intended version 3.3.74044

Framework update is in progress. Returning Autoupdate

I also previously attempted LCM Inventory, but Prism rejected the request with:

Failure to schedule the request due to Changing configuration is not allowed while LCM operations in progress

If you recommend it, I can repeat cluster restart_genesis now, wait a few minutes, and immediately retry Inventory to capture the current behavior/results. Would you like me to do that?

jamali.ahmad
Vanguard
Forum|Forum|15 hours ago
September 18, 2026

So when you running upgrade_status and lcm_upgrade_status it is not giving you anything is happening?

Is it possible to upload a copy of Genesis.out and lcm_ops.out from each leader (from Genesis leader and lcm leader?

When you running nice what is showing you?

FrankoRojas
Author
Adventurer
Forum|Forum|15 hours ago
September 18, 2026

Hi Jamali,

I checked both status commands and found an interesting state.

upgrade_status reports that all three SVMs are up to date with the target AOS 6.8.1 release:

SVM 10.44.0.110 is up to date
SVM 10.44.0.120 is up to date
SVM 10.44.0.130 is up to date

However, lcm_upgrade_status reports:

LCM autoupdate is in progress

Ongoing upgrades in current batch:
No upgrade is in progress

I also checked the LCM logs on the confirmed LCM leader 10.44.0.120.

There is no file named lcm_ops.out on this CVM. The LCM-related files available under /home/nutanix/data/logs include lcm_leader.log and lcm_op.trace.

The current lcm_op.trace is particularly interesting:

{"leader_ip": "10.44.0.120", "event": "Reseting the node bank"}

Getting information for node uuid 2fd57079-00b8-41b5-bac7-259f949e766b
Getting information for node uuid 1e342ac1-ade6-4937-a50b-20ff947026d2
Getting information for node uuid 830f3d4a-d063-4ddd-bc67-629aca33b3a3

Node 2fd57079-00b8-41b5-bac7-259f949e766b not updated yet.
Local version: 3.0.48302, intended version 3.3.74044

Framework update is in progress. Returning Autoupdate
No task found for kLcmUpdateOperation

This appears consistent with what Genesis is reporting: the framework AutoUpdate state remains active because CVM 10.44.0.110 has not reached the intended framework version, but LCM cannot find an active update operation.

Regarding the logs you requested, the current genesis.out is available, and I can provide a sanitized copy/excerpt. Since this version does not have a file named lcm_ops.out, would you like me to provide lcm_op.trace instead, or is there another specific LCM log/path you would like me to collect?

Also, regarding your question about nice, could you please provide the exact command/options you would like me to run so I can capture the specific output you are looking for?

jamali.ahmad
Vanguard
Forum|Forum|14 hours ago
September 18, 2026

Please ignore the last line of my last post.i myself dont know what i meant :)

Would you also run below command from any cvm:

ecli task.list include_completed=false operation_type_list=kLcmInventoryOperationnutanix@CVM~$ ecli task.list include_completed=false operation_type_list=kLcmUpdateOperation

BTW is this is a production or home lab?

When you are saying this version dont havr lcm_ops.out ,would you explain a bit about what you mean?

LMohammed
Outrider
Forum|Forum|12 minutes ago
September 19, 2026

Hi @FrankoRojas As of now we have :

Target AOS is healthy/up to date on all 3 CVMs.
Two CVMs have LCM framework 3.3.74044.
CVM .110 is still on 3.0.48302.
lcm_upgrade_status says AutoUpdate is in progress.
But lcm_op.trace says No task found for kLcmUpdateOperation.
Genesis is therefore repeatedly waiting for .110 to reach the intended version.

Just wonderin, have you performed the LCM through manually upgrade (dark-site) or Automatic (connected-site) please ?

Have you checked if there’s any dependencies and per-requisites for AOS/ahv before starting the upgrade?

Thank you!

3*NCA & 3*NCP-MCI & 2*NCP-US & NCS and Microsoft & Dell and VmWare certified

Sign up

Login to the community