Hi Rob,

I hope you're well. I'm pretty sure I've used this same ecli to mark LCM tasks aborted (LCM inventory tasks that just sat forever). I'm not sure if it'll help with the issue, but I do think it'd make the logs more interesting from a fresh run.

Might also be interesting to see how Steve fixed similar.

https://next.nutanix.com/how-it-works-22/lcm-framework-stuck-updating-3927

Though yours is talking about not being able to find the uuid, I wonder if you can find the uuid from the earlier logs?

Thanks for the reply - thats exactly one of the previous discussions i was looking at to help fix this. Not many the same issue i think but similar.

Do you know what might be missing the uuid? a config file used by a service or some other field or list?

I'm not clear on the logs exactly what component it is  ‘Unable to find node_uuid for **node3** ‘

I'll see about ending the tasks – may take some time to organize to make sure interested parties are all ok with my changes.

Hey,

I’m not sure but I see the uuid from ncli host ls in my zookeeper.out log.

Might be worth looking in ncli host ls and confirming you have a uuid for each host first, then have a grep through the zookeeper logs to see if any of those show up - or better what format of the uuid for the other hosts you have in there?

Interesting update

nutanix@NTNX-0fcb0e3e-A-CVM:X.Y.Z.238:~$ acli host.list
#Hypervisor IP  Hypervisor DNS Name  Host UUID                             Compute Only  Schedulable  Hypervisor Type  Hypervisor Name  CVM IP
X.Y.Z.247   X.Y.Z.247         d4c6a6b9-ad84-429f-8ac7-d4b671e5c1e7  False         True         kKvm             AHV              X.Y.Z.237
X.Y.Z.248   X.Y.Z.248         1f918e7b-b20d-4608-aa9d-e282ada101bd  False         True         kKvm             AHV              X.Y.Z.238
X.Y.Z.249   X.Y.Z.249         050e9777-d7b3-4957-a1a0-3f6a060611ac  False         True         kKvm             AHV              X.Y.Z.239   ****problem node****

nutanix@NTNX-7e11bd65-A-CVM:X.Y.Z.239:~$ zkcli ls /appliance/logical/lcm/mercury_config
1f918e7b-b20d-4608-aa9d-e282ada101bd
d4c6a6b9-ad84-429f-8ac7-d4b671e5c1e7

Seems that uuid is missing from here - 050e9777-d7b3-4957-a1a0-3f6a060611ac

Not clear what these fiels actually do to help
nutanix@NTNX-7e11bd65-A-CVM:X.Y.Z.239:~$ zkcli cat /appliance/logical/lcm/mercury_config/1f918e7b-b20d-4608-aa9d-e282ada101bd
1.3
nutanix@NTNX-7e11bd65-A-CVM:X.Y.Z.239:~$ zkcli cat /appliance/logical/lcm/mercury_config/d4c6a6b9-ad84-429f-8ac7-d4b671e5c1e7
1.3

suspect i need to     zkcli write /appliance/logical/lcm/mercury_config/050e9777-d7b3-4957-a1a0-3f6a060611ac

Also does anyone know if runing ‘genesis restart ‘ on the current leader will cause issues with workload?  I presume i may need to do this to trigger the leader to switch. If there is any doubt i can do a manual node restart from cli.

I’d agree on wanting to write that in similar to Steve’s. Spinning genesis should be fine, it is one of the first things I do whenever I’m troubleshooting something because Genesis knows all ;)

Update.

I created the missing file using zkcli cat /appliance/logical/lcm/mercury_config/050e9777-d7b3-4957a0-3f6a060611ac

I then rebooted every node sequencially using maintenance mode and cli

No change in lcm or reboot processes stuck

Tried genesis restart command - no effect

I then ran

ergon_update_task --task_uuid='XXXXXXXXXXXX' --task_status=succeeded or (aborted)

aganst the affected tasks - now tasks are clear, but still LCM framework page is stuck with message:

‘ LCM Framework Update in Progress, please check back when the update process is completed. ‘

‘ Error calling Entities : Error: Not Found ‘

I then ran another set of restarts on nodes

The 2 nodes with lcm framework up to date have this output for their uuids
nutanix@NTNX-e0c1425c-A-CVM:X.Y.Z.237:~/data/logs$ allssh zkcli cat /appliance/logical/lcm/mercury_config/d4c6a6b9-ad84-429f-8ac7-d4b671e5c1e7
================== X.Y.Z.238 =================
1.3================== X.Y.Z.239 =================
1.3================== X.Y.Z.237 =================
1.3

Fro the new one i created using zcli write uuids has no value
nutanix@NTNX-e0c1425c-A-CVM:X.Y.Z.237:~/data/logs$ allssh zkcli cat /appliance/logical/lcm/mercury_config/050e9777-d7b3-4957a0-3f6a060611ac
================== X.Y.Z.238 =================
================== X.Y.Z.239 =================
================== X.Y.Z.237 =================

Does anyone know the significance of the 1.3 value?

I checked it against the value in a different fully supported nutanix cluster run nx hardware and it has value true in these files

Hmm, I'm not sure sorry.

I wonder if it's worth writing 1.3 on the end for consistency anyway and see if lcm changes at all.

Totally guessing now though I'm afraid!

Support asked if i could remove the commands for  'ergon_update_task' from my post.

I dont think i can do this(i cant edit anything) - im leaving a note for others that you shouldnt do this without direct interaction with Nutanix Support.

I think they want to avoid user's doing this as a normal operation or troubleshooting process.

As an update - no further with fixing the LCM - still issues with updating LCM.

The cluster seems to be running with ok health(recently recovered from failed state due to hosting issues)

Prism reports slightly different error now:

Operation failed. Reason: Prechecks failed: Found remote version None while 2.5.0.3.33256 was expected on node 10.15.17.239. This may be a caching issue. Please ensure all local caches are cleared and wait a few minutes for any remote caches to get invalidated before retrying. Please check KB 7784

This seems related to the LCM version mismatch between nodes found in cvm commands.

I think it might be an idea to remove the node from the cluster completely and re-add - from what i can tell i think i need 4 nodes to do this operation via this process https://portal.nutanix.com/page/documents/details?targetId=Web-Console-Guide-Prism-v6_5:wc-removing-node-pc-c.html

Did you ever resolve this?

Hi,

No sorry, i couldnt fix the lcm error directly from CLI - I still have a plan to add a new a new node temporarily and delete and re-add the (broken LCM) node, but havent got around to this yet.

I’ve just got off a call with Nutanix support, and we ended up fixing it by deploying the new version of LCM Framework via CLI on the LCM_Leader CVM