CPU Resource Constraints & Overprovisioned VM's Reduce below 2 vCPU

Hi ​@GilesB 

https://download.nutanix.com/documentation/v11202/Release-Notes-AHV-v11_2_0_2.pdf
https://download.nutanix.com/documentation/v7606/Release-Notes-AOS-v7_6_0_6.pdf

Nothing significant related to cpu and ram peeks is mentioned in the known issues for the new versions.

What is the state of the cpu and ram of your cvms and nodes ?

Sorry for delay replying.

We’ve still been struggling.  Have moved about 50 workloads to a different cluster, and changed our protection policies to spread our the processor impact of VSS snapshots across a large number of workloads.

We have stabilsed and brough peak CPU ready times down from 20+% to <4% but from working with support it does appear to be related to changes in 7.6.  1) updtaed micro-services need more resources 2) updated scheduling means near millisecond syncronisity on tasks across large numbers of VM’s 

There’s some other optimisations we will look at moving forwards, but they need more business decisions and longer projects to re-arrange storage containers etc.

Graph attahced showing 3 hours from today, not CPU usage at bottom, and other graphs showing CPU ready across CVM’s and some groups of machines.  Patterns repeated across all machines groups on the cluster.
 

What security profiles are configured? Elevated and Strict can reduce cluster performance. 

Hello,

I have encountered a similar issue in my environment and would like to share the details to find out whether other administrators are experiencing the same behavior.

I have a Nutanix cluster consisting of 10 NX-8170-G9 nodes. After upgrading the environment to AHV 11.2 and AOS 7.6, I started the firmware upgrade process for some components. Unfortunately, during this process, one of the servers experienced a hardware failure and is currently undergoing an RMA replacement.

On the next business day, we noticed that the VMs were not being properly balanced across the cluster nodes. One node was hosting almost half of the cluster's VMs, with a load of approximately 300%, while the remaining nodes were operating at around 30% load.

We contacted Nutanix Support, which is still investigating the issue. During the troubleshooting process, the engineers placed the overloaded node into maintenance mode and rebooted it. However, after this procedure, the overload shifted to another node in the cluster.

As a temporary workaround to mitigate the impact, we created anti-affinity rules to prevent certain VMs from running on the overloaded node, allowing them to be distributed across the remaining nodes.

Support advised that a service responsible for VM metrics needs to be restarted. However, they cannot proceed with this action until the faulty hardware has been replaced, as the cluster is currently operating with only one node of fault tolerance.

At this point, we do not know whether restarting this service will resolve the issue, whether this is a potential bug in this AOS/AHV release, or whether another underlying problem is responsible. The investigation is still ongoing.

In the meantime, managing the environment under these conditions has become quite challenging, particularly due to the uneven VM distribution and the high load concentrated on individual nodes.

Is anyone experiencing similar behavior after upgrading to AOS 7.6 and AHV 11.2? The Acropolis Dynamic Scheduler (ADS) is expected to balance VM placement across the cluster nodes. Are there any known configuration requirements, limitations, or issues in this release that could explain this behavior?

I would appreciate any insights or experiences that could help clarify the issue.