Solved

CVM RAM Usage High after upgrade to 5.0.2

  • 24 April 2017
  • 9 replies
  • 12017 views

Badge +2
Hi all.
I have just (12 hours ago) upgraded our Nutanix Cluster to version 5.0.2 (from 4.7.3). After the uppgrade the health says ”Critical” because of ”CVM RAM Usage High”
We did not have that issue before the upgrade. It seams that the new firmware version is consuming more memory? Or?

Interesting thing is, is it look like memory consumtion is pretty much the same as before. When a look at the CVMs in vsphere client it look like the only using 50% - 60%.

Any ideas?
Thanks

/Kenneth
icon

Best answer by KennethH 16 May 2017, 14:38

Hi dlink7 and aluciani.

Do not think it is the hypervisor, because it is the same version as before upgrade, only thing changed was the upgrade to 5.0.2

My CVM:s only had 16 Gb or RAM and since v 5 the recommended amount is 32 Gb aparently. I have increased to 32 Gb so the error is not there any more.

But the general feeling is that the upgrade did not do any good for my cluster, only I higher memory consumtion. But it is not an big issue right now, I need to add a node soon anyway to the cluster.
View original

9 replies

Userlevel 4
Badge +21
Hi KennethH

Why hypervisor? I noticed that VMware had a bug out for high memory usage a couple of weeks ago. Not sure if its related.
Userlevel 7
Badge +35
Hi KennethH

Wondering if you saw the post from dlink7 - Thanks
Badge +2
Hi dlink7 and aluciani.

Do not think it is the hypervisor, because it is the same version as before upgrade, only thing changed was the upgrade to 5.0.2

My CVM:s only had 16 Gb or RAM and since v 5 the recommended amount is 32 Gb aparently. I have increased to 32 Gb so the error is not there any more.

But the general feeling is that the upgrade did not do any good for my cluster, only I higher memory consumtion. But it is not an big issue right now, I need to add a node soon anyway to the cluster.
Badge
I'm in a similar situation currently, although mine are running with 24 gb of ram. I already called support on this and they didn't have a solution other than moving resources around. Moving to 32 gb might be a solution, but it's only recommended for post-dudupe systems from what I have read. I guess I'm going to lose another 24 gb of ram. Thats quite a bit of CVM over head... 13% of my cluster memory for storage.
Badge +2
I've been having a similar issue starting with 5.0.1 through 5.0.2. I'm using AHV and have always had my CVMs at 32GB. I bumped one to 34 to see if it made a difference but after a few days, I got the alert on it as well. Same as with high CPU usage on CVMs.
Badge
So we have a systemic issue here then. I guess its time to get back on with support.
Badge +9
Hi,

Did you guys also upgrade NCC when doing this?

In the newest NCCs, the Threshold values have been changed for High RAM Usage, which triggers the alert.
I believe this has been changed from 500MB to 1GB.

The same goes for the High CPU Usage.
With that said, these alerts could most probably be false positives, have you seen any services that have been killed due to low RAM?

Kind Regards
Badge
Nothing has been killed so far, just a very annoying "red alert". I'm running NCC version 3.0.1.1, which I believe is the newest. My alert is triggering as follows:
Main memory usage in Controller VM 172.28.1.63 is high. 998576 KB of memory is free.
Main memory usage in Controller VM 172.28.1.61 is high. 752012 KB of memory is free.

If the new alert threshold is 1gb, that would explain the alert... but not the cause.
Badge +9
The "cause" has always been there, as the RAM usually goes up and down on a normal basis, such as when curator scans are running.
Earlier, the alert got triggered at 500MB, and they have raised it a bit to aggresive to 1GB which causes false positives. Most probably the cluster is behaving just like it did before the upgrade, but now you are actually getting notified due to a more aggresive threshold value.

In later releases the Service Leaders will be balanced better which causes better RAM Utilization over the board. As it is right now, most leader roles are all on the same CVM, which then gets this alert when cluster tasks are running.

As long as the services are running fine without restart you are in the clear, but if you are uncertain open up a case with support and they can help you with moving roles around, aswell as fine-tune the alert thresholds.

Kind Regards

Reply