Skip to main content

Prism Central (PC) VM is running out of space

  • 22 September 2020
  • 9 replies
  • 7452 views

Similarly to a CVM, Prism Central VM can run out of space.
There are three main contributors:

  • localhost_access_log.txt not being rotated properly
  • Nutanix Insights collector log file have consumed more disk space than expected on Prism Central and Controller VMs
  • multiple Nutanix Guest Tool installers are stored in the home directory of Prism Central

While the engineering team is working on improving the product behaviour, there are some things you can do about preventing a PC failure:

  • Log in to the PC VM.
  • Navigate to the /home partition
  • List the contents of the directory by size and look for any large unused files that can be deleted.

Some common sub-directories under /home where large files are likely to be found. Files in these locations can be deleted to free up space.
 
:exclamation:  Delete the files in the folder but NOT the folder itself.

  • /home/nutanix/software_downloads/ - Delete any old versions other than the versions you are currently upgrading
  • /home/nutanix/software_uncompressed/ - Delete any old versions other than the versions you are currently upgrading
  • /home/nutanix/data/cores - Delete old stack traces that are no longer needed
  • /home/nutanix/data/log_collector/ - Delete old NCC Logs with NCC-logs-2018-07-20-11111111111111-1032057545.tar format
  • /home/nutanix/data/logbay/bundles/ - Delete old NCC Logbay bundles
  • /home/nutanix/foundation/isos/ - Old ISOs
  • /home/nutanix/foundation/tmp/ - Temporary files that can be deleted

For more causes, resolution and commands read KB-8950 NCC Health Check: Prism Central VM <PCVM_IP> home partition disk usage exceeds critical limit 90 %.
For the same issue on the CVM see CVM Non-Root Partitions Space Usage High - clear space safely.

This topic has been closed for comments

9 replies

LionelS

Thanks for this topic!

Quick question, is it also possible on the PCVM to delete the content of /home/nutanix/data/installer/ as mentionned for the CVMs? 

I’ll otherwise open a case as advised on KB 5228.

Alona
Rank
Userlevel 6
Badge +5

Hi LionelS,

There should be only one the currently installed version in that location. Having that directory empty will cause the upgrade pre-checks to fail and would require remediation to allow for the upgrade to happen.

LionelS

Hi Alona,

Many thanks for your feedback. Exactly, there is only one version there. I’ll let it where it is. Good to know for the pre-checks.

B

Hi Alona, 

For my PCVM has also issue with free space on /home partition. 

The biggest folder is /data/logs. There are lot of log files under this folder. Is it possible to delete the old ones? 

nutanix@NTNX--A-PCVM:~$ sudo du -aSxh | sort -h | tail -n 10
353M    ./tmp
368M    ./data/logbay/taskdata/0e43a33c-81d3-4d89-80dd-e5eb6beb2281
368M    ./data/logbay/taskdata/0e43a33c-81d3-4d89-80dd-e5eb6beb2281/NTNX-Log-2020-12-04-1607088215-3239883081208344587-PC-10.225.4.8-CW.zip
371M    ./data/installer/el7.3-release-euphrates-5.16.1.3-stable-2d5fc1e3a7875f2fbf66400fdce3f0483b746eb1/pkg/nutanix-epsilon-2.9.8.tar.gz
565M    ./data/installer/el7.3-release-euphrates-5.16.1.3-stable-2d5fc1e3a7875f2fbf66400fdce3f0483b746eb1/images/svm.tar.xz
578M    ./data/installer/el7.3-release-euphrates-5.16.1.3-stable-2d5fc1e3a7875f2fbf66400fdce3f0483b746eb1/images
659M    ./data/installer/el7.3-release-euphrates-5.16.1.3-stable-2d5fc1e3a7875f2fbf66400fdce3f0483b746eb1/pkg/nutanix-core-el7.3-release-euphrates-5.16.1.3-stable-2d5fc1e3a7875f2fbf66400fdce3f0483b746eb1.tar.gz
3.4G    ./data/installer/el7.3-release-euphrates-5.16.1.3-stable-2d5fc1e3a7875f2fbf66400fdce3f0483b746eb1/pkg
4.1G    ./data/logs/sysstats
4.7G    ./data/logs
 

Is the scavenger service not working properly in my environment? Or scavenger service does not delete the old files in /data/logs folder?

 

Kaspars
Userlevel 1

This whole process is a joke. Customers shouldn't have to deal with this. The disk space is clearly undersized and there should be an automatic process which clears the disk space for next update. I have to clean all the Prism Central created garbage every time I want to upgrade the Prism Central and every time there is less disk space available even cleaning the directories mentioned above. If Prism Central is reserving 2500 GB disk space why this is an issue in first place???

Linux
Userlevel 1

This whole process is a joke. Customers shouldn't have to deal with this. The disk space is clearly undersized and there should be an automatic process which clears the disk space for next update. I have to clean all the Prism Central created garbage every time I want to upgrade the Prism Central and every time there is less disk space available even cleaning the directories mentioned above. If Prism Central is reserving 2500 GB disk space why this is an issue in first place???

Completely agree to this.

I have been forced to do this manually aswell for the last year. It should absolutely be possible to expand the disk-space without messing up the PC-VM. 
Add the possibility to store stuff on a Volume-Group please, so we dont have to deal with this.

GNU/Linux
J
Userlevel 4
Badge +12

I logged in just to say “I Agree”. We’ve had to deal with this issue for at least 12 months now with several tickets to support to clean up the partitions or fix the log rotations.  I’m told that it’s a known bug.  I see there a 2021.8 release now but I can’t update to this because i’m out of space and can’t download the installer.

“Not enough disk space in x.x.x.x for saving /home/nutanix/software_downloads/prism_central/pc.2021.8-f948d198de58b1b1e511431dbef0b34d20c82739-x86_64.tar.gz, 11.88 GiB needed, but 10.21 GiB usable”

J
Badge

Well I can at least confirm that pc.2021.8 solves the issue with the missing logrotate for /home/apache/ikat_access_logs/prism_proxy_access_log.out

D

I just had to run through this as well. It was very hard for myself. But I called support and they helped me.  I tried all above but man, getting space back was a pain.

Ray Davis
Terms & Conditions