CVM Non-Root Partitions Space Usage High - clear space safetly

Nutanix reserves space on the SSD-tier of each CVM for its infrastructure. These files and directories are located in the /home folder that you see when you log in to a CVM. The size of the /home folder is capped at 40 GB so that the majority of the space on SSD is available for user data.

Due to the limited size of the /home partition, it is possible for it to run low on free space and trigger Prism Alerts, NCC Health Check failures or warnings, or Pre-Upgrade Check failures. These guardrails exist to prevent /home from becoming completely full, as this causes data processing services like Stargate to become unresponsive. Clusters with multiple CVMs having 100% full /home partition will often result in downtime for user VMs.

The Scavenger service running on each CVM is responsible for the automated clean-up of old logs in /home and improvements to its scope were made in AOS 5.5.9, 5.10.1, and later releases. For customers running earlier AOS releases, or in special circumstances, it may be necessary to manually clean up files out of certain directories in order to bring space usage in /home down to a level that will allow future AOS upgrades.

This condition can be reported in two scenarios:

• The NCC health check disk_usage_check reports that the /home partition usage is above a certain threshold (by default 75%)
• The pre-upgrade check test_nutanix_partition_space checks if all nodes have a minimum of 5.6 GB space on the /home/nutanix directory before performing an upgrade

The following error messages will be generated in Prism by the test_nutanix_partition_space pre-upgrade check:

Not enough space on /home/nutanix directory on Controller VM [ip]. Available = x GB : Expected = x GB

Failed to calculate minimum space required

Failed to get disk usage for cvm [ip], most likely because of failure to ssh into cvm

Unexpected output from df on Controller VM [ip]. Please refer to preupgrade.out for further information

WARNING: DO NOT treat the Nutanix CVM (Controller VM) as a normal Linux machine. DO NOT perform "rm -rf /home" on any of the CVMs. It could lead to data loss scenarios.

Only delete the files inside these directories. Do not delete the directories themselves.

• /home/nutanix/data/cores/
• /home/nutanix/data/binary_logs/
• /home/nutanix/data/ncc/installer/
• /home/nutanix/data/log_collector/

Solution:

1. Parsing the space usage for "/home".
2. Check for files that can be deleted from within the list of approved directories:
   • Removing Old Logs and Core Files

   • Removing Old ISOs and Software Binaries

3. Check space usage in /home to see that it is now below 70%.

For more details caveats and commands please read

KB-1540 What to do when /home partition or /home/nutanix directory is full.

KB-8086 NCC - Health Check command fails with error "No space left on device"

KB-7621 LCM Pre-Upgrade Check: test_nutanix_partition_space