Skip to main content

 

Environment

  • AOS Version: 6.8.8

  • Hypervisor: AHV 2022.10

  • CVM: Nutanix CVM version 5.18.3

  • VM Guest OS: Ubuntu 20.04 LTS

  • Storage Type: vDisk attached as AHV data disk

Description

When I create a new vDisk, write data to it, and then delete the vDisk in Prism, the underlying storage space is not reclaimed immediately. I still see the same physical usage reported in Prism Element and on the CVM filesystem, even after:

  1. Deleting all snapshots and clones

  2. Running fstrim -v /mountpoint inside the VM

  3. Executing the Reclaim Space action in Prism

  4. Triggering a manual Curator scan (curator_cli scan start)

I expect the deleted blocks to be freed right away, but they remain allocated for hours or even days.

Steps to Reproduce

  1. In Prism Element, create a new 50 GB vDisk and attach it to a running VM.

  2. Inside the VM, format and mount the vDisk:

    mkfs.ext4 /dev/nvme1n1
    mount /dev/nvme1n1 /mnt/testdisk

  3. Write 20 GB of test data copying from windows os disk

  4. Unmount and delete the vDisk in Prism.

  5. Observe that the used capacity in Prism Element under Storage > Logical Usage vs Physical Usage remains unchanged.

What I've Tried

  • Verified no snapshots or clones exist for the VM/vDisk

  • Ran fstrim inside the guest OS

  • Used the Reclaim Space feature in Prism

  • Triggered curator_cli scan start on the CVM

  • Checked CVM storage paths (du -sh /home/nutanix/data/stargate-storage/*)

Expected Result

Deleted vDisk blocks should be reclaimed immediately (or within minutes), reducing physical usage on the cluster.

Actual Result

Physical usage remains constant; freed blocks are not reclaimed until much later (hours to days).

Request for Help

  • Has anyone experienced similar behavior or found a reliable method to force immediate space reclamation? Are there any additional CLI commands, configuration settings, or best practices I’m missing? Any guidance would be greatly appreciated.

    Also i would not like to go and run command from cvm everytime, i want to use API endpoints to do all above step using postman.
    - Create a VM
    - Create and attach disk to VM
    - Write data to disk 
    - Delete disk , as there is no option to detach disk in nutanix
    - Check for space
    - I need to do this in parallel 30 threads 
    That would mean everyday 30 X 50 GB space would be created and filled and needs to cleanup immediately.
     


Thank you!

Hilam

That is not how nutanix works. So all data is written as extends in the extend store. When you delete data (vdisk, or data in the disks) the data is not freed. Curator needs to clean that up, as the extends can still be used by other vdisks/snapshots/etc. When deleting a disk it is put in the recycle bin for 245 hours and after that curator needs to pick it up te get it cleaned. 

​@JeroenTielen 

I'm working with an application that continuously performs the following workflow in a Nutanix environment:

  1. Creates a virtual disk and attaches it to a VM.

  2. Writes data to the disk.

  3. Converts this disk into a VM disk and attaches it to a Demo VM (Before attach delete attached disks if any)

  4. Creates a VM recovery point (snapshot).

  5. Deletes the original data disk after it's no longer needed.

  6. Repeats the above steps for up to 40 disks in parallel.

  7. Create a disk from disk reference attached to VM whose RP was created

To manage space, I also delete the VM recovery point as soon as it exceeds two consecutive snapshots.

My concern is that with this volume of activity, I may run into disk space issues. Even though I’m deleting the temporary disks and old recovery points, I want to ensure that space is actually being reclaimed immediately and effectively.

I'm using the Nutanix APIs for all these operations.

My questions are:

  • Is there anything specific I should be doing to ensure the space is reclaimed immediately after deleting disks and recovery points?

  • Are there any best practices or configuration settings in Nutanix that I should be aware of to optimize space usage in this high-churn environment?

Any guidance or suggestions would be appreciated!

Thanks!

  • Is there anything specific I should be doing to ensure the space is reclaimed immediately after deleting disks and recovery points?ful

You can run a curator scan immediate when needed. But this will run automatically ever hour (partial scan) and every 6 hours (full scan) so it doesnt make any sense.

  • Are there any best practices or configuration settings in Nutanix that I should be aware of to optimize space usage in this high-churn environment?

Best practices is to keep it as is 😉 Eventually the space is freed. You can disable the recycle bin to skip the extra 24 hour wait.

  1. You can run a curator scan immediate when needed. But this will run automatically ever hour (partial scan) and every 6 hours (full scan) so it doesnt make any sense.

    How to do it using APIs or SDK ?
  2. Best practices is to keep it as is 😉 Eventually the space is freed. You can disable the recycle bin to skip the extra 24 hour wait.

    I can not ask some other user to disable their recycle bin, from my application’s perspective, what can i do ?
  3. Also there are so many existing products who does backup and replication to nutanix, how they manage storage issues and delayed storage scans ?
  1. You can run a curator scan immediate when needed. But this will run automatically ever hour (partial scan) and every 6 hours (full scan) so it doesnt make any sense.

    How to do it using APIs or SDK ?
  2. Best practices is to keep it as is 😉 Eventually the space is freed. You can disable the recycle bin to skip the extra 24 hour wait.

    I can not ask some other user to disable their recycle bin, from my application’s perspective, what can i do ?
  3. Also there are so many existing products who does backup and replication to nutanix, how they manage storage issues and delayed storage scans ?

 

  1. You can find it here: https://www.jeroentielen.nl/nutanix-ssh-scripts But dont do this, this will make impact cluster performance if you are going to do this over and over. . 
  2. Recycle bin is cluster wide. And yes leave it on ;) 
  3. I dont get the question. 

This is how nutanix works, and I think you are making it to complex for yourself. Just create your images and delete te virtual machines. Nutanix will take care of this and curator will cleanup the unused space. If you are running out of storage then probably the cluster is already running out of space and you need to expand. ;) 

Reply


Terms & Conditions