It seems that when updating AOS Nutanix is creating backups of /home/apache/www/ called PrismUI.tar.gz . Those are then stored in /home/apache/www/console/ . It further seems they tried (but failed) to exclude this console directory. (Don't have the exact tar command line any more but I guess Nutanix can find it) This means that later backups include all previous ones, recursively. I.e. backup #1 is OK. But backup #2 will include #1. Backup #3 will even include #1 and #2 (including another copy of #1). And so on. Which means the file size of subsequent backups grows FAST. For one Cluster the file grew from the usual ~125 MB to over 3GB. This in turn leads to the backup being rejected and recreated constantly. Probably bad for the SSDs.
This happened on all Versions I was able to check: ESXi: 5.20.1.1, 5.20.2 AHV: 5.18.1.2 5.19.0.5 6.0.1 6.0.1.6 6.0.1.7 6.0.2
Here the most egregious example:
$ sudo find / /home/ -xdev -iname PrismUI.tar.gz | xargs -d'\n' ls -lah
-rwxr-x---. 1 nutanix apache 118M Oct 30 2020 /home/apache/www/console/el7.3-release-euphrates-5.18.1.1-stable-4546d2908cb8495b316deb45de63b7f5e52541a1/PrismUI.tar.gz
-rwxr-x---. 1 nutanix apache 236M Jan 11 2021 /home/apache/www/console/el7.3-release-euphrates-5.18.1.2-stable-b1b096696c0c034570545912a00d39746e901f36/PrismUI.tar.gz
-rwxr-x---. 1 nutanix apache 485M Jan 14 2021 /home/apache/www/console/el7.3-release-euphrates-5.19.0.5-stable-22254494062244528e7a1ea132c23fe4005cb89c/PrismUI.tar.gz
-rwxr-x---. 1 nutanix apache 2.0G Oct 12 10:09 /home/apache/www/console/el7.3-release-fraser-6.0.1.6-stable-a922e5fdfbffa9800546950baa9072d49c590a8c/PrismUI.tar.gz
-rwxr-x---. 1 nutanix apache 1001M Aug 17 15:51 /home/apache/www/console/el7.3-release-fraser-6.0.1-stable-c5a21b612ea2da5fadb6524673cd75c0074c982f/PrismUI.tar.gz
-rw-r-----. 1 nutanix nutanix 485M Nov 12 15:55 /home/nutanix/PrismUI.tar.gz
(The last file above is the new backup currently being created)
$ tar -tvf /home/apache/www/console/el7.3-release-euphrates-5.18.1.1-stable-4546d2908cb8495b316deb45de63b7f5e52541a1/PrismUI.tar.gz --wildcards "*.gz"
tar: *.gz: Not found in archive
tar: Exiting with failure status due to previous errors
$ tar -tvf /home/apache/www/console/el7.3-release-euphrates-5.18.1.2-stable-b1b096696c0c034570545912a00d39746e901f36/PrismUI.tar.gz --wildcards '*.tar.gz'
-rwxr-xr-x nutanix/apache 123612132 2020-10-30 08:52 console/el7.3-release-euphrates-5.18.1.1-stable-4546d2908cb8495b316deb45de63b7f5e52541a1/PrismUI.tar.gz
$ tar -tvf /home/apache/www/console/el7.3-release-euphrates-5.19.0.5-stable-22254494062244528e7a1ea132c23fe4005cb89c/PrismUI.tar.gz --wildcards '*.tar.gz'
-rwxr-xr-x nutanix/apache 247074288 2021-01-11 13:49 console/el7.3-release-euphrates-5.18.1.2-stable-b1b096696c0c034570545912a00d39746e901f36/PrismUI.tar.gz
-rwxr-xr-x nutanix/apache 123612132 2020-10-30 08:52 console/el7.3-release-euphrates-5.18.1.1-stable-4546d2908cb8495b316deb45de63b7f5e52541a1/PrismUI.tar.gz
$ tar -tvf /home/apache/www/console/el7.3-release-fraser-6.0.1-stable-c5a21b612ea2da5fadb6524673cd75c0074c982f/PrismUI.tar.gz --wildcards '*.tar.gz'
-rwxr-x--- nutanix/apache 508014736 2021-01-14 14:59 console/el7.3-release-euphrates-5.19.0.5-stable-22254494062244528e7a1ea132c23fe4005cb89c/PrismUI.tar.gz
-rwxr-x--- nutanix/apache 247074288 2021-01-11 13:49 console/el7.3-release-euphrates-5.18.1.2-stable-b1b096696c0c034570545912a00d39746e901f36/PrismUI.tar.gz
-rwxr-x--- nutanix/apache 123612132 2020-10-30 08:52 console/el7.3-release-euphrates-5.18.1.1-stable-4546d2908cb8495b316deb45de63b7f5e52541a1/PrismUI.tar.gz
$ tar -tvf /home/apache/www/console/el7.3-release-fraser-6.0.1.6-stable-a922e5fdfbffa9800546950baa9072d49c590a8c/PrismUI.tar.gz --wildcards '*.tar.gz'
-rwxr-x--- nutanix/apache 508014736 2021-01-14 14:59 console/el7.3-release-euphrates-5.19.0.5-stable-22254494062244528e7a1ea132c23fe4005cb89c/PrismUI.tar.gz
-rwxr-x--- nutanix/apache 1049239743 2021-08-17 15:51 console/el7.3-release-fraser-6.0.1-stable-c5a21b612ea2da5fadb6524673cd75c0074c982f/PrismUI.tar.gz
-rwxr-x--- nutanix/apache 247074288 2021-01-11 13:49 console/el7.3-release-euphrates-5.18.1.2-stable-b1b096696c0c034570545912a00d39746e901f36/PrismUI.tar.gz
-rwxr-x--- nutanix/apache 123612132 2020-10-30 08:52 console/el7.3-release-euphrates-5.18.1.1-stable-4546d2908cb8495b316deb45de63b7f5e52541a1/PrismUI.tar.gz
Subsequent backups for 6.0.1.7 and 6.0.2 then kept continuously failing and retrying till I temporarily moved the old backups out of the way. Leading to a successful 180MB backup.
For outsourcing reasons I'm not currently allowed or able to open actual tickets. But I (and I guess pretty much everyone else including nutanix) still want to see this fixed. So I submit it to the void and hope the right person listens. Good thing I don't actually need support with this. If it's fixed in one of the next release that will be sufficient for me.
