Hi,

did you install any NGT tools? See https://next.nutanix.com/backup-and-recovery-29/vss-snapshot-failed-12608

No.

Hello @kheraman 

We now raise an alert if Application Consistent Snapshots are attempted when Nutanix Guest Tools (NGT) is not installed on the VM or if the NGT communication link is down. The following document might help; 

https://portal.nutanix.com/#/page/kbs/details?targetId=kA00e000000CqILCA0

Can you install NGT on the VMs and see if you are again seeing the alert.

hello @kheraman

Are you still having an issue?

For successful application consistent VM snapshots, the following need to happen:

1. Backup snapshot is triggered with app-consistent option enabled.
2. The CVM reaches out to the NGT service running on the VM via TCP/IP to signal that VSS snapshot is needed.
   1. This requires TCP/IP communication be possible both ways, but should not need DNS since the NGT service will inform Prism of the IP, and the NGT installation has cluster IP info. (NAT could be a problem)
   2. The NGT service must, of course, be installed and running, and able to reach the CVM on port 2074.
      (more detail on what ports are needed for full NGT function here) 
   3. Communication uses a pre-shared key which is part of NGT installation, and an identifier which is unique to the VM. To have this work, NGT installation must be unique per VM using the “mount iso” option from Prism.
      If cloning VMs, you can pre-install NGT and then mount the ISO again on the clone before powering on. The NGT service will fetch updated identifier info during service start if the NGT ISO is found in the VM’s CDROM drive.
3. The NGT service requests VSS Quiesce operation from the Windows OS.
   1. In a quiesce operation, all new changes to disk are held in hot-backup in memory on the VM until the snapshot is finished.
   2. All pending changes to disk must finish before the snapshot can happen.
   3. This requires sufficient memory on the VM to hold all new changes long enough to complete the snapshot, otherwise application consistent snapshot will fail. If you’re seeing intermittent failures, this is where to focus.
   4. This process can be impacted by high workload, slower disk performance, hypervisor memory or CPU contention, VM memory or CPU contention, or any Windows VSS specific issue which prevents quiesce completion.
      1. Options for resolution include re-balancing workloads, adding resources at the host or VM, or adjusting scheduled jobs so that snapshots can run at lower-IO times. 
4. Once VSS signals back to NGT that quiesce is good, NGT service signals to Prism and then snapshot is taken. Prism then signals back to NGT, which relays to VSS, at which point new pending disk operations are allowed to flow to disk.

Where this error message indicates “an internal error” I would actually be looking at VSS and the Nutanix Guest Tools service on the user VM itself. A different error should be seen if Prism cannot reach the NGT service on the VM. There is yet another error when NGT has not been enabled. 

The KB article “Taking app-consistent (VSS) snapshots using NGT fails on Windows VMs” covers one scenario where the culprit is anti-virus software on the VM. The KB also gives some good general steps for exploring the issue with Event Viewer and the vssadmin command. These are often essential in identifying and resolving the issue. The important thing to look for in ‘vssadmin list writers’ and ‘vssadmin list providers’ is the last error state. If the last attempt was successful we’ll see an indication of no error. If you just tried the backup and still see no error here, our problem is happening before VSS gets triggered.

The article “Nutanix Guest Tools Troubleshooting Guide” provides further guidance on validating the Nutanix Guest Tools installation.