Nutanix Disaster Recovery

We're designing a Metro/synchronous replication setup between two AHV clusters (PRD and DR), using a single Prism Central instance deployed at a third site to manage both clusters, host the Witness Service, and run our Protection Policies / Recovery Plans.Additional context: DR is also running some VMs directly (on a separate, non-replicated local container) due to a temporary memory capacity constraint on PRD. These VMs are not part of the synchronous Protection Policy — they're independent workloads currently living in DR.Our question: If the third site with the PC becomes completely unreachable at the same time PRD goes down, what is the supported method to manually promote the DR-side replicas and bring the protected VMs online at DR?

Hi everyone,I am trying to stop/destroy a 3-node Nutanix AHV cluster running on HPE ProLiant DX hardware in an offline/dark-site environment to reconfigure network IPs.Whenever I run "cluster stop" or "cluster destroy", the command fails with the following critical error: "CRITICAL MainThread cluster:3611 Destroy is in progress on the nodes: ['192.168.10.18']" Environment details:- Hardware: HPE ProLiant DX (Enterprise AOS)- Nodes: 3 (192.168.10.14, 192.168.10.15, 192.168.10.18)- Current State in 'cluster status': * CVM: 192.168.10.14 Up * CVM: 192.168.10.15 Up, ZeusLeader * CVM: 192.168.10.18 Up, Unconfiguring- User VMs: All powered off (State: Off)What is happening:- On CVM .18, ~/data/logs/genesis.out is stuck in a loop reporting: "INFO node_manager.py:1124 Unconfiguring node... Using legacy firewall framework as node is in unconfigured state"- Inspecting $(which cluster) around line 3611 shows an RPC check against the SVMs checking for ongoing destroy operations. Because node .18 r

Hi Guys,  I have a technical question for implementing multiple site into single site as a DR location using sync repository where RPO 0.Thank you for answering this question. Regards,Andreas Kuncoro 

EnvironmentNutanix AOS: 7.3.1.4 AHV: 10.3.1.4 PC Version: 7.3.1.5 Dataprotection API: v4.1 Authentication: JWT Bearer TokenIssueI am attempting to invoke the Dataprotection v4.1 Compute Changed Regions API:POST/api/dataprotection/v4.1/content/recovery-points/{rpUuid}/vm-recovery-points/{vmRpUuid}/disk-recovery-points/{diskRpUuid}/$actions/compute-changed-regionsThe API request is made using a JWT bearer token which is fetched from below API and fails with an authentication error.Discover cluster API - /api/dataprotection/v4.1/config/recovery-points/{extId}/$actions/discover-clusterCVM LogApi request with message ID: 51, URI:/api/dataprotection/v4.1/content/recovery-points/.../$actions/compute-changed-regionsEncountered error kAuthenticationRequirederror_detail Invalid cookies present in the requestSending an error response back to the clientClient RequestPOST https://<cluster>:9440/api/dataprotection/v4.1/content/recovery-points/.../$actions/compute-changed-regionsAuthorization:

We are using the Data Protection v4.1 API compute-changed-regions endpoint from Prism Central 7.5 (AOS 11.0.0.1) to retrieve changed regions for VM recovery points.For most VMs, the workflow completes successfully. However, for VMs with large changed data disks, the API consistently fails with:401 UNAUTHORIZEDThe failure occurs approximately 15 minutes after the operation starts.Our current steps are:compute-changed-regions is invoked successfully for cluster prism element API - api/dataprotection/v4.1/content/recovery-points/{recoveryPointExtId}/vm-recovery-points/{vmRecoveryPointExtId}/disk-recovery-points/{extId}/$actions/compute-changed-regions Nutanix begins computing changed regions. The VM disks are highly fragmented and may contain a very large number of changed extents. Results are returned through pagination. The client continues fetching paginated results. Total processing time exceeds approximately 15 minutes. A subsequent page retrieval request returns 401 UNAUTHORIZED.Add

i  have two cluster  main  cluster  and DR cluster , what  is the best practice to upgrade two cluster , can i start with main or DR first , what about replication  it is effected with upgade? 

I am setting up recovery plans for disaster recovery between our two clusters. I have been through the documentation multiple times, and I still am confused by the test failover/failback subnets that are configured as part of the recovery plan. Production is easy. We have the primary cluster in one location in Subnet A. We have a recovery cluster in another location in Subnet B. I just set the production subnets to be Subnet A and B at each location respectively with their matching gateway and prefix. The Test Failover/Failback subnets continue to confuse me. The documentation indicates they should be isolated and non-routable subnets which makes sense. But WHERE are those subnets defined? We use external IPAM. Each cluster is connected to a core switch stack at its location. We have a few different subnets/VLANs defined on those core switches, with a defined IP range, /24 in each case. All VMs have static IP addresses, though the core switches do run DHCP for each subnet. We also have

Hi Experts !We recently upgraded one of our Nutanix clusters to AOS 6.8.1.8 and started facing an issue with the Cerebro service shortly after.Symptoms observed:Remote replication alerts after the upgrade.Remote site configuration warning indicating that some CVM/SVM IPs were not properly configured on the peer site.Cerebro entering a crash loop?cluster status showing constantly changing/high PIDs for Cerebro.cerebro.FATAL reporting an error similar to: Check failed: citer != (pd.second)->snapshot_uuid_map().end() with a snapshot stuck in a pending action.It looks like there is a stale or orphaned snapshot operation in the WAL / metadata, likely triggered after the upgrade while replication configuration was not fully consistent.Has anyone already seen this behavior after an AOS upgrade?Did you resolve it by fixing the remote site configuration only, or did it require Nutanix Support intervention to skip/clean the offending WAL operation?Thanks :)

Are we able to configure a recovery plan in Nutanix Leap when we have Production and DR clusters (10 Clusters each) managed across two separate PCVMs (both running version 7.5.0.5)?Currently, while creating the recovery plan, I can select individual VMs, but I do not see an option to select at the cluster level. Is this expected behavior when using two PCVMs, and how can we properly configure a recovery plan in this setup?Any guidance on achieving this design would be helpful.

Hi I have a customer that is asking me for some feature on Nutanix (AHV) that may provide some kind of immutability for the VM snapshots, so in the event of an admin account hack the snapshots could not be destroyed.I was looking for something like that on AHV and I discovered Secure Snapshots with Approval Policies in Prism Central and I’d like to confirm a few points regarding requirements and licensing.https://portal.nutanix.com/page/documents/details?targetId=Disaster-Recovery-DRaaS-Guide-vpc_7_5:ecd-approval-policies-dr-pc-c.htmlContext:The customer has two AHV clusters, each one with his own Prism Central Licensing: NCI Pro + Advanced Replication add-on (Metro/Sync already in use) + NUS Pro Goal: prevent accidental or malicious deletion of snapshots/recovery points, not VMs themselves (it would be also great but I think it can’t be protected with aprobal policies)So from the documentation, I understand that Secure Snapshots allows attaching an Approval Policy to a Protection Poli

HiThis thread is related with the previous one that I opened some days ago but it was already closed as solved.https://next.nutanix.com/nutanix-disaster-recovery-29/replicated-recovery-points-not-found-error-45362In summary:I’ve recently deployed a pair of AHV clusters, each one with his on Prism Central. Later I’ve created a specific container for testing purpose called “Replica-Container” on both sites I’ve moved some test VMs to that specific container and and I’ve created a manual bi-directional syncronous replica between both sites. I’ve manualy added the test VMs to the protection plan and I’ve waited some time for them to replicate between sites. Finaly I’ve created two different Recovery plans Recovery Plan1 - recover VMs from site1 at site2 Recovery Plan2 - recover VMs from site2 at site1 I’ve validated both plans (succesfuly) and I’ve performed a Test.The initial test worked fine for plan2, however Recovery Plan1 shows this error:  I assumed that I should have to wait more t

Hi,I’m currently configuring synchronous replication (manual failover, no Witness VM yet) between two AHV clusters managed by Prism Central 7.5.0.5 (each cluster has it’s own PC).Since there is no Witness deployed, this is not Metro Availability, but a synchronous Protection Policy with manual failover. As soon as I have the Witness VM I will test the Metro Availability feature.Current setup Created a container named Metro_CPD1 on both clusters. Created 3 test VMs on Cluster1 and placed them in the Metro_CPD1 container. From PC1 I’ve created a Synchronous Protection Policy (ReplicaMetro-CPD1-to-CPD2) with manual failover. From PC1 I’ve created a Recovery Plan (Recovery-metro-replica1) : Primary location: Cluster1 Recovery location: Cluster2 Execution mode: Manual Added the 3 VMs to the Recovery Sequence. Configured Network Mappings (Prod LAN and Test LAN with IP pools). After creating the Recovery plan I do a “Validate” and it completes successfully on both site

Currently we are running VM’s workload in VXRail VMware, these VM’s replicating to Azure cloud using recovery vault.Now these VM’s migrating to Nutanix native AHV environment.Please advise how to replicate VM’s from Nutanix to Azure cloud using which tool,Thanks in advance.

Hi, We have two cluster, Prod and DR. We have protection policies all set up, as well as recovery plans. However, when performing a planned failover (for testing) ​​​​​​, the execution of the recovery fails. But it also succeeds… with warnings. The warnings say as follow:------VMs/Recovery Points detected on Prism Element(s) with unsupported licenseIMPACTSome VMs might not be recovered.CAUSESome of the VMs/Recovery Points are located on Prism Element(s) with unsupported license.RESOLUTIONMake sure that the Prism Element(s) have supported license. Supported License includes Ultimate License or Pro license with add-on advanced replication.-----The (planned) failover is also successful - so why the warnings? We have Pro licence. As far as I understand it, we need Ultimate licence for Test failover, but not for the planned failover (as long as we don’t do staging in bringing VMs back up). Are these warnings a red herring? Also, what about unplanned failover? Will we also just get the warni

We are using Nutanix Data protection APIs to compute the changed regions between VM recovery points, where we are seeing in-consistent behaviour. API version - v4.1API First we do discover cluster on Prism central IP - /api/dataprotection/v4.1/config/recovery-points/{extId}/$actions/discover-cluster Then we Compute VM change regions on Prism Element IP - api/dataprotection/v4.1/content/recovery-points/{recoveryPointExtId}/vm-recovery-points/{vmRecoveryPointExtId}/disk-recovery-points/{extId}/$actions/compute-changed-regionsSo for Second request the start offset is provided as 0 to compute the regions from the start. It provides the change regions and let’s say the last offset changed is n. Now when we request the same compute regions API with the start offset n, it stills gives more compute regions which should not be the case as the last request ended the compute regions in n. We check below meta data response for if the response in paginated - isPaginated - flag nextOffset - extraIn

Hi.Currently, virtual machine A running on cluster A (consisting of three machines) is being remotely backed up to cluster B (consisting of one machine) using Prism Element's protection domain.In this state, is it possible to remotely back up virtual machine B running on cluster B to cluster A using the protection domain?Is this a problem?kind regards.

Our current setup is physical prod servers and VMs are in 10. 172.31.40.1 segment and DR physical servers are in 172.31.30.1 segment and DR VMs are in 10. 172.31.40.1 segment and we configured Async DR setup. Currently we configured IPsec VPN in HQ and DR through FortiGate firewall. If production site  down, how can we access the production VM servers in DR site. This is our concern please provide solution for this design.

I’m configuring Nutanix DR with zero compute but when i try to deploy MST from marketplace i receive the following errors:Credential is invalidVariable new_secret_keySecret value for variable is emptyVariable new_access_keySecret value for variable is emptyCredential default_credSecret value for credential is emptyI’ve tried to configure default credentials from self service but with no luck.thanks in advance

Hi everyone,A customer currently operates a VMware vSAN cluster at their primary site and is planning to deploy a new HCI platform at a secondary site. One of the key requirements is that the new HCI cluster must be able to replicate workloads to the existing VMware vSAN environment.My question is the following:Is it feasible to propose Nutanix with AHV for the secondary site while keeping VMware vSAN at the primary site? If yes, which third-party solution would be recommended to ensure reliable replication between Nutanix AHV and VMware vSAN? If not, what alternative HCI architecture would you recommend to meet this requirement? Thanks in advance for your insights and feedback.

Hello,First post here!I previously developed an automation flow using the Nutanix V3 API that interacts with Prism Central and orchestrates the "Test Failover", "Failover" and "Clean Up" DR Actions for a specified list of Virtual Machines.For the V3 API version, I utilised both the "recovery_plans" and "recovery_plan_jobs" API endpoints.These do not appear to be present within the V4 API (not that I can find in the various V4 API documentation).Are there plans to add these and if so, are there any timescales for when these will be added?Thanks.Regards,Paul 

I am working on setting up DR replication between two 3 node Nutanix clusters running AOS and AHV. Can someone help clarify something for me? It does not appear that you can replicate a PC VM, and the clusters can only be registered to a single PC instance. So:If the cluster hosting the PC VM were to go down, how would you be able to initiate a manual failover with the controlling PC VM also down?Do I need to stand up PC on each cluster and register each to a separate PC instance to avoid this single point of failure?Is bi-directional replication possible with two clusters registered to the same PC? It appears that once a protection policy is setup to replicate from cluster A-->B that A is no longer an option to setup as a replication target from cluster B.

Hi. It appears that you cannot backup PC to an AWS storage gateway, it must be able to reach AWS natively. Is this true? Thank you!

Hi everyoneI have to deploy a Metro Availability architecture for two AHV clusters (no ESXi involved) and I have several questions so I’d appreciate some feedback from those who have deployed similar setups. The main goal is to achieve the lowest possible RPO and RTO between two datacenters with a robust and simple architecture. Both clusters will be separated by about 100m on two separate buildings and they will be connected with FC cables (<5ms).As far as I know, only ESXi with vMSC supports true RTO = 0 However, since this environment is AHV-only, my understanding is that in a failover event the VMs from Site A would need to be powered on in Site B rather than running continuously on both sides... Is that correct?That being said, here you are the possible scenarios to deploy Metro Availability between both sites: Two AHV clusters in active–active mode, without an external Witness Two AHV clusters in active–active mode, with an external Witness (this one looks nice for me) Two

With the Starter license, am I able to configure asynchronous replication in both directions— from site A to site B and from site B to site A?

Hi,Thoughts from the community please on this one.We have two clusters, in two different data centers with NearSync to replicate VMs.  So my problem is when I do a unplanned fail-over I get a VM with a new GUID, all good, until my backup software (Veeam) sees this as a new VM and creates a fresh full backup.  How do I deal with this in a better way? A full backup of all my VMs is going to need a lot of space and time!