Skip to main content
Question

AHV Metro Availability: unexpected automatic failover with witness reportedly unreachable - how to handle planned site isolation?


Daniel Martinez
Forum|alt.badge.img+2

Hi,

I’m looking for clarification on AHV Metro Availability behavior and the supported procedure for temporarily isolating an entire site during network maintenance (switching replacement).

The environment consists of two Nutanix AHV clusters with synchronous replication, managed through Prism Centrals (one per site) and a Witness VM deployed at a third site over a VPN. As aditional info AOS and PC versions are PC/AOS 7.6 and AHV 11.0.2 .

The customer reported the following sequence during a switch replacement:

  1. They disconnected the VPN to the witness, intending to make it unreachable from both clusters (that should avoid automatic launch of any recovery plan).

  2. Then they disconnected one swich on cluster A, so cluster B was unable to reach it.

  3. According to the customer, at that moment recovery automatically started and the VMs from cluster A were recovered on the cluster B.

  4. As a result they reported duplicate VMs and had to determine which instances to retain (splitbrain scenario).

They say the same behavior occurred when repeating the maintenance on the opposite site.

 

That happens some months ago, so I cannot confirm whether the witness was actually unreachable from both sides, whether recovery was automatic or manually triggered, or whether the original and recovered VMs were simultaneously running. 

My understanding was that automatic recovery following an inter-site partition requires valid witness arbitration. However, I don’t want to assume that disconnecting the witness is a supported way to suppress automatic failover.

My questions are:

  • If both clusters lose access to the witness before losing connectivity to each other, under what conditions could automatic failover still occur?

  • What is the supported procedure to temporarily disable automatic recovery during a planned full-site shutdown or network isolation?

  • Is there a setting on the recovery plan, protection policy, or elsewhere specifically for this purpose?

  • How should pausing synchronous replication be coordinated with disabling automatic recovery?

  • Which events or logs would identify what triggered and authorized the recovery?

The goal is to understand what caused the issue and establish a documented maintenance procedure for future switch replacements. Links to official documentation or relevant KBs would be appreciated.

Thanks!

3 replies

Forum|alt.badge.img+3

Hi,
since I do not have much knowledge on metro on top of AHV,I hope someone else can validate my answers:
 

  • If both clusters lose access to the witness before losing connectivity to each other, under what conditions could automatic failover still occur?
    I dont think automatic failover happens, as automatic failover will be responsiblity of witness
    Please read this :
    Nutanix Disaster Recovery pc.7.6 - AHV Metro Recovery Workflows

  • What is the supported procedure to temporarily disable automatic recovery during a planned full-site shutdown or network isolation?
    switch to manual failover (disable enable automatic failover)
    Nutanix Disaster Recovery pc.2024.3.1 - Creating a Recovery Plan

  • Is there a setting on the recovery plan, protection policy, or elsewhere specifically for this purpose?
    option is there in recovery plan

  • How should pausing synchronous replication be coordinated with disabling automatic recovery?to pause the syn , you will do it from protection policy, select the policy and choose pause sync replication 

  • Which events or logs would identify what triggered and authorized the recovery?
    for this you have to go deep into the logs, but still tasks, audits and DR alerts could be useful as well.

    I hope someone corrects me if I have provided false information :)


Daniel Martinez
Forum|alt.badge.img+2

Thanks for the answers!

As you mentioned in order to prevent the automatic failover I think that modifying the disaster recovery trigger from automatic to manual would be enough, but also pausing the sync replication could be a good practice as well.

The big mistery here is why the Prism Central launched an automatic failover if witness was (in theory) out of connectivity.

 


JeroenTielen
Forum|alt.badge.img+8

If witness was cut off and clusters could not talk to each other the automatic failover should NOT happen as this will lead into duplicated VM's. There must be something else happening. 

 

For next time, set recovery plans to manual if you want to test something.