Skip to main content
Question

After a power loss, cluster starts, but, VM freeze at VM Register at 0%

  • Nov 22, 2026 4:44 AM UTC
  • 1 reply
  • 51 views

Forum|alt.badge.img+2

Hypervisor IP  Hypervisor DNS Name  Host UUID                             Node state       Connected  Node type       Schedulable  Hypervisor Name  CVM IP    
172.16.4.7     172.16.4.7           81b17d48-fc0d-470d-8e6b-164e1cbcc95d  AcropolisNormal  False      Hyperconverged  True         AHV              172.16.4.10
172.16.4.8     172.16.4.8           8bbb63b9-0ba6-4d01-8e29-5d588b1f3d0f  AcropolisNormal  True       Hyperconverged  True         AHV              172.16.4.11
172.16.4.9     172.16.4.9           7293aec6-e687-4d85-9f77-d2c832c61ba5  AcropolisNormal  False      Hyperconverged  True         AHV              172.16.4.12
 

nutanix@NTNX-J100FKV4-A-CVM:172.16.4.11:~$ ncli host ls

    Id                        : 000590a2-c194-bf65-3101-3868dd0ee240::5
    Uuid                      : 81b17d48-fc0d-470d-8e6b-164e1cbcc95d
    Name                      : ntnxnodo1
    IPMI Address              : 172.16.4.4
    Controller VM Address     : 172.16.4.10
    Controller VM NAT Address :
    Controller VM NAT PORT    :
    Hypervisor Address        : 172.16.4.7
    Hypervisor Version        : Nutanix 20230302.103014
    Host Status               : NORMAL
    Oplog Disk Size           : 385.88 GiB (414,335,346,763 bytes) (1.8%)
    Under Maintenance Mode    : false (life_cycle_management)
    Metadata store status     : Metadata store enabled on the node
    Node Position             : Node physical position can't be displayed for this model. Please refer to Prism UI for this information.
    Node Serial (UUID)        : J100FB84
    Block Serial (Model)      : J100FB84 (HX2320)

    Id                        : 000590a2-c194-bf65-3101-3868dd0ee240::6
    Uuid                      : 8bbb63b9-0ba6-4d01-8e29-5d588b1f3d0f
    Name                      : ntnxnodo2
    IPMI Address              : 172.16.4.5
    Controller VM Address     : 172.16.4.11
    Controller VM NAT Address :
    Controller VM NAT PORT    :
    Hypervisor Address        : 172.16.4.8
    Hypervisor Version        : Nutanix 20230302.103014
    Host Status               : NORMAL
    Oplog Disk Size           : 385.88 GiB (414,335,269,229 bytes) (1.8%)
    Under Maintenance Mode    : false (life_cycle_management)
    Metadata store status     : Metadata store enabled on the node
    Node Position             : Node physical position can't be displayed for this model. Please refer to Prism UI for this information.
    Node Serial (UUID)        : J100FKV4
    Block Serial (Model)      : J100FKV4 (HX2320)

    Id                        : 000590a2-c194-bf65-3101-3868dd0ee240::7
    Uuid                      : 7293aec6-e687-4d85-9f77-d2c832c61ba5
    Name                      : ntnxnodo3
    IPMI Address              : 172.16.4.6
    Controller VM Address     : 172.16.4.12
    Controller VM NAT Address :
    Controller VM NAT PORT    :
    Hypervisor Address        : 172.16.4.9
    Hypervisor Version        : Nutanix 20230302.103014
    Host Status               : NORMAL
    Oplog Disk Size           : 385.88 GiB (414,335,269,229 bytes) (1.8%)
    Under Maintenance Mode    : false (planned_outage_host_maintenance)
    Metadata store status     : Metadata store enabled on the node
    Node Position             : Node physical position can't be displayed for this model. Please refer to Prism UI for this information.
    Node Serial (UUID)        : J100FB85
    Block Serial (Model)      : J100FB85 (HX2320)
nutanix@NTNX-J100FKV4-A-CVM:172.16.4.11:~$ ^C
nutanix@NTNX-J100FKV4-A-CVM:172.16.4.11:~$ acli host.list
Failed to connect to server: [Errno 111] Connection refused
nutanix@NTNX-J100FKV4-A-CVM:172.16.4.11:~$ ncli host ls

    Id                        : 000590a2-c194-bf65-3101-3868dd0ee240::5
    Uuid                      : 81b17d48-fc0d-470d-8e6b-164e1cbcc95d
    Name                      : ntnxnodo1
    IPMI Address              : 172.16.4.4
    Controller VM Address     : 172.16.4.10
    Controller VM NAT Address :
    Controller VM NAT PORT    :
    Hypervisor Address        : 172.16.4.7
    Hypervisor Version        : Nutanix 20230302.103014
    Host Status               : NORMAL
    Oplog Disk Size           : 385.88 GiB (414,335,346,763 bytes) (1.8%)
    Under Maintenance Mode    : false (life_cycle_management)
    Metadata store status     : Metadata store enabled on the node
    Node Position             : Node physical position can't be displayed for this model. Please refer to Prism UI for this information.
    Node Serial (UUID)        : J100FB84
    Block Serial (Model)      : J100FB84 (HX2320)

    Id                        : 000590a2-c194-bf65-3101-3868dd0ee240::6
    Uuid                      : 8bbb63b9-0ba6-4d01-8e29-5d588b1f3d0f
    Name                      : ntnxnodo2
    IPMI Address              : 172.16.4.5
    Controller VM Address     : 172.16.4.11
    Controller VM NAT Address :
    Controller VM NAT PORT    :
    Hypervisor Address        : 172.16.4.8
    Hypervisor Version        : Nutanix 20230302.103014
    Host Status               : NORMAL
    Oplog Disk Size           : 385.88 GiB (414,335,269,229 bytes) (1.8%)
    Under Maintenance Mode    : false (life_cycle_management)
    Metadata store status     : Metadata store enabled on the node
    Node Position             : Node physical position can't be displayed for this model. Please refer to Prism UI for this information.
    Node Serial (UUID)        : J100FKV4
    Block Serial (Model)      : J100FKV4 (HX2320)

    Id                        : 000590a2-c194-bf65-3101-3868dd0ee240::7
    Uuid                      : 7293aec6-e687-4d85-9f77-d2c832c61ba5
    Name                      : ntnxnodo3
    IPMI Address              : 172.16.4.6
    Controller VM Address     : 172.16.4.12
    Controller VM NAT Address :
    Controller VM NAT PORT    :
    Hypervisor Address        : 172.16.4.9
    Hypervisor Version        : Nutanix 20230302.103014
    Host Status               : NORMAL
    Oplog Disk Size           : 385.88 GiB (414,335,269,229 bytes) (1.8%)
    Under Maintenance Mode    : false (planned_outage_host_maintenance)
    Metadata store status     : Metadata store enabled on the node
    Node Position             : Node physical position can't be displayed for this model. Please refer to Prism UI for this information.
    Node Serial (UUID)        : J100FB85
    Block Serial (Model)      : J100FB85 (HX2320)
 

1 reply

Forum|alt.badge.img+2
  • Author
  • Adventurer
  • Nov 22, 2026 7:00 AM UTC

Acropolis service may be in restart loop with the CRITICAL message by memory_model_manager.py

Article #KB-18070

Last modified onJun 25th 2026

Summary:

Acropolis service may end up in a restart loop due to an unexpected memory model entry.

Versions affected:

AOS 6.8.x, AOS 6.10.x

AHV

Description:

 

On AOS clusters running 6.8.x or 6.10.x versions that have at least one AHV node, the Acropolis service may enter a crash loop following a cluster-wide outage. Although this can happen during planned service restarts, including upgrades, it is more likely to occur if services have been disrupted across multiple nodes, such as when a network switch is temporarily down.

Important: Because an external cause for the original cluster outage may seem to explain why VMs are down in the environment, and the cluster data resiliency is OK afterward, the Acropolis service restart loop may not be obvious. However, VMs may still be down as a result because VM manageability is prevented by this condition.

Symptoms

  • An alert for the Cluster service restarting is generated for Acropolis and the relevant services
  • An alert for "Inconsistent Virtual Switch State Detected" after the customer tried to modify or add a bridge. The rolling restart task is in a no-progress state in Ergon when trying to put the host in maintenance mode
  • In the Prism Element (PE) UI, the VMs show IP addresses, but they are not reachable on the network. This is due to the Acropolis service being unstable, leaving the VMs in a halt off state
  • The "Launch Console" option in PE is greyed out for VMs
  • acli command on the Controller Virtual Machines (CVM) is unable to list the VMs, and/or acli host.list command entries may show "Connected False"
  • Prism Central may be down, or it may cause any acropolis task initiated on the registered PEs from Prism Central to enter a no progress state at 0%. For example, image creation, VM power state, etc.

The acli host.list command will error out or show "Connected False"

nutanix@CVM:~$ acli host.list
Failed to connect to server: [Errno 111] Connection refused
nutanix@CVM:~$ acli host.list
Hypervisor IP Hypervisor DNS Name Host UUID Node state Connected Node type Schedulable Hypervisor Name CVM IP
x.x.x.x x.x.x.x fc2-4214-b0a0-2ea53a4382f5 AcropolisNormal False Hyperconverged True AHV x.x.x.x
x.x.x.x x.x.x.x 3f2-4821-8b0f-9ccf95cb931b AcropolisNormal True Hyperconverged True AHV x.x.x.x
x.x.x.x x.x.x.x 4e4-431f-8557-795e5390a679 AcropolisNormal False Hyperconverged True AHV x.x.x.x


The Acropolis service may be in a restart loop if an unexpected memory model entity is detected.

nutanix@CVM:~$ allssh 'grep -w CRITICAL /home/nutanix/data/logs/acropolis.out'

On those CVMs, /home/nutanix/data/logs/acropolis.out log may have the following traces:

Case 1: CRITICAL memory_model_manager.py:185 X > Y failed

2024-12-03 22:09:18,707Z CRITICAL memory_model_manager.py:185 X > Y failed, Stack:
File "build/bdist.linux-x86_64/egg/util/misc/decorators.py", line 50, in wrapper
File "build/bdist.linux-x86_64/egg/util/hypervisor/connection.py", line 638, in _reconnect_loop
File "build/bdist.linux-x86_64/egg/acropolis/hypervisor/kvm/host.py", line 2011, in _reconcile_host_agent
File "build/bdist.linux-x86_64/egg/acropolis/hypervisor/kvm/host.py", line 1833, in _maybe_reconcile_host_state
File "build/bdist.linux-x86_64/egg/acropolis/host/memory_model_manager.py", line 156, in update_host_memory_model
File "build/bdist.linux-x86_64/egg/acropolis/host/memory_model_manager.py", line 185, in _maybe_update_version_in_use
2024-12-03 22:09:18,719Z INFO 65183 /src/bigtop/infra/infra_server/cluster/service_monitor/service_monitor.c:97 StartServiceMonitor: Child 1302692 exited with status: 1

Case 2: CRITICAL memory_model_manager.py:83 X == 1 failed

2025-02-18 01:26:24,629Z CRITICAL memory_model_manager.py:83 2 == 1 failed, Stack:
File "build/bdist.linux-x86_64/egg/util/misc/decorators.py", line 50, in wrapper
File "build/bdist.linux-x86_64/egg/util/master/master.py", line 125, in _wait_for_leadership
File "build/bdist.linux-x86_64/egg/acropolis/master/master.py", line 110, in _acquired_leadership
File "build/bdist.linux-x86_64/egg/acropolis/host/memory_model_manager.py", line 83, in initialize
2025-02-18 01:26:24,637Z INFO 64607 /src/bigtop/infra/infra_server/cluster/service_monitor/service_monitor.c:97 StartServiceMonitor: Child 64609 exited with status: 1

We will also see the error signature below in the Acropolis logs during the restart loop on  the leader

2025-05-14 20:46:58,685Z ERROR stats_backend.py:98 arithmos_publish_all task failed: Unable to connect to host x.x.x.x
2025-05-14 20:46:58,685Z INFO stats_backend.py:103 Traceback (most recent call last):
File "build/bdist.linux-x86_64/egg/acropolis/server/stats_backend.py", line 96, in _execute_batch_helper
File "build/bdist.linux-x86_64/egg/acropolis/server/arithmos_backend.py", line 785, in _publish_all
File "build/bdist.linux-x86_64/egg/acropolis/server/arithmos_backend.py", line 1349, in _publish_all_agent_entities
File "build/bdist.linux-x86_64/egg/acropolis/hypervisor/hv_stats_util.py", line 2569, in get_host_vm_net_bindings
File "build/bdist.linux-x86_64/egg/acropolis/hypervisor/kvm/ahv_gateway_connection.py", line 71, in wrapper
File "build/bdist.linux-x86_64/egg/acropolis/hypervisor/kvm/nic_mixin.py", line 745, in get_vm_nics_with_status
File "build/bdist.linux-x86_64/egg/acropolis/hypervisor/kvm/host.py", line 370, in ahv_gateway_handle
File "build/bdist.linux-x86_64/egg/acropolis/hypervisor/kvm/ahv_host_connection.py", line 392, in ahv_gateway_handle
File "build/bdist.linux-x86_64/egg/acropolis/hypervisor/kvm/acropolis_kvm_base_connection.py", line 70, in _raise_connection_error
acropolis.common.exception.AcropolisRemoteConnectionError: Unable to connect to host x.x.x.x

Solution:

 

This issue is resolved in:

  • AOS 6.10.X family: AOS 6.10.1.7
  • AOS 7.0.X family: AOS 7.0

Upgrade AOS to versions specified above or newer.

Engage Nutanix Support to recover the impacted cluster.