How It works

In many ways, identifying the problem is harder than solving it. At least in IT.In order to better understand the performance of Nodes in a cluster and User VMs, ESXTOP for ESXi hypervisors and TOP for Linux OS based machines provide us an immediate birds eye view of the performance of the host. It is important to understand the output as a tool towards problem-solving. Here I aim to discuss scenarios you might need to isolate and troubleshoot High CPU observed at the Hypervisor or User VMs.ESXTOP : This command when run lists live CPU statistics specific to the Node the command is run on.esxtop outputIf you press “M” it shows you memory metric and “N” for network etc. As Always ‘H’ is for help. We will focus on a few CPU statistics:The output will show you all the VMs and the following metrics corresponding to them.Some Important ones discussed below:%USED, %RDY, %CSTP , %MLMTD and %SWPWT Note:To convert CPU ready % value  to ms(milliseconds)CPU ready % = ((CPU summation ready value i

PCI device enumeration is one of a large number of concepts that transitioned from the physical world. Originally, being a bus and a slot number now, of course, is a virtualised concept.When OS boots during the POST process the local devices are enumerated (checked for presence and size). If an expected device is found at the PCI slot that is a failure of the test. In AHV deployments, the Controller VM (CVM) runs as a VM and disks are presented using PCI passthrough. That means that if AHV PCI devices enumeration has changed CVM may not be aware of the fact and attempt to direct I/O to the devices that are no longer accessible via known PCI bus and slot. This can happen after hardware replacement on an AHV host and the CVM will not boot or will boot with only part of the expected devices accessible. Compare PCI slot numbers of SCSI controllers on AHV host and the CVM and update the enumeration of the devices on the CVM. For more details look at KB-7154 AHV | CVM might not boot after ha

Anyone have an issue where you are not able to apply labels anymore to VMs after upgrading to 5.18.1.1 AOS and Prism Central pc-2020.9.0.1?  I am unable to apply labels after upgrade.

Hello,I’m trying to stop the cluster before shutting down and powering off the CVM Nodes. I’m using the following command to attempt to stop the cluster, however, it doesn’t seem to work, the command and the output is pasted, please tell me what am I doing wrong?$nutanixclusterstatus = Invoke-VMScript -VM $cvms[0] -GuestCredential $nutanixlogincreds -ScriptText {/usr/bin/echo 'I agree' | /usr/local/nutanix/cluster/bin/cluster stop} -ScriptType Bash ScriptOutput-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|  /usr/bin/echo: /usr/bin/echo: cannot execute binary file|  2020-12-05 04:30:17 INFO zookeeper_session.py:143 cluster is attempting to connect to Zookeeper|  2020-12-05 04:30:17 INFO cluster:2716 Executing action stop on SVMs 10.65.241.247,10.65.241.248,10.65.241.249,10.65.241.246|  2020-12-05 04:30:17 WARNING genesis_utils.py:279 D

The cloud connect feature enables you to back up and restore copies of virtual machines and files to and from an on-premise cluster and a Nutanix Controller VM located on the Amazon Web Service (AWS) or Microsoft Azure cloud. When a new cloud remote site is created, Nutanix will automatically spin up a single node Nutanix cluster in EC2 (currently m1.xlarge) or Azure Virtual Machines (currently D3) to be used as the endpoint.The Cloud Connect feature enables you to back up and restore copies of virtual machines and files to and from an on-prem cluster and a Nutanix Controller VM on the Amazon Web Service (AWS) or Microsoft Azure cloud. The Nutanix Controller VM is created on an AWS or Azure cloud in a geographical region of your choice. It is a single-node cluster with a 30 terabyte (TB) disk attached to the node, with a usable disk capacity of 20 TB.The following figure shows a logical representation of a “remote site” used for Cloud Connect: Since a cloud based remote site is similar

Below are new knowledge base articles published on the week of November 29-December 5, 2020.KB 9623 - NCC Health Check: cvm_network_error_check KB 9963 - NCC Health Check: host_boot_disk_uvm_check KB 10232 - Pre-Upgrade Check: test_duplicate_disk_ids KB 10375 - Nutanix Files : Windows clients error "Insufficient system resources exist to complete the requested service" KB 10379 - [Karbon] kubectl commands return "Forbidden" error for domain users assigned the "Cluster Admin" or "Viewer" role KB 10384 - How to find who powered off/on a VM running on AHVNote: You may need to log in to the Support Portal to view some of these articles.

Protection Domain–Based Data Protection is configured using Prism Element. Nutanix supports several types of protection strategies including one-to-one or one-to-many replication. These strategies are as follows:Per-VM Backup. The ability to designate certain VMs for backup to a different site is particularly useful in branch office environments. Typically, only a subset of VMs running in a branch location require regular back up to a central site. Such per-VM level of granularity, however, is not possible when replication is built on traditional storage arrays. In these legacy environments replication is performed at a coarse grain level, entire LUNs or volumes, making it difficult to manage replication across multiple sites. Selective Bi-directional Replication. In addition to replicating selected VMs, a flexible replication solution must also accommodate a variety of enterprise topologies. It is no longer sufficient to simply replicate VMs from one active site to a designated passiv

SFP give us freedom of choice and granular control over the media being used in the network segment. Mix and match optic fibre and copper of various throughputs and lengths of the segment. Very convenient and flexible solution and easily replaceable SFP modules.  When troubleshooting issues or during maintenance, the hardware details of the SFP module plugged into a host NIC may need to be identified and verified. Typically the useful information is printed on the SFP label but who has the time for this. More importantly, it would mean taking the SFP module out and that means another maintenance window. Using ethtool on AHV and XenServer will help with retrieving information like vendor, model, part number, serial number, transceiver type, cable length, connector type, signal quality, and more. For more details refer to KB-9102 How to identify plugged-in SFP module model, manufacturer, and hardware details

Do you often get confused between alert generated and Nutanix Cluster Check (NCC) failure?Here are some points to understand them both:Alert: Mechanism to report underlying issues in the system. NCC: Tool to check the cluster health and report alert if required. (If there is a NCC failure then it does not always generates an alert.) Three important sections of the above diagram:Notifications received - Cluster Health service. Configuration stored - IDF database. Alert reported - Alert Manager.Configuration and Alert Reporting:The alert reporting is based on the above configuration.We can also fetch the above information via "ncli alerts get-alert-config" and update using "ncli alerts update-alert-config" Tools:For checking the alerts received to nos-alert: ZygradeFor checking the alert: Insights Logs/Command: Alert Manager leader: alert_tool To check if the alert notification is send to email recipients: "alert_manager.INFO" log file in the alert manager leader. To check about the aler

Below are the top knowledge base articles for the month of November 2020.KB 7503 - NX Hardware [Memory] – G6, G7 platforms - DIMM Error handling and replacement policy KB 4116 - NX Hardware [Memory] – Alert - A1187, A1188 - ECCErrorsLast1Day, ECCErrorsLast10Days KB 4141 - Alert - A1046 - PowerSupplyDown KB 1540 - What to do when /home partition or /home/nutanix directory on a Controller VM is full KB 1113 - HDD/SSD Troubleshooting KB 2090 - AHV | Host Networking KB 4158 - Alert - A1104 - PhysicalDiskBad KB 4409 - LCM: (LifeCycle Manager) Troubleshooting Guide KB 4519 - NCC Health Check: check_ntp KB 4272 - Alert - A6516 - Average CPU load on Controller VM is critically high KB 2473 - NCC Health Check: cvm_memory_usage_check KB 6945 - How Upgrades Work at Nutanix KB 4639 - How to place CVM and host in maintenance mode KB 4273 - NCC Health Check: aged_third_party_backup_snapshot_check and  aged_entity_centric_third_party_backup_snapshot_check KB 7386 - NCC Health Check: power_supply_chec

Below are new knowledge base articles published on the week of November 22-28, 2020.KB 8564 - NCC Health Check: robo_readonly_state_check KB 8566 - NCC Health Check: robo_cluster_witness_network_check KB 8567 - NCC Health Check: robo_cluster_nodes_ping_check KB 9100 - 2 node cluster upgrade with witness VM unreachable causes downtime KB 9736 - Unregistering hosting AOS/PE Cluster from Prism Central with CMSP Enabled KB 9961 - Alert: 111081: XPilot Playbook is waiting for Approval to retry KB 10290 - How to create Windows installation ISO with built-in VirtIO drivers KB 10292 - Unable to filter using AppTier on Flow UI KB 10294 - Unable to Uninstall Vmware tools in Windows after Migration by Move from Esxi to AHV KB 10300 - Uploaded AOS upgrade binary is unavailable on Prism KB 10303 - LCM - SPP upgrades failure on HPE DXNote: You may need to log in to the Support Portal to view some of these articles.

AHV creating disk with 4k physical sector size. I have a requirement to create 512-bytes sector size. Finding no options parameter with vm.disk_create. Is there any other way to achieve this requriement. 

I created a Project in Prism Central and assigned a bunch of users to it, expecting them to be able to provision their own VMs with cloud-init configs customised to their liking.I found this awesome example code https://www.nutanix.dev/code_samples/create-linux-vm-customised-with-cloud-init/ which shows it can be done with API calls, however, as a user in the Project, there does not seem to be any options in the PC UI that allow me to specify a custom script when creating a VM.Are custom scripts deliberately hidden when using Projects? How are my users supposed to customise their VMs?

I’m looking for a way to expand storage a 6-node VMware cluster (ESX 6.7 U3), adding one or more Storage-only node could be an option. But, as I know, Storage-only nodes come with AHV, so I’d like to understand how I could add those nodes to the existing cluster.Trying to picture the idea, I'm assuming that SO nodes would work as new storage pool and cannot be expanded the existing one, am I right? I'm looking for documentation that give me some direction about the idea I have in mind.Any idea about this? 

Hi Team,I am using “Nutanix_Cluster_as_Built_Windows_v3.1.1” but received below error.“C:\Users\anwarsa\Desktop\Nutanix_Cluster_as_Built_Windows_v3.1.1>generate_document.exe -c "Microland Ltd." -n https://<FQDN>Enter Nutanix Cluster loginLogin: adminEnter Nutanix Cluster Password for login: adminPassword:Generating document for the https://<FQDN> cluster.ConnectionError - https://<FQDN>- HTTPSConnectionPool(host='https', port=443): Max retries exceeded with url: //<FQDN>:9440/api/nutanix/v1/cluster (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x00000000043540F0>: Failed to establish a new connection: [Errno 11001] getaddrinfo failed',))”looks like connection error , but wondering if anyone has script or latest version to try again .Thanks

Nutanix AHV is a bare metal Type-1 hypervisor developed by Nutanix. Nutanix AHV can be directly installed on any Nutanix certified OEM hardware server i.e SuperMicro NX, IBM CS , Lenovo HX , HPE ProLiant, Cisco UCS, Dell XC and many more being added along the way.AHV is built upon the CentOS KVM foundation and extends its base functionality to include features like HA and live migration, etc. Full hardware virtualization is used for guest VMs (HVM).In AHV deployments, the Controller VM (CVM) runs as a VM and disks are presented using PCI pass-through.  This allows the full PCI controller (and attached devices) to be passed through directly to the CVM and bypass the hypervisor. AHV does not leverage a traditional storage stack like ESXi or Hyper-V. All disk(s) are passed to the VM(s) as raw SCSI block devices. This keeps the I/O path lightweight and optimized.KVM Architecture:Within KVM there are a few main components: KVM-kmod KVM kernel module Libvirtd An API, daemon and manag

We’re over a hump day, and I would like to share an insight that may come handy to those of you that upload binaries to their clusters manually. Did you know that you have seven days to use the upgrade package that you uploaded to the cluster manually before it is removed from the file system? That’s right. Even if you have triggered pre-upgrade checks for the version, but the package is not used it deleted automatically. Make sure to clean up uncompressed software location and check the /home space on a CVM before you re-attempt the upload.KB-10300 Uploaded AOS upgrade binary is unavailable on Prism has more information To wrap your head around upgradesAll things considered — upgrade sequence and preaparation guidelinesFoundation of a successful upgrade - up to date FoundationNutanix upgrades FAQ: AOS, LCM, NCC, Hypervisors, Files

The Acropolis Operating System (AOS) provides the core functionality leveraged by workloads and services running on the platform Building upon the distributed nature of everything Nutanix does, we’re expanding this into the virtualization and resource management space.  AOS is a back-end service that allows for workload and resource management, provisioning, and operations. This gives workloads the ability to seamlessly move between hypervisors, cloud providers, and platforms. Acropolis Services: It is an internal service which runs on each CVMAn Acropolis Worker runs on every CVM with an elected Acropolis Leader which is responsible for task scheduling, execution, IPAM.Acropolis Leader Task scheduling & execution Stat collection / publishing Network Controller (for hypervisor) Acropolis Worker Stat collection / publishing VNC proxy (for hypervisor) Above diagram shows conceptual view of the Acropolis Leader / Worker relationship Dynamic SchedulerEfficient scheduling of resou

Volumes support ability to boot an Operating System over iSCSI for physical servers. In this configuration, a host can start a supported Operating System from a LUN instead of a Local Disk instance.This procedure describes how to configure Network Adapter BIOS Settings to enable this feature.Important Points:Refer to Intel Ethernet Adapter Vendor Documentation for more details about  Intel iSCSI Boot Configuration. According to Intel, we might be able to configure these settings through Adapter's Properties > Data Options Tab in Microsoft Windows Device Manager. Check Volumes Requirements and Supported Clients for a list of the supported network hardware and clients. Refer to Enable Nutanix Volumes and read about the procedures to perform on the Nutanix cluster. Before performing an iSCSI target discovery of the Nutanix cluster, configure BIOS boot settings for the network adapter as described in these stepsFor Step-By-Step Guide, Refer to Link

Below are new knowledge base articles published on the week of November 15-21, 2020.KB 10272 - Prism leader unavailable during LCM upgrade KB 10302 - Mounting ISO image on CVM for HPE platform KB 10308 - Xi-Leap: How to add public floating to a VM in XI Leap KB 10309 - Xi-Leap: How to update public IP of the Onprem VPN Gateway device KB 10311 - Installation of AHV 20190916.294 timeouts if done through PhoenixNote: You may need to log in to the Support Portal to view some of these articles.

A Nutanix cluster relies upon passwordless secure-shell (SSH) connectivity between the controller VMs (CVMs) and the hosts. If you are ever prompted for a password when attempting to connect from a CVM to a host using SSH (instead of being taken directly to the host shell), this could indicate that there is an issue with the SSH key exchange. This could also manifest as other issues such as a hypervisor upgrade failing due to the inability to copy the upgrade bundle to the host. However, please be aware that a prompt for a password could also indicate that a username is being attempted for connection which is not configured for passwordless authentication (i.e. not using the “root” username to login to an AHV or ESXi host).A host SSH key exchange issue can sometimes be resolved by verifying that an entry for the public key from each CVM is maintained within the authorized_keys file of each of the hosts. If an entry for any of the CVMs is missing, it can simply be added back with a manu

Hello,Can I please get an explenation of the different commands? Cheers!

Below are new knowledge base articles published on the week of November 8-14, 2020.KB 9233 - Alert - A150005 - AcropolisDefaultVSwitchError KB 9406 - Alert - A150004 - AcropolisVSwitchConfigFailed KB 9544 - NCC Health Check: conntrack_connection_limit_check KB 9675 - NCC Health Check: ahv_bridge_config_check KB 10180 - VM migration failures between AHV hosts due to network MTU mismatch KB 10208 - Alert - A130160- Host Network Uplink Configuration Failed KB 10209 - A6416 - Common port group between ESXi hosts is absent KB 10222 - Alert - A111076 - OVA Upload Interrupted KB 10223 - Alert - A200402 - vNUMA VM Pinning Failure KB 10238 - Nutanix Files Unavailable due to Stale ARP entries when ARP Flooding is disabled in Cisco ACI KB 10239 - Citrix VDI and Daylight Savings Time KB 10241 - Alert ID - A500104 - Entity Sync failed for the Availability Zone KB 10242 - Alert ID - A130149 - Guest Power Operation Failed KB 10245 - Unable to upgrade Era from 2.0 to 2.0.0.1 and above KB 10249 - Alert

Let’s say you want to know what are the uses and differences between Prism Element, Prism Central and Prism Pro   Nutanix Prism is the centralized management solution for Nutanix environments, however, Prism comes in different flavors depending on the functionality needed.   Prism Element: It is a service already built into the platform for every Nutanix cluster deployed. It provides the ability to fully configure, manage, and monitor Nutanix clusters running any hypervisors, however, It only manages the cluster it is part of.    Prism Central: It is an application that can be deployed in a VM or in a scale/out cluster of VMs (Prism Central Instance) that allows you to manage different clusters across separate physical locations on one screen and offers an organizational view into a distributed environment. Each Prism Central VM can manage 5,000 to 12,500 VMs, where Prism Central Instance (a three Prism Central VMs cluster) can manage up to 25,000 VMs.   Prism Pro: It is a feature set

Hi Team,We have three node (NX-8155 ) cluster connected to nexus switch.Vlan Tag for CVM and Host : 600Vlan Tag for Guest/Prod : 10All are in same subnet ( Host IP : 172.21.10.x/24 , CVM IP : 172.21.10.x/24  and Guest/Prod Network : 172.21.10.x/24  )We have allowed Layer-2 on Nexus, trunk port traffic and added both Vlans ( 600 and 10)Question:Can we keep all under same subnet ? If so how can I communicate inter-VLan , for the purpose of CVM to communicate NTP,DNS,Monitoring and Internet. Else Can I keep CVM, Host and Guest in same Vlan ? How this will impact broadcast domain and any issue with congestion. Else Can we have two different subnet range for (CVM, Hosts) and Guest with different Vlan (600) and (10)