Deployment Success

What is Nutanix Guest Tool (NGT)? Nutanix Guest Tools (NGT) is a software based in-guest agent framework which enables advanced VM management functionality through the Nutanix Platform. The solution is composed of the NGT installer which is installed on the VMs and the Guest Tools Framework which is used for coordination between the agent and Nutanix platform. The NGT installer contains the following components: Guest Agent Service Self-service Restore (SSR) aka File-level Restore (FLR) CLI VM Mobility Drivers (VirtIO drivers for AHV) VSS Agent and Hardware Provider for Windows VMs App Consistent snapshot support for Linux VMs (via scripts to quiesce) This framework is composed of a few high-level components: Guest Tools Service Guest Agent The figure shows the high-level mapping of the components:    Important notes:  NGT uses TCP/IP network connectivity secured with SSL. The installation includes identifiers unique to the VM and the cluster, but you can pre-install NGT on a clone ba

Listing the recommended order of Upgrade for various components of Nutanix Cluster :   1. Start with Prism Central(PC) - (If Deployed) Upgrade NCC  Upgrade PC Run NCC health checks. Check for impacted clusters.   2. Then Upgrade Prism Element (PE): Upgrade NCC on all your clusters Run NCC health checks Correct any issues found before moving forward.   3. Upgrade Foundation to the latest on each PE.   4. Update Lifecycle Manager (LCM) Perform LCM inventory (Which updates the LCM framework to the latest software) During this time, Do not upgrade anything else on that cluster. This step only updates the LCM framework itself and also performs inventory. No reboots or upgrades to the server are performed.    5. Upgrade AOS on each PE cluster.   6. Update Firmware versions using LCM This step should be performed only if all the previous steps are complete and health checked.    7. Upgrade Hypervisor.    For AHV Clusters : Upgrade AHV to bundled version from the PE

Nutanix presents us with many management interfaces like HTML5(Prism), REST API, acli and ncli for managing and troubleshooting and maintaining work infrastructure.  We will look into how to access the Nutanix Command line Interface and what the capabilities and purpose of acli and ncli command-lets areaCLI: Acropolis Command Line Interface  Utility to create, modify and manage VMs in AHV. Provides extra abilities(commands) to manage AHV host networking, manual snapshot etc Cannot manage Nutanix Cluster and so we have ncli nCLI:  Nutanix Command Line Interface Utility to manage the entire Nutanix cluster operations. Ncli is more extensive and complex command set To access the CLI, You can install it on your local machine Check out how to install ncli on your local machine here. From any Controller VM SSH to any CVM as a nutanix user and type ncli and hit return to enter the ncli command shell and will be the same process for acli.Once inside the shell, you can view the list o

Reliable and Accurate Time Sync is mandatory for distributed services to work in a reliable / efficient manner.Network Time Protocol (NTP) is used across different devices and services on a network to maintain reliability and integrity of services, data and other critical functions.Nutanix - AOS, built on web-scale engineering principles, distributes roles and responsibilities to all nodes within the system to form a large cluster of services working together. Accurate time sync becomes a vital requirement for all the different components to work reliably and help keep up system integrity.Accurate time sync, not just offers integrity and smooth operations but offers a lot of value even when things don’t work as they should. During troubleshooting of any service, timestamps are used to understand and co-relate root-cause, impact of the problem.In order for a distributed system such as Nutanix AOS to work smoothly - NTP is of critical importance. CVMs (Controller Virtual Machine) that co

Let’s say you want to know what are the uses and differences between Prism Element, Prism Central and Prism Pro   Nutanix Prism is the centralized management solution for Nutanix environments, however, Prism comes in different flavors depending on the functionality needed.   Prism Element: It is a service already built into the platform for every Nutanix cluster deployed. It provides the ability to fully configure, manage, and monitor Nutanix clusters running any hypervisors, however, It only manages the cluster it is part of.    Prism Central: It is an application that can be deployed in a VM or in a scale/out cluster of VMs (Prism Central Instance) that allows you to manage different clusters across separate physical locations on one screen and offers an organizational view into a distributed environment. Each Prism Central VM can manage 5,000 to 12,500 VMs, where Prism Central Instance (a three Prism Central VMs cluster) can manage up to 25,000 VMs.   Prism Pro: It is a feature set

Many users are not aware that a recent change has been made to the default password setting of new Nutanix nodes. Specifically, the default password for the IPMI interface is now the serial number of the node itself (using capital letters). Please note that the node serial number is different from the block serial number. You can find more information regarding this change as per the Common BMC and IPMI Utilities and Examples Knowledge Base article.Also, if you desire to change the IPMI password, you can do so using the IPMI management utility located within the file system of the operating system running on the node. Further, you can even change the password, without having an operating system installed/running, by using the utility from a bootable DOS environment. You can find more information regarding this within the Changing the IPMI Password section of the NX Series Hardware Administration Guide.

What is the difference between the Redundancy Factor and Replication Factor? Redundancy factor 3 is a configurable option that allows a Nutanix cluster to withstand the failure of two nodes or drives in different blocks. By default, Nutanix clusters have redundancy factor 2, which means they can tolerate the failure of a single node or drive. The larger the cluster, the more likely it is to experience multiple failures. Redundancy Factor 3 requirements: Min 5 nodes in the cluster. CVM with 32GB RAM configured. For guest VM to tolerate a simultaneous failure of 2 nodes or 2 disks in different blocks, VM data must be stored on a container with replication factor 3. NOTE: Nutanix cluster with FT2 enabled, can host storage containers with RF=2 and RF=3. Redundancy Factor 2 requirements: Min 3 nodes in the cluster. CVM with 24GB RAM configured. Some background to understand Redundancy Factor. Cassandra Key Role: Distributed metadata store Description: Cassandra stores and manag

Shutting down and Restarting a Nutanix Cluster requires some considerations and ensuring proper steps are followed - in order to bring up your VMs & data in a healthy and consistent state. Nutanix is a Hypervisor agnostic platform, it supports AHV, Hyper-V, ESXi and XEN. This makes it all the more important to read the following Nutanix KB, which details the steps required to gracefully shutdown and restart a Nutanix cluster with any of the hypervisors. Nutanix KB : How to Shut Down a Cluster and Start it Again?  

Please consider the possibility of incorporating the existing IP scheme in the new infrastructure. If changing the IP address is the only option we can utilize a script to change the CVM IP address. You can use the external IP address reconfiguration script in the following scenarios: Change the IP addresses of the CVMs in the same subnet. Change the IP addresses of the CVMs to a new or different subnet.In this scenario, the external IP address reconfiguration script works successfully if the new subnet is configured with the required switches and the CVMs can communicate with each other in the new subnet. Change the IP addresses of the CVMs to a new or different subnet if you are moving the cluster to a new physical location.In this scenario, the external IP address reconfiguration script works successfully if the CVMs can still communicate with each other in the old subnet.  Following is the summary of steps that you must perform to change the IP addresses on a Nutanix cluster. 

Nutanix, like any other system is composed of several components including hardware, software and firmware. If a component has to communicate with another at any given moment, the connection has to be pre-integrated and supported. How do you keep-up and understand what is compatible with what? Easy! with our compatibility matrix. For example, you have a Nutanix block model NX-8035-G5 and you want to know the supported AOS and Hypervisor for the chosen Model. The matrix shows the AOS, AHV, and hypervisor compatibility for Nutanix NX and SX Series platforms and SW-Only Models qualified by Nutanix (such as Cisco UCS, Dell PowerEdge, HPE ProLiant and others listed here). For other platforms not listed here, such as Dell XC, Lenovo HX, and others, please see your vendor documentation for compatibility. The matrix is presenting the fields in a cascading style, to show a clear components dependencies. Also in the compatibility matrix, you can see the Nutanix Ready Solution where we show the

 Ever wondered what are some of the main services/components that make up Nutanix?The following is a simplified view of the main Nutanix cluster components.All components run on multiple nodes in the cluster and depend on connectivity between their peers that also run the same component. Cluster components ZeusKey Role: Access interface for Zookeeper·  A key element of a distributed system, zeus is a method for all nodes to store and update the cluster's configuration. This zeus configuration includes details about the physical components in the cluster, such as hosts and disks, and logical components, like storage containers. The state of these components, including their IP addresses, capacities, and data replication rules, are also stored in the cluster configuration.·  Zeus is the Nutanix library that all other components use to access the cluster configuration, which is currently implemented using Apache Zookeeper. MedusaKey role: Access interface for Cassandra·  Distributed sy

Want to know how to change the default credentials on the cluster? On Nutanix cluster’s if you have the default credentials you will receive an INFO message (default_password_check) in NCC health check informing you the same, you can change your CVM, Hypervisor and IPMI password using the guides below, you can also use script to change it on all nodes at once. Nutanix portal document  

Hi, A customer moved his Nutanix Cluster (with Hyper-V) from a DC to another, after powering the Nodes up, the IPs of all Hyper-V host and CMs released, I logged it locally to Hyper-V hosts and configure the internal IP (192.168.5.1/28) and the external IP same like before shutting the cluster down. I repeated the previous step with CVMs, I went through cd/etc/sysconfigs/ and edit network-scripts file and added the the external IP in the eth0 and the internal one (192.168.5.2/28) in eth1. Now Hyper-V FC is working fine but cannot start VMs due to the Nutanix cluster issue, whenever I tried to start cluster from any CVM, I get this message “WARNING genesis_utils.py:1211 Failed to reach a node where Genesis is up. Retrying” Is there any way to fix this issue or to repair cluster configuration without disrupting existing data? Thanks in advance

You have a nice OVF or an OVA image and you are ready to deploy that virtual appliance to AHV. There is just one more step required. Well, maybe more than one. It is not possible to import OVA and OVF files to AHV. Extraction of the files of the image as well as upload of the VMDK files is required prior to deployment.Please follow instructions provided in KB-3621 AHV: Import OVA and OVF image

Nutanix AHV is a bare metal Type-1 hypervisor developed by Nutanix. Nutanix AHV can be directly installed on any Nutanix certified OEM hardware server i.e SuperMicro NX, IBM CS , Lenovo HX , HPE ProLiant, Cisco UCS, Dell XC and many more being added along the way.AHV is built upon the CentOS KVM foundation and extends its base functionality to include features like HA and live migration, etc. Full hardware virtualization is used for guest VMs (HVM).In AHV deployments, the Controller VM (CVM) runs as a VM and disks are presented using PCI pass-through.  This allows the full PCI controller (and attached devices) to be passed through directly to the CVM and bypass the hypervisor. AHV does not leverage a traditional storage stack like ESXi or Hyper-V. All disk(s) are passed to the VM(s) as raw SCSI block devices. This keeps the I/O path lightweight and optimized.KVM Architecture:Within KVM there are a few main components: KVM-kmod KVM kernel module Libvirtd An API, daemon and manag

Nutanix VirtIO includes device drivers specifically used by Windows VMs hosted in the Nutanix environment to enhance their stability and performance. This concept is very similar to VMware Tools for ESXi environments.The VirtIO bundles various drivers including:Balloon Driver Ethernet Adapter RNG Device SCSI pass-through controller Serial Driver SCSI ControllerThe VirtIO package is found on the Support Portal under AHV (please select “VirtIO” from the corresponding drop-down menu).To note, the device driver versions contained within the various available Nutanix VirtIO packages may be the same if there have been no updates for the drivers between the package releases. To correlate the driver versions associated with each VirtIO package release, please reference KB 5491 in the Support Portal.Further to note, beginning with VirtIO package release version 1.1.6 all driver versions match the VirtIO package version.

In order to check Nutanix AOS, AHV or 3rd Party Hypervisor Compatibility with certified hardware, you can visit the Nutanix portal and select "Compatibility Matrix". Home > Documentation > Compatibility MatrixYou can check compatibility for:Hardware model, for e.g. NX, HPE, Dell, Cisco UCS AOS Version Hypervisor Version (AHV, ESXi, Hyper-V, XEN)You can filter by hardware model, AOS version or Hypervisor Version.Visit Compatibility Matrix (Nutanix portal account required) to check for hardware compatibility with AOS and Hypervisors (AHV / ESXi / Hyper-V / XEN).From the same page, you can also check AHV Guest OS compatibility as well. 

Giving thought to change your replication factor from 2 to 3? What are the impacts and things to consider?   First, let’s take a look at what replication factor is. Redundancy factor is a configurable option that allows a Nutanix cluster to withstand the failure of nodes or drives in different blocks.   By default, Nutanix clusters have redundancy factor 2, which means they can tolerate the failure of a single node or drive.  So RF3 means cluster can tolerate the failure of 2 nodes or drive… Basic Maths isn’t it?   Redundancy factor 3 has the following requirements: Redundancy factor 3 can be enabled at the time of cluster creation or after creation too. A cluster must have at least five nodes for redundancy factor 3 to be enabled. For guest VMs to tolerate the simultaneous failure of two nodes or drives in different blocks, the data must be stored on containers with replication factor 3. Controller VMs must be configured with a minimum of 28 GB(20 GB default+8 GB for the featur

After converting a VM running Windows 2012 r2 from VMware to Nutanix using Move version 3.4.1, we are receiving event IDs 257 and 259 regularly as shown below.Log Name:      ApplicationSource:        vmStatsProviderDate:          3/15/2021 8:35:34 AMEvent ID:      257Task Category: GeneralLevel:         ErrorKeywords:      ClassicUser:          N/AComputer:      ServerADescription:The "vmStatsProvider" can not be initialized. "vmGuestLib" returns error "VMware Guest API is not running in a Virtual Machine" (2). Log Name:      ApplicationSource:        vmStatsProviderDate:          3/15/2021 8:35:34 AMEvent ID:      259Task Category: Guest Library APILevel:         ErrorKeywords:      ClassicUser:          N/AComputer:      ServerADescription:Unable to start "vmGuestLibrary". Error: "VMware Guest API is not running in a Virtual Machine" (2).VMware Tools is still installed on the server however it won’t uninstall cleanly after the migration so I’ve disabled all of the ‘VMware’ services i

A traditional Nutanix cluster requires a minimum of three nodes, but Nutanix also offers the option of a two-node cluster for ROBO implementations and other situations that require a lower cost yet high resiliency option. Unlike a one-node cluster (see Single-Node Clusters), a two-node cluster can still provide many of the resiliency features of a three-node cluster. This is possible by adding an external Witness VM in a separate failure domain to the configuration (see Configuring a Witness (two-node cluster)). Nevertheless, there are some restrictions when employing a two-node cluster. The following links will provide you guide lines and information abut configuring the two node clusters:Two-Node Cluster Guidelines Two-Node ClustersAsk any questions to clarify any concerns about the the two node clusters.

All, I have a 6 Node 1065 system in 2 blocks.  Recently one of the CVMs (node 2, block A) crashed.  When rebooted it was just going in loops.  When diagnosed it seems the SSD (not the SATADOM) had failed and we replaced it.  When we try to boot the CVM, it still just loops. We were told to boot that node with Phoenix which the cluster provided me for download.  I do that and it doesn’t load Phoenix and gets errors instead.  I’m looking for a suggestion of how to get the node back to 100%.  At this point (and throughout) the ESXi on the SATADOM has booted fine and I guess if I didn’t care about the storage side I could just ignore this but I’d like the system to be fully healthy. Any suggestion about how to get the CVM working again would be appreciated. Thank you Johan

In some cases, you might have to permanently remove a physical node / host from a Nutanix cluster. There are two scenarios in node removal. Permanently Removing an online node Removing an offline / not-responsive node in a 4-node cluster, at least 30% free space must be available to avoid filling any disk beyond 95%. You cannot remove nodes from a 3-node cluster because a minimum of three Zeus nodes are required. Some Points to consider before initiating node removal: Sufficient Disk space available on other nodes in the cluster User Virtual Machine relocation (if required) Any software upgrade should not be running Checklist on verifying cluster health status Data resiliency is “OK” (green) in Prism Run a complete “ncc report” either from prism or CVM cli:                                 ncc health_checks run_all Depending on the size of data, node removal can be lengthy process, which involves relocating data from the node to other healthy nodes in the cluster. Node removal also remo

Although Prism is user friendly, it will be nice to know some commands that are useful for cluster management.   Cluster status The cluster status or “cs | grep -v UP” command gives the status(up or down ) of all the CVMs and their services. Running this command is helpful when there are alerts in the prism about services being down on certain CVMs.   Metadata ring The command “nodetool -h0 ring” shows if all the CVMs are a part of the metadata ring. Being in a metadata ring is essential for the CVM to be an active part of the cluster and handle its designated responsibilities. It is essential to run this command after every hardware replacement to make sure the node is back sharing responsibilities of the cluster.  Health check Most of the problems in the cluster are easily diagnosed by the health check command “ncc health_checks run_all” At the end of every WARN/INFO/ERR/FAIL plug in we see a KB to further troubleshoot the issue. This check is essential to make sure if the cluster he

The Prism interface allows the investigation of the disk I/O latency. As a result, following questions are raised. Note: Nutanix recommends that maximum latency readings should not be used as a measure of cluster performance and health. Average latency is a useful measure of cluster performance and health. What should be the average latency on a production cluster? What should be the maximum latency? What point is the latency too high? How to investigate the high latency? Consider the following for latency investigations. The end-user impact for any performance investigation. If the impact is not measurable by the end-user, then any investigation of performance statistics is going to reveal normal and healthy cluster operations. VM combinations, traffic type at the time, write or read size, sequential versus non-sequential, read versus write factors on which investigations are dependent. Latency Variables in a Nutanix ClusterThe following points provide you with the information

Let’s say you have Nutanix cluster with 24 CPU cores and 2 sockets and now you’re confused regarding the terminology.  vCPU and cores per CPU and how to provision CPU to a Virtual Machine confusing you? Let’s break the terminology down in simple terms!In the world of Hardware, we have sockets and cores. In a host, there would be 2 sockets(or CPU) and 12 cores in each socket, resulting in 24 cores.So what is a vCPU?vCPU corresponds to the number of sockets for the VM. Cores per vCPU correspond to cores in a socket, so in conclusion, if you have provisioned your VM with the following configuration:2  vCPU 4 core per vCPUIn this scenario, your VM would have 2 sockets and 8 cores in total.How can I provision my vCPU, is there a guide or documentation regarding it?Absolutely yes, please give the following article a readCPU Configuration So is there a way to overprovision my CPU?Absolutely yes, please give the following article a readCPU Oversubscription