How It works

Planning to upgrade the entire Nutanix environment? Here are a few tips and tricks to plan and perform the activity in a smooth manner.   If you are new to Nutanix upgrades, don't hesitate to check this KB out to understand the overview of upgrades at Nutanix.   When confused which component to upgrade first, below listed generic order can be of help.   Step Component Target Reboots On the fly Quick links 1 NCC on Prism Central NCC framework Health service on PC Yes NCC 3.9 2 Prism Central PCVM Prism Central VM No PC 5.17 3 NCC on Cluster NCC framework Health service on all CVMs Yes NCC 3.9 4 AOS CVM All CVMs, one at a time No AOS 5.15 5 LCM Inv

Below are the top knowledge base articles for the month of June 2020. KB 4116 - Alert - A1187, A1188 - ECCErrorsLast1Day, ECCErrorsLast10Days KB 7503 - G6, G7 platforms - DIMM Error handling and replacement policy KB 1540 - What to do when /home partition or /home/nutanix directory is full KB 4141 - Alert - A1046 - PowerSupplyDown KB 7604 - Disk space usage for root on Controller VM has exceeded 80% KB 4158 - Alert - A1104 - PhysicalDiskBad KB 1113 - HDD/SSD Troubleshooting KB 4409 - LCM: (LifeCycle Manager) Troubleshooting Guide KB 2090 - AHV | Host and Guest Networking KB 4519 - NCC Health Check: check_ntp KB 3357 - NCC Health Check: ipmi_sel_cecc_check KB 2473 - NCC Health Check: cvm_memory_usage_check KB 2486 - NCC Health Check: cvm_mtu_check KB 4273 - NCC Health Check: aged_third_party_backup_snapshot_check and  aged_entity_centric_third_party_backup_snapshot_check KB 1863 - NCC Health Check: sufficient_disk_space_check KB 3741 -  Nutanix Guest Tools Troubleshootin

Hi Probably a very simply question and more of a sanity check. I have many Win 0S servers to perform OS upgrades on. Looking to do the following..   Ensure VM has ngt installed Power down VM Clone VM Power up clone, perform OS upgrade 2 weeks later delete original VM   I assume the powered down VM doesn't actively contribute to CPU usage but storage will be ring fenced. Therefore I will see storage decrease whilst I'm upgrading VMS and before I delete the original.   Any disadvantages to making the cloned VM my new production VM rather than the other way round?   Is it ok to delete the original VM which the clone is being based off?      

I’m running PC Scaleout (5.16.1.2) on ESXi and need to migrate all 3 nodes to a new datacenter cluster and datastore (same vcenter). Is there a recommended process for this or can I simply power down one of the 3 nodes, migrate it, power it up and wait for all cluster services to start, and then move on to the next one? 

Below are new knowledge base articles published on the week of June 21-27, 2020. KB 9565 - [Nutanix Objects] Deploy failure due to docker pull issues via proxy KB 9566 - Windows VMs May Lose Network Connectivity With VirtIO 1.1.5 KB 9572 - [Karbon] Cluster deployment fails with error message "CDROM device must have IDE or SATA type adapter". KB 9578 - Hyper-V: SCVMM cannot migrate VM because version on destination host is not compatible Note: You may need to log in to the Support Portal to view some of these articles.

Hi Guys - Looking for some info with the powershell cmdlets I am writing a powershell script to list out all the ahv host name and ip address| cvm  name and ip address impi host name| ip address in a cluster What is the command to get the cvm name and their ip address in a cluster. I am not quiet sure about “Cluster Type and Region when I issue  Get-NTNXCloudCvm command… Any help would be much appreciated!   Thanks, Mahendra.

Have a situation where you need to upgrade cluster components or restart CVMs or hosts or just wish to check if your cluster health is normal?   Below listed are a few quick checks to evaluate the cluster stability. Note that the commands listed are to be run from an SSH session to any one of the CVMs in the cluster or PCVM where applicable.   No. Check Purpose CLI Command GUI method Expected output 1 Cluster services (Cluster and PC) To ensure all services are running fine on all CVMs cluster status -N/A- No services should be in 'DOWN' state 2 Metadata ring structure To ensure that the metadata ring structure is stable nodetool -h 0 ring -N/A- All the nodes must be listed and should have 'UP' and 'NORMAL' states 3 Data Resiliency To ensure that the cluster c

Is there a cli command that will tell me which VMs do NOT have NGT installed? Is there a command that will show me CD-ROM status for all VMs? (e.g. No drive available, or drive not empty) I know about ncc health_checks ngt_checks ngt_installer_version_check Thanks  

Hello guys, currently we have converged infrastructure with Cisco UCS, netapp storage and virtualization with VMware , now we want to make this our DR and want to build primary DC with Nutanix HCI, so my question is can Nutanix AHV can replicate to converged DR site? Thanks....

Have you always wanted to get a peek into the network traffic flowing in and out of your Nutanix infrastructure? Whether you got your first IT job for years or been a veteran for several years, you would have definitely at some point been keen to get a detailed look at your network traffic.    At Nutanix, the AHV hypervisor is based on the open-source KVM module and uses the CentOS Linux distribution. And yes, it comes with the popular packet capture utility tcpdump.   A few examples when you would want to use tcpdump in your Nutanix infrastructure would be: VM(s) lose network connectivity on a particular host A host has multiple network links and you wish to identify which is active VM(s) are not reachable from an external network In that particularly annoying issue, the switch vendor asks you to contact Nutanix support   The good news is that tcpdump is installed by default in AHV hosts and it can be run to inspect traffic on physical NIC’s as well as VM inte

Anyone have any insights to share about using the filter option in the /alerts/list API method in Prism Central? Here is that method in the documentation: https://www.nutanix.dev/reference/prism_central/v3/api/alerts/postalertslist/ It looks like there isn’t much information available about the filter option. I have found a (very) little bit of information online about FIQL, but I have no idea what field names are available for alerts in Prism Central.

Is there a way I can inject a NIC driver into the Phoenix.iso?  We use a Broadcom 10Gb and the firmware file is not present on the Phoenix.iso so it can’t load the nic’s.

The Nutanix REST API allows administrators to create and run scripts on their Nutanix clusters. On a Nutanix cluster: API v1 and v2 runs through the Prism service. API v3 runs through Aplos services. Both of the above are services running on a CVM/PC-VM in a Nutanix cluster. Prism: Prism is the management gateway for component and administrators to configure and monitor the Nutanix cluster. This includes Ncli, the HTML5 UI, and REST API. Prism runs on every node in the cluster and uses an elected leader like all components in the cluster. All v1 and v2 API calls runs through Prism. Aplos: It’s an intentful orchestration engine + intentful API proxy.  All v3 REST API calls runs through Aplos. Looking at the logs of the above 2 services can tell us a lot of things about our REST API and HTTP issues on PC-VM/CVM. The logs for these services are located on a CVM/PC-VM at: API v3 logs: ~/data/logs/aplos.out API v1 and v2 logs: ~/data/logs/prism_gateway.log HTTP defines a set of reque

Hello Team, Do we have script which we can run to get the cluster details from Prism Central. Like Total CPU and usage CPU ,Total Memory and usage memory and same for storage, and also want to schedule that report in email. Please help me with script to get the details

We are using Image configuration to upload and convert the VHDXs and so far it works except when we attempt to upload 4K sector aligned VHDXs from an old HyperV cluster, the upload completes successfully but the conversion fails. Is it possible to migrate and convert 4K sector  aligned VHDXs to a Nutanix AHV cluster? Error:    

I am testing Postman for API calls and one of the errors I am getting is 502 - Oops Server Error, which is related to the Apache portion. Question is, which log contains this? I’ve checked all the logs in ~/data/logs/ and can’t find that response. 

Below are new knowledge base articles published on the week of June 14-20, 2020. KB 8507 - Cosmetic high latency spikes may be observed in Prism at time of low IOPS on a cluster KB 8932 - NCC Health Check: pc_vm_resource_resize_check KB 9423 - Beam Cost Governance for Nutanix On-Prem stops reporting cost analytics post Cluster upgrade to AOS 5.15 KB 9488 - AHV | CentOS/RHEL 6.8 may hang durin the boot when tboot package is used KB 9496 - Prism Central UI showing license expired red banner even after disabling prism pro features KB 9503 - Genesis crashing on a ESXI Node | No services are up | pyVmomi issue | ParserError: 'xml document KB 9516 - Using Nutanix objects as static web server for LCM dark sites KB 9519 - Nutanix Move | How to Configure HTTP(s) Proxy on Move KB 9534 - Hyper-V: Hyper-V 2019 fails during Foundation with "InstallerVM timeout occurred, current retry 0" KB 9538 - Prism logins fail if service account AD permissions are default KB 9545 - Copier reports inc

Hi, A customer moved his Nutanix Cluster (with Hyper-V) from a DC to another, after powering the Nodes up, the IPs of all Hyper-V host and CMs released, I logged it locally to Hyper-V hosts and configure the internal IP (192.168.5.1/28) and the external IP same like before shutting the cluster down. I repeated the previous step with CVMs, I went through cd/etc/sysconfigs/ and edit network-scripts file and added the the external IP in the eth0 and the internal one (192.168.5.2/28) in eth1. Now Hyper-V FC is working fine but cannot start VMs due to the Nutanix cluster issue, whenever I tried to start cluster from any CVM, I get this message “WARNING genesis_utils.py:1211 Failed to reach a node where Genesis is up. Retrying” Is there any way to fix this issue or to repair cluster configuration without disrupting existing data? Thanks in advance

Ques- There is 5 node cluster with 200 TB raw disk data, administrator want to enable eraser coding to obtain more space. What will we the usable space after enable eraser coding Options – 100, 125, 150, 175 TB ? Please share the calculation formula ?   Ques2 - Administrator want to perform DR test after 6 months, which snapshot he should use? Option – latest snapshot, oldest one  

Is it possible for a single bridge can have multiple bonds? in which usecase it will be applicable? -Jai

Below are new knowledge base articles published on the week of June 7-13, 2020. KB 9456 - Alert - A400114 - PolicyEngineServiceDown KB 9468 - Different Docker hosts can see volumes created with Nutanix DVP KB 9478 - How to clear stuck LCM inventory tasks ? KB 9480 - Windows 10, version 2004 or Windows Server, version 2004 VMs may fail to boot on AHV KB 9487 - AHV | VM update operations initiated from Prism Central may fail with "Entity CAS version mismatch" error due to missing machine_type attribute KB 9492 - Move VM IP address may change post deployment KB 9494 - Disks from ISCSI volumes change drive letters on Windows VMs after upgrade AHV 2016* to 2017* Note: You may need to log in to the Support Portal to view some of these articles.

Hi all, A newbie question. It seems I still have an old crash dump directory on one of my AHV hosts. A ls -lahtr /var/crash shows the single directory from back in April. At the time the issue was resolved, and the faulty DIMM that was causing the issue was replaced. That said, clearly the dump file was not removed. In terms of cleaning this up, is it OK to delete the dump directory within the /var/crash directory and then rerun the ncc health check. Or is there a better method for clearing crash dumps from Nutanix clusters. Many thanks, Rob

Please answer the below questions : When one cvm goes down(in a 10 node cluster with RF3) for 20 mints, then Guest VM’s new write and read IO would be served by anther's CVM and all these will be traveling across 10 g network. A : Will those new IOs served from local copy via another CVM or IOs will be served from the replica copy B: Will cluster starts to build a new replica to accommodate the missing copy or RF3   Ques 2 - When new Write IO request comes, first it will write the data on Oplog then synchronously send to other CVM’s Oplog. All host in clusters having 2 SSD and 6 HDDs. Will Write IO process by both SSD in every host or only one SSD is hsving Oplog partition active at the same time ?  I think there is only one oplog per CVM/host whether the host is aving all flash drive or 2 SSD and remaining HDD. But I am not sure about this statement, Please clarify..

Is it possible to recover data from an overwritten entity - does the overwritten VM or its storage still exist anywhere? I'm asking as I recovered a VM from a backup but realised in hindsight that the logs from the VM in its failed state would be useful for diagnosing the issue. Thanks in advance. 

What is the DIMM error?   A memory error is an event that leads to the logical state of one or multiple bits being read differently from how they were last written. For example, If 1 was written in a memory cell and while reading the same memory cell, it returns 0.    Memory errors can be classified into two types:   Soft errors, which randomly corrupt bits but do not leave physical damage. Soft errors are transient in nature and are not repeatable. Soft errors can be because of electrical or magnetic interference (e.g. due to cosmic rays, alpha particles, leakage, random noise).   Hard errors, which corrupt bits in a repeatable manner because of a physical/hardware defect or an environmental problem. Hard error can also occur if DIMM is not seated properly.    All memory systems in use in servers today are protected by error detection and   correction codes. These server machines employ error correcting codes (ECC), which allows the detection and correction of one or m