How It works
Have questions about how the Nutanix Platform works? Looking to get started - start here!
- 1,298 Topics
- 1,981 Replies
Below are new knowledge base articles published on the week of July 5-11, 2020. KB 8935 - NCC Health Check: recovery_point_limit_exceeding_check KB 9479 - Deprecation of manual multihoming CVM network configuration KB 9621 - Xi Leap - After failover on Xi, floating IP is not getting assigned to the VM KB 9626 - AHV | Common VM NIC management workflows KB 9628 - AHV | Windows VM running in UEFI mode with SATA disk may fail to boot after disk expansion KB 9632 - 3rd party backup restore jobs can fail having INTERNAL ERROR: ErrorCode: 4 in AOS 5.15.1 KB 9640 - AHV | Cluster memory usage graphs may show different numbers if "HA Reservation" option is enabled Note: You may need to log in to the Support Portal to view some of there articles.
You may get an alert from NCC stating that the deduplication fingerprinting has been disabled, but it is enabled on a container: WARN: Fingerprinting is disabled by stargate, but configured on Storage Container(s) As we know, Stargate is the service that is managing storage on a Nutanix cluster, so Stargate has some conditions on which it will automatically disable deduplication. If the metadata usage is higher than 250Gb on a node. If the metadata usage is higher than 50% of the total metadata space on a node. The second condition is common on small size clusters, for example NX-1000 series, with single-SSD nodes if the SSD size is small. The metadata is located on the SSDs and the total metadata size on the node is equal to the capacity of: One SSD in single-SSD nodes Two SSDs in double-SSD nodes Four SSDs in all-flash nodes So, for example, if you have a node with 1 SSD which is 480Gb in size, 50% of the metadata capacity will be 240Gb (in fact it will b
Below are new knowledge base articles published on the week of May 24-30, 2020. KB 9390 - Alert - A130338 - ServiceBadScore KB 9431 - LCM operation on ESXi hosts fails with "LAG is configured with X uplinks however there are Y NICs added in the teaming policy" Note: You may need to log in to the Support Portal to view some of these articles.
Below are the top knowledge base articles for the month of May 2020. KB 7503 - G6, G7 platforms - DIMM Error handling and replacement policy KB 4116 - Alert - A1187, A1188 - ECCErrorsLast1Day, ECCErrorsLast10Days KB 1540 - What to do when /home partition or /home/nutanix directory is full KB 7604 - Disk space usage for root on Controller VM has exceeded 80% KB 4141 - Alert - A1046 - PowerSupplyDown KB 4409 - LCM: (LifeCycle Manager) Troubleshooting Guide KB 4158 - Alert - A1104 - PhysicalDiskBad KB 1113 - HDD/SSD Troubleshooting KB 2090 - AHV | Host and Guest Networking KB 2473 - NCC Health Check: cvm_memory_usage_check KB 8792 - NCC checks: same_hypervisor_version_check, duplicate_cvm_ip_check, same_timezone_check, esx_sioc_status_check, power_supply_check, orphan_vm_snapshot_check giving ERR KB 4519 - NCC Health Check: check_ntp KB 2486 - NCC Health Check: cvm_mtu_check KB 4273 - NCC Health Check: aged_third_party_backup_snapshot_check KB 3357 - NCC Health Check: ipmi_
Below are new knowledge base articles published on the week of May 31-June 6, 2020. KB 9442 - LCM BIOS/BMC Upgrade fails when node does not respond to IPMI power reset KB 9460 - Move: ESXi-AHV migration connection limits KB 9463 - Pre-upgrade check: Hypervisor Upgrade (test_host_upgrade_versions_compatible) KB 9467 - How to enable Karbonctl in Karbon darksite environment Note: You may need to log in to the Support Portal to view some of these articles.
What is the DIMM error? A memory error is an event that leads to the logical state of one or multiple bits being read differently from how they were last written. For example, If 1 was written in a memory cell and while reading the same memory cell, it returns 0. Memory errors can be classified into two types: Soft errors, which randomly corrupt bits but do not leave physical damage. Soft errors are transient in nature and are not repeatable. Soft errors can be because of electrical or magnetic interference (e.g. due to cosmic rays, alpha particles, leakage, random noise). Hard errors, which corrupt bits in a repeatable manner because of a physical/hardware defect or an environmental problem. Hard error can also occur if DIMM is not seated properly. All memory systems in use in servers today are protected by error detection and correction codes. These server machines employ error correcting codes (ECC), which allows the detection and correction of one or m
Below are new knowledge base articles published on the week of June 7-13, 2020. KB 9456 - Alert - A400114 - PolicyEngineServiceDown KB 9468 - Different Docker hosts can see volumes created with Nutanix DVP KB 9478 - How to clear stuck LCM inventory tasks ? KB 9480 - Windows 10, version 2004 or Windows Server, version 2004 VMs may fail to boot on AHV KB 9487 - AHV | VM update operations initiated from Prism Central may fail with "Entity CAS version mismatch" error due to missing machine_type attribute KB 9492 - Move VM IP address may change post deployment KB 9494 - Disks from ISCSI volumes change drive letters on Windows VMs after upgrade AHV 2016* to 2017* Note: You may need to log in to the Support Portal to view some of these articles.
Below are new knowledge base articles published on the week of June 14-20, 2020. KB 8507 - Cosmetic high latency spikes may be observed in Prism at time of low IOPS on a cluster KB 8932 - NCC Health Check: pc_vm_resource_resize_check KB 9423 - Beam Cost Governance for Nutanix On-Prem stops reporting cost analytics post Cluster upgrade to AOS 5.15 KB 9488 - AHV | CentOS/RHEL 6.8 may hang durin the boot when tboot package is used KB 9496 - Prism Central UI showing license expired red banner even after disabling prism pro features KB 9503 - Genesis crashing on a ESXI Node | No services are up | pyVmomi issue | ParserError: 'xml document KB 9516 - Using Nutanix objects as static web server for LCM dark sites KB 9519 - Nutanix Move | How to Configure HTTP(s) Proxy on Move KB 9534 - Hyper-V: Hyper-V 2019 fails during Foundation with "InstallerVM timeout occurred, current retry 0" KB 9538 - Prism logins fail if service account AD permissions are default KB 9545 - Copier reports inc
The Nutanix REST API allows administrators to create and run scripts on their Nutanix clusters. On a Nutanix cluster: API v1 and v2 runs through the Prism service. API v3 runs through Aplos services. Both of the above are services running on a CVM/PC-VM in a Nutanix cluster. Prism: Prism is the management gateway for component and administrators to configure and monitor the Nutanix cluster. This includes Ncli, the HTML5 UI, and REST API. Prism runs on every node in the cluster and uses an elected leader like all components in the cluster. All v1 and v2 API calls runs through Prism. Aplos: It’s an intentful orchestration engine + intentful API proxy. All v3 REST API calls runs through Aplos. Looking at the logs of the above 2 services can tell us a lot of things about our REST API and HTTP issues on PC-VM/CVM. The logs for these services are located on a CVM/PC-VM at: API v3 logs: ~/data/logs/aplos.out API v1 and v2 logs: ~/data/logs/prism_gateway.log HTTP defines a set of reque
Below are new knowledge base articles published on the week of April 26-May 2, 2020. KB 8482 - Prism shows a spinning icon preventing AHV upgrades on mixed (ESXi+AHV) clusters KB 8874 - Nutanix Files - Alert - A160053 - File Server Share Backup diff path translation failed KB 9111 - LCM update is disabled since there is an emergency update available KB 9215 - NCC Health Check: dimm_hppr_check KB 9240 - ESXi Host may not enumerate all NICs due to outdated drivers KB 9289 - Licensing Conversion - LOD (Life of Device) to CBL (Capacity Based) License KB 9298 - Using tcpdump tool on AHV host KB 9301 - Windows: How to Reset Local Administrator Password KB 9310 - How to Disable the Windows 10 First Sign-in Animation KB 9323 - Upgrade or imaging of Windows 2019 might fail due to vmms registration problems Note: You may need to log in to the Support Portal to view some of these articles.
Have you always wanted to get a peek into the network traffic flowing in and out of your Nutanix infrastructure? Whether you got your first IT job for years or been a veteran for several years, you would have definitely at some point been keen to get a detailed look at your network traffic. At Nutanix, the AHV hypervisor is based on the open-source KVM module and uses the CentOS Linux distribution. And yes, it comes with the popular packet capture utility tcpdump. A few examples when you would want to use tcpdump in your Nutanix infrastructure would be: VM(s) lose network connectivity on a particular host A host has multiple network links and you wish to identify which is active VM(s) are not reachable from an external network In that particularly annoying issue, the switch vendor asks you to contact Nutanix support The good news is that tcpdump is installed by default in AHV hosts and it can be run to inspect traffic on physical NIC’s as well as VM inte
Cover changing the default passwords on the Nutanix CVM (Controller VM), hypervisor, Prism Element and IPMI.
Find the best way to change all of the administrators password on Nutanix. This what I have done: #Changing the admin acct of PE ncli user reset-password user-name=admin password='$NewPassWD$@pe' #Changing the root acct of AHV for i in `hostips`;do echo "--$i--";ssh root@$i 'echo -e "$NewPassWD$@ahv" | passwd --stdin root';done #Changing the nutanix acct of CVM echo "$NewPassWD$@cvm" | sudo passwd --stdin nutanix #Change the ADMIN acct of the IPMI for i in `hostips`; do ssh root@$i 'for id in `/ipmitool user list |grep 'ADMIN '|cut -d" " -f1`; do echo $id; /ipmitool user set password $id "ADMINXXXXXXX"; done' ; done ncc health_checks system_checks default_password_check https://portal.nutanix.com/kb/6153
Below are new knowledge base articles published on the week of June 21-27, 2020. KB 9565 - [Nutanix Objects] Deploy failure due to docker pull issues via proxy KB 9566 - Windows VMs May Lose Network Connectivity With VirtIO 1.1.5 KB 9572 - [Karbon] Cluster deployment fails with error message "CDROM device must have IDE or SATA type adapter". KB 9578 - Hyper-V: SCVMM cannot migrate VM because version on destination host is not compatible Note: You may need to log in to the Support Portal to view some of these articles.
Below are the top knowledge base articles for the month of June 2020. KB 4116 - Alert - A1187, A1188 - ECCErrorsLast1Day, ECCErrorsLast10Days KB 7503 - G6, G7 platforms - DIMM Error handling and replacement policy KB 1540 - What to do when /home partition or /home/nutanix directory is full KB 4141 - Alert - A1046 - PowerSupplyDown KB 7604 - Disk space usage for root on Controller VM has exceeded 80% KB 4158 - Alert - A1104 - PhysicalDiskBad KB 1113 - HDD/SSD Troubleshooting KB 4409 - LCM: (LifeCycle Manager) Troubleshooting Guide KB 2090 - AHV | Host and Guest Networking KB 4519 - NCC Health Check: check_ntp KB 3357 - NCC Health Check: ipmi_sel_cecc_check KB 2473 - NCC Health Check: cvm_memory_usage_check KB 2486 - NCC Health Check: cvm_mtu_check KB 4273 - NCC Health Check: aged_third_party_backup_snapshot_check and aged_entity_centric_third_party_backup_snapshot_check KB 1863 - NCC Health Check: sufficient_disk_space_check KB 3741 - Nutanix Guest Tools Troubleshootin
Planning to upgrade the entire Nutanix environment? Here are a few tips and tricks to plan and perform the activity in a smooth manner. If you are new to Nutanix upgrades, don't hesitate to check this KB out to understand the overview of upgrades at Nutanix. When confused which component to upgrade first, below listed generic order can be of help. Step Component Target Reboots On the fly Quick links 1 NCC on Prism Central NCC framework Health service on PC Yes NCC 3.9 2 Prism Central PCVM Prism Central VM No PC 5.17 3 NCC on Cluster NCC framework Health service on all CVMs Yes NCC 3.9 4 AOS CVM All CVMs, one at a time No AOS 5.15 5 LCM Inv
Below are new knowledge base articles published on the week of May 3-9, 2020. KB 9220 - Changing ownership of applications, blueprints and users to other projects in CALM not supported. KB 9326 - "Open in Prism" button on the Insights Portal is deactivated KB 9340 - [Objects][MSP] Troubleshooting Deployment Failures due to docker pull issues KB 9341 - Calm: Cloning ESXi Template fails with "The resource 'xxx' is in use. com.vmware.vim.vpxd.dvs.portinuse.label" KB 9342 - Genesis crash-looping due to missing entries in iso_whitelist.json file Note: You may need to log in to the Support Portal to view some of these articles.
Logbay is a tool which helps in collecting logs sees fewer CVM/PCVM (Controller VM/PC VM) resources, such as CPU, memory, system partition /home disk space, and disk IO Easier and more flexible to use via cli, including for targeted time-windows or specific components Backward compatibility Logbay is supported from AOS 5.5.7 through 5.5.z, and from 5.9 and higher. AOS 5.5 through 5.5.6 requires the use of log_collector. Logbay is not supported with EOL versions of AOS and PC. By logging into any of the CVM we can run the log collector and collect logs for specific day, time and duration Check cluster time using following command $CVM: date Run the logbay command $CVM: logbay collect --from=2019/04/09-14:00:00 --duration=+6h15m Change the day,time and duration according to the requirement If no flags are given it will collect logs for last 4 hours The logs will be collected from each CVM and will be stored in $CVM:/home/nutanix/
When you run ESXi hypervisor on Nutanix you may get a question: should I install VMware tools or Nutanix Guest Tools (NGT) or both? Are NGT and VMware tools compatible between each other? The answer is yes, they are compatible and can be both safely installed on the same virtual machine. If you install NGT in guest VMs, you can use the following advanced features (NGT applications): File Level Restore CLI Nutanix VM Mobility VSS requestor and hardware provider for Windows VMs Application-consistent snapshot for Linux VMs You can read in more details about the NGT here. VMware tools in turn has different features and a different purpose. Installing VMware Tools eliminates or improves these issues: Low video resolution Inadequate color depth Incorrect display of network speed Restricted movement of the mouse Inability to copy and paste and drag-and-drop files Missing sound Provides the ability to take quiesced snapshots of the guest OS Synchronizes the time in the guest o
Below are new knowledge base articles published on the week of May 10-16, 2020. KB 9304 - Object Store Deployment Failure: type[CREATE]:code: 400, message: Primary MSP deployment in failed state status_code: 400 KB 9329 - ESXi | 1 click upgrade failing with 'Could not find a trusted signer' KB 9333 - Prism Central - Adding same email address under both "report settings" as well as "schedule settings" causes duplicate emails to be sent to recipient KB 9349 - [Nutanix Objects][Common Deployment Failure Scenario] Failed to create Envoy VM | Prism Central is not able to reach MSP and Envoy VMs or slow image download KB 9354 - Error: Node XXX cannot be removed: Cluster needs at least 5 usable nodes KB 9368 - Karbon UI stuck at loading in case of old browser version KB 9377 - [Karbon] Darksite Deployment fails in Proxy Environment in ETCD Deployment Stage KB 9383 - AHV | manage_ovs tool shows error "Failed to send RPC request. Retrying." when trying to change bridge configuration KB 9389 - D
Below are new knowledge base articles published on the week of June 28-July 4, 2020. KB 9466 - Pre-Upgrade Check: test_is_nos_upgrade_supported KB 9541 - SSR with error "Disk attached successfully but partition couldn't be mounted" KB 9564 - Cost Governance Memory Metrics KB 9575 - Prism Metro Availability error: Prechecks for starting metro availability failed for the protection domain to the remote site KB 9601 - [Karbon 2.1] How to create a Kubernetes cluster with Calico via API Note: You may need to log in to the Support Portal to view some of these articles.
This article covers the first component in a Generic Nutanix Environment Upgrade cycle - NCC. This is the first installation of the component-wise Upgrade guides series. Check out this article for a quick walk-through of the overview for a cluster-wide upgrade cycle. Nutanix Cluster Check (NCC) is a framework of scripts that can help diagnose cluster health. NCC runs on clusters and on Prism central. The workings of NCC on PC and the cluster are pretty much the same except the checks involved in the respective modules are different. NCC also includes the Logbay plug-in. Logbay is the utility used to collect logs from Controller VMs and hosts that are useful for troubleshooting issues. NCC Guide for the latest version (as of 7th of July, 2020) can be found here. To check for the release notes or Guides, navigate to Nutanix Support Portal > Menu> Documentation > NCC and filter out the NCC version from the top right section of the dropdown. Note: NCC upgrades are do
Quick tip for a hump day. Rushing in to check something quickly on the IMPI. Can’t be bothered with the Web interface. Jumping into a CVM CLI only to find out that the ipmitool isn’t working and the error isn’t helpful. Could not open device at /dev/ipmi0 or /dev/ipmi/0 or /dev/ipmidev/0: No such file or directoryGet Device ID command failedUnable to open SDR for reading Good news is the IPMI is most likely fine. Chances are all you need to do is to use a few extra keys as part of the ipmitool command. Remember that for all calls executed from a CMV the ipmitool has to authenticate with the IPMI and the calls are traversing the network technically even if it is only a virtual switch. KB-1345 Using ipmitool on the CVM gives full command syntaxis and is rich with examples. For Dell hardware refer to KB-6007 [ipmi] Could not open device at /dev/ipmi0 on Dell hardware. If you are a proud owner of AHV host there is an extra option available for you which is to check the ipmitool service
Below are new knowledge base articles published on the week of May 17-23, 2020. KB 4781 - A Windows 7 or Windows 10 User VM Fails to Power On with an NVIDIA vGPU Profile in ESXi 6.5b KB 9218 - Prism Element alternative UPN login fails KB 9357 - Oops - Server error when updating Container settings in Prism KB 9373 - AHV | 10Gbps NIC shows as 1Gbps despite auto negotiation enabled on Intel X550 NIC cards KB 9394 - Objects - Deployment might fail with Asymmetric routing / Policy Based routing configured in environment | Deployed cluster greyed out | IAM service is not healthy KB 9400 - AHV | Troubleshooting Virtual Machine boot failures KB 9403 - LEAP - Entity Sync tasks generated continously after PC upgrade to 5.16.1 KB 9407 - [Objects] Objects 2.1 Darksite Deployment failing at deploying metadata store KB 9420 - /tmp filled up by /tmp/paramiko_logs file KB 9422 - Error updating the current checks parameter. Health check schedule interval is invalid. KB 9425 - Official Guidan
Starting from AOS 5.10.2 and newer, if you run the ‘cluster status’ command, you may see that there is a new service running on the cluster called Xtrim. That service was not present before 5.10.2: Xtrim is the service that is going to improve the performance of the cluster. If we look a bit deeper, it improves the write performance by periodically trimming the SSDs. In Nutanix all the disks are passed-through from the host to the CVM and the CVM is managing the disks, so the Xtrim service running on CVMs is doing the trimming of the physical SSDs. So, what is trim? TRIM is a command with the help of which the operating system can tell the solid state drive (SSD) which data blocks are no longer needed and can be deleted, or are marked as free for rewriting. In other words, TRIM is a command that helps the operating system know precisely where the data that you want to move or delete is stored. That way, the solid state drive can access only the blocks holding the data. Further
Prism Central is a centralized management tool that runs as a separate instance comprised of either a single VM or a set of VMs. This multi-cluster view, allows you to monitor and manage multiple clusters through a single web console. If you would like to deploy a Prism Central instance, you can either follow the 1-Click method or the manual method. Do you have a Prism Central(PC) instance deployed in your environment and are planning to upgrade it? Here is the second installation of the Upgrade Series as promised to provide you quick tips and tricks to plan and execute your upgrade activities with ease. Prism Central Guide for the latest version (as of 10th of July, 2020) can be found here. To check for the release notes or Guide, navigate to Nutanix Support Portal > Menu> Documentation > Prism and filter out the PC version from the top right section of the dropdown. Note: PC upgrades is necessarily just a reboot activity on the Prism Central Virtual Machine(s)
Login to the community
Login with your account
Enter your username or e-mail address. We'll send you an e-mail with instructions to reset your password.