Deployment Success

Reliable and Accurate Time Sync is mandatory for distributed services to work in a reliable / efficient manner.Network Time Protocol (NTP) is used across different devices and services on a network to maintain reliability and integrity of services, data and other critical functions.Nutanix - AOS, built on web-scale engineering principles, distributes roles and responsibilities to all nodes within the system to form a large cluster of services working together. Accurate time sync becomes a vital requirement for all the different components to work reliably and help keep up system integrity.Accurate time sync, not just offers integrity and smooth operations but offers a lot of value even when things don’t work as they should. During troubleshooting of any service, timestamps are used to understand and co-relate root-cause, impact of the problem.In order for a distributed system such as Nutanix AOS to work smoothly - NTP is of critical importance. CVMs (Controller Virtual Machine) that co

Just curious if there is going to be an update to the Move API documentation to explain how we can create a migration plan from ESX to AHV. Currently the only example shown is for AWS to AHV and when we try to substitute that payload with values from ESX it fails. Create plan example is here:http://{{ move-ip-address }}:8082/#/v2api/plans/createplan We’ve tried using this same template to create a plan for ESX, but it seems to choke on the network identification section with a return code of {'ApiVersion': '2.0.0', 'Code': 20512, 'Message': "Invalid Network Mapping for VM 'NX-TEST'. Ensure sufficient user permissions on source environment to fetch network information.", 'State': 'Error'}(we have validate permissions) This appears to be the part of the payload that it doesn’t like:From the AWS example: "Networks": [            {                "SourceID": "vpc-8e1c3de7",                "TargetUUID": "343eaa8f-a251-4ec0-8ede-729c97f7cddc"            } We’ve substituted these AWS specific

NCC check giving you warning that /home space usage is above the threshold(75%) and now you’re worried how will it affect your production and what all files and directories can you safely delete. Let us guide you through the steps to safely reduce the space usage of /home directory in the CVM. Before let us understand why you’re receiving this warning.Nutanix reserves space in the SSD under /home for its infrastructure and is capped at 40GB, and sometimes it is possible to run low on the space usage and triggering an alert.IMPORTANT NOTE → DO NOT treat the Nutanix CVM (Controller VM) as a normal Linux machine. DO NOT perform "rm -rf /home" on any of the CVMs. It could lead to data loss scenarios. Contact Nutanix Support in case you have any doubts.Now that we got that warning noted on a serious note, let us answer the question :  How can we reduce space usage? First, you need to get an idea of space usage.You can run the following script to parse the usage of /home partition and give u

We have shared a post on upgrade preparations and considerations. What now? When mapping upgrades it is only sensible to account for Nutanix Files as part of the process. What really happens once you click that button?First, the pre-upgrade checks will run to make sure that the cluster is able to be upgraded. If any of the pre-upgrade checks fail, you will see information about this in Prism and the actual File Server upgrade will not start. Users will have to click "Back to Versions" and start the upgrade again after the issue reported by the pre-checks is resolved. Once the File Server upgrade beings, each File Server VM is upgraded one-at-a-time onto the new Nutanix Files version. While an FSVM is down for the upgrade, users connected to shares hosted by this node may experience a loss of connectivity for a duration of roughly 20-30 seconds. After this short period, another FSVM will pick-up on hosting those shares, and users will regain access to their files. After each FSVM comple

 You might have noticed vCenter alarms regarding CVM memory usage being high,  and when you look at Prism console for the CVM, you will see it reports 100% but no prism alert will has posed up.This is a known visual artifact with passthrough VMs that could be bypassed by creating a customize alarm in vCetner for VM memory usage and apply it to all other VMs or to the folders containing the VMs. Disabling the builtin vCenter alarms in conjunction with enabling this customized alarm can by pass the alarm reported for VCM while continue reporting any VM high memory usage. For better understanding of this issue and its by pass please review the KB 4465 in your nutanix portal.  

What is the color of your Cluster Health Heart? We use the following stoplight style color code to report the quick view of the cluster health, in the form of a color coded heart.    Cluster Healthy     Cluster has at least one Warning Alert     Cluster has at least one Critical Alert  To view the specific alert(s), on the Health Dashboard click on any of the selections from the Home Dashboard, or click on the Heart directly to be taken to the Health Dashboard overview. Selection Options:Remote Sites Hosts Services Protection Domains …Once you have selected a specific Alert to review, you will have several options for the Alert.Run Check Turn Check Off Alert Policy ScheduleMake a note of the Alert name, and locate the Alert ID by searching for the Alert name in the Cluster Settings, Alert Policies view.  False Positive Alerts Occasionally there will be an alert that will inaccurately show a warning or failure. When this occurs there are a few options to correct

While much is shared and posted about LACP configuration in AOS* the questions “How?” remains open. Let’s take a look at the workflow:Connect to CVM via SSH. Make sure you are connected to correct CVM by checking its name and IP. Follow steps from KB 2852 to verify cluster health. Do not proceed if cluster cannot tolerate failure of at least 1 node. Put the node and CVM in the maintenance mode. Connect to host via IPMI, as LACP configuration process might cause network disconnect. Perform required configuration steps (enable/disable LACP). Once configuration is completed, make sure both host and CVM are accessible via network. Please also make sure that all NICs in bond are operational, by shutting down links one by one and verifying connectivity. If all tests are successfully completed remove CVM and node from maintenance mode.For detailed instructions as well as commands examples please refer to KB-3263.For complete command reference Acropolis Command-Line Interface (aCLI).* For more

Hi guys, I'm somewhat new to Nutanix and I was curious if there was a powershell command that will allow you to set the project for a VM. Essentially I want to use scripts to deploy VM's and then when the VM is deployed move it into a project so we can track resources. The deployment part of this is easy enough, but nothing in powershell seems to indicate it can even do this. Is this something that can even be done? I really like the powershell integration, and I really like the SSP Prism Central and Projects. I'd really really like to be able to use both of these together. Thanks

If you already have a functioning “Syslog Server” and you would like to forward cluster logs to it, you can follow instructions in:“https://portal.nutanix.com/#/page/docs/details?targetId=Advanced-Admin-AOS-v511:set-rsyslog-config-c.html”However, presently (as of fall of 2019) the activity entities like Alerts/Events/Audits in prism central cannot be redirected to remote syslog servers. This can be changed in future releases of PC and AOS will be reflected in Release notes.  

Let’s say you have Nutanix cluster with 24 CPU cores and 2 sockets and now you’re confused regarding the terminology.  vCPU and cores per CPU and how to provision CPU to a Virtual Machine confusing you? Let’s break the terminology down in simple terms!In the world of Hardware, we have sockets and cores. In a host, there would be 2 sockets(or CPU) and 12 cores in each socket, resulting in 24 cores.So what is a vCPU?vCPU corresponds to the number of sockets for the VM. Cores per vCPU correspond to cores in a socket, so in conclusion, if you have provisioned your VM with the following configuration:2  vCPU 4 core per vCPUIn this scenario, your VM would have 2 sockets and 8 cores in total.How can I provision my vCPU, is there a guide or documentation regarding it?Absolutely yes, please give the following article a readCPU Configuration So is there a way to overprovision my CPU?Absolutely yes, please give the following article a readCPU Oversubscription 

"cluster -f destroy" does not help. This is a new cluster, but during one click ESXi upgrade, tasks for stuck and now nothing can happen. I am fed up with tasks getting stuck and not completing, I would like to destroy the cluster and individually update ESXi on each node. We are runing XC740xd-12

Is there a KB or documentation providing the JSON format and any additional things that may be coded in? For example, in going through the install.nutanix.com portal to generate the JSON file it doesn't ask for NTP server of the hypervisor. In looking at the file, though, there's this line: "hypervisor_nameserver": "8.8.8.8", I'd like to have our Nutanix deployments in code as much as possible (right now we're looking to various scripts for vSphere configurations). Being able to configure as much as possible up front and possibly make this a template for future deployments.

When expecting to witness changes in storage utilisation performance, increase in free space in particular, please be aware that the change may not take effect immediately.There are two main types of Curator scans:Scheduled Scans Curator Full Scan — in 6 hrs after the last Full Scan Curator Partial Scan — in 1 hour after the last Partial Scan. Triggered Scans  — as a response to a situation in the cluster where Curator is urgently required.For more information:KB-2101, KB-2924.Posts here in NEXT community curator task type list or a search.Nutanix University video on Curator.Nutanix Bible on monitoring Curator tasks Application Mobility Fabric (AMF)

Suppose you have Erasure Coding (EC-X) enabled on all of your containers. Over time you find that workload structure and type has changed and is no longer suitable for EC-X. What do you do?Since migrating VMs between containers is a disruptive task alternative approach would be to disable EC-X on a container.In preparation for the task there are a couple of factors we recommend are taken into consideration:There will be increased I/O at the storage layer. Once EC-X is disabled on a container additional data will be written to the container and replicated. The influx of the data can be estimated by reversing EC-X gains. Expect temporary increase of storage utilisation in addition to calculations above. Parity data will not be removed immediately. Parity data size can be reverse calculated based on the same EC-X gains estimates.    There are no guardrails in terms of storage utilisation once EC-X is disabled. Please make sure there is sufficient space for the decoded data to be written —

Below are new knowledge base articles published on the week of October 13-19, 2019.KB 8018 - NCC Health Check: ns_proto_consistency_check KB 8094 - NCC Health Check: disk_status_check KB 8148 - NCC Health Check: cpu_avx_check KB 8315 - Files: chmod/chown on TLD fails on first mount with CentOS 6 KB 8356 - [Nutanix Objects] Objects 1.0.1 is missing from LCM Inventory KB 8358 - [Nutanix Objects] Unable to download access keys in browsers like Mozilla Firefox KB 8360 - LCM pre-check test_esx_ha_enabled failed but HA is actually enabled on vCenter KB 8362 - Alert - A130209 - IO failures to a data source in an external repository KB 8364 - Windows prompts for restart after migrating vms on a VDI cluster to a node with a different CPU generation KB 8366 - AHV | Guest VM running CentOS 7 may hang when CPU is hot added KB 8367 - AHV | VM may be restarted unexpectedly due to memory corruption KB 8371 - Nutanix Move - Upgrading using the offline bundle may show an unexpected error KB 8379 - AHV

I'm running 3 node cluster, based on 5.10 version (2019-02-11) There are some alerts for each node: Detected Incompatible AHV Version Detected older AHV Version I've checked the Settings-Upgrade menu - there it nothing available to upgrade for AHV and AOS.

You have a Nutanix cluster with ESXi as a hypervisor, configured with Virtual Distributed Switch and you have a need to configure restriction on bandwidth for different network types, like vMotion, Management, iSCSI etc in the event of contention. VMware provides a feature known as Network I/O Control(NIOC).Want to know more about NIOC and how it works in our environment? NIOC Guide in Nutanix So how many versions of NIOC are there and which is compatible with Nutanix?There are two versions of NIOCVersion 2 Version 3 NIOC version 2 was introduced after vSphere 5.5.VMware vSphere version 6.x introduced.NIOC v3, which provided additional features like artificial features limits and reservation on system traffic. In vSphere 6.0x, we recommend using NIOC v2 with vDS, instead of NIOC v3  due to CPU overhead and some abnormal behaviour during our internal testing and evaluation.  In vSphere 6.5 Update 1, Nutanix fully supports vDS in combination with NIOC v3.Want to know in-depth about our c

Should there be a need to change IP address settings of an existing vCenter Server registered with Prism Element or Prism Central, the goal can be achieved in three steps:Unregister the vCenter Server from the cluster before changing the IP address of the vCenter Server. After you change the IP address of the vCenter Sever, you should register the vCenter Server again with the new IP address with the cluster. Instructions for Prism Element Instructions for Prism Central Change vCenter Server IP address settings. Register your cluster with the new vCenter Server following instructions below: Managing vCenter Server Registration Changes in Prism Element Managing vCenter Server Registration Changes in Prism Central Optional steps:run NCC Health Check: check_vcenter_connection troubleshoot if necessary following KB-4492

Migrating a Nutanix cluster between vCenter servers is one of those tasks that are not executed that often but when they are it is vital that everything goes as smooth as possible. Ensure the cluster is in excellent health by executing NCC health check. Presence of a distributed vSwitch must be addressed in advance.From there it is relatively straightforward:Create a new vSphere cluster in the new vCenter Server. Set the vSphere cluster settings on the new vCenter Server to the Nutanix recommended configuration (Configuring HA, DRS, and EVC). Unregister the Nutanix cluster from the old vCenter Server. Move the hosts from the old vCenter Server to the new vCenter Server. Register the Nutanix cluster to the new vCenter Server.For more details please see KB-4876.

Let’s say you are running AOS version 5.10.7, and want to upgrade to a different Nutanix Files version or confused regarding which Prism Central version you should install or upgrade to?Nutanix gives you a feature, Software Product Interoperability, to easily identify the compatible versions between software like Prism Central and Nutanix files with the AOS.All we need to do is first select the Software Type(AOS, Nutanix Files or Prism Central) and then the target software type and then we can find the AOS version compatible.Points to be taken into consideration Prism Central does not follow the Long Term Support Release (LTS) and Short Term Support Release (STS) tracks. Nutanix Files (formerly known as AFS) does not follow the Long Term Support Release (LTS) and Short Term Support Release (STS) tracks. Refer to the following link to know more about LTS and STS tracksLTS/STS InformationInterested to know more?How upgrades happen in Nutanix environment  Let’s learn about upgrades Prism

Nutanix Move is a cross-hypervisor migration solution to migrate VMs with minimal downtime supporting three types of sources for migration: migration of VMs running on an ESXi hypervisor managed by vCenter; migration of EBS backed EC2 instances running on AWS; migration of VMs running on a Hyper-V hypervisor. Nutanix Move also supports the migration of AWS EC2 VMs to AHV on the Nutanix cluster. You can perform the following operations with Nutanix Move. Migrate powered on or powered off VMs. Note: For AWS, the migration takes place in powered on state. For ESXi, the powered state is retained. Pause and resume migration Schedule migration Schedule data-seeding for the virtual machines in advance and cut over to a new AHV cluster Manage VM migrations between multiple clusters from a single management interface Sort and group VMs for easy migration Monitor details of migration plan execution even at the individual VM level Cancel in-progress migration for individual VMs Migrate all AHV c

Let's say that you have 25 Gbps SFP+ Network card in your host and have created a Windows Virtual Machine with a 1 Gbps Nic card and now confused regarding the bandwidth for the virtual machine and how the Virtual Nic of the VM is different than the Host Nic when it comes to bandwidth and functionality. vNic is the software NIC emulation in the VM and is just a driver and in latest VMs the NIC is para-virtualised, which means that the NIC and Operating System is aware of the Host Physical NIC capability and understands that the vNic is just a driver. So what basically is this driver supposed to do? The vNic driver is really an API between the guest and the hypervisor so the vNic bandwidth is totally disconnected from any physical hardware. So what is the Physical NIC card then? Physical Nic is the physical network adapter connected to your physical switch and is responsible for the transfer of packets in your environment. So how Para virtualised vNic is better, does it have any advanta

Let say for example you're running ESXi 6.5 U1 and want to upgrade to the latest ESXi version but confused whether the current hardware and AOS will support the latest ESXi version.Nutanix gives you a feature to identify the compatability of different Hypervisors with NX environment easily using the Compatibility Matrix. You just need to visit the compatibility matrix page, filter by using your hardware and AOS and you will see the hypervisors which are supported.Still confused before the hypervisor upgrade?Refer the following document regarding general guidelines before the hypervisor upgrade.Hypervisor-upgrade-guidelines

What can be encrypted? What configuration is supported? At which layer the data is encrypted? Which layer encryption is more secure?Nutanix offers three options of data encryption:Data-at-Rest Encryption - Self Encrypted Drives (SEDs) - cluster's native or external KMS for software-only encryption. If the Controller VM cannot get the correct keys from the key management server (KMS), it cannot access data on the drives. If a drive is re-seated, it becomes locked. If a drive is stolen, the data is inaccessible without the KEK (key-encrypting-key) (which cannot be obtained from the drive). If a node is stolen, the key management server can revoke the node certificates to ensure they cannot be used to access data on any of the drives. Data-at-Rest Encryption - Software Only For AHV, the data can be encrypted on a cluster level. This is applicable to an empty cluster or a cluster with existing data. For ESXi and Hyper-V, the data can be encrypted on a cluster or container level. The c

Looking to strengthen security of your HCL cluster? Consider keeping only necessary ports open. Following is the list of firewall ports that must be kept open to successfully access the Nutanix cluster. Prism web console: 9440, 80 SSH to both CVM and Hypervisor: 22 Cluster remote support: 80, 8443 vCenter remote console: 443, 902, 903 from both the user host and vCenter vCenter from Prism web console: 443, 80 Citrix MCS: virtual IP, Port 9440 (TCP) Xtract for VMs (Move): ESXi hosts (TCP 443, 902); AHV (TCP and UDP 2049, 111) Following is the list of ports that must be kept open for the 1-Click upgrade. *.compute-*.amazonaws.com:80,443 release-api.nutanix.com:80 ntnx-portal.s3.amazonaws.com and s3*.amazonaws.com Information above is extracted from KB-1478 which also explains what to do when configuring the entire range of IP address for AWS is not acceptable and using FQDN wildcards is not an option supported by the firewall the environment. KB-1202 Lists port number