Nutanix Cloud Manager

 Is it really possible that a networking issue, which exists on the other side of a large/vast network, could manifest locally on a host as NIC CRC errors (rx_crc_errors)?Yes!The way that frames are moved across a network with cut-through switching (which is the model used by current/high-performant data center switches) differs from the traditional store-and-forward model.With store-and-forward switching, frames are entirely received (and error-checked) on a switch before being passed along to the next switch inline. If errors are found within a frame, the frame is not passed along to the next switch.However, with cut-through switching, a switch is simultaneously receiving a frame and already passing it along to the next switch inline! Error checking is then only completed on a frame after it has already left a switch. Accordingly, any errors within the frame simply get passed along to the next switch inline until the frame reaches its final destination.In this way, when troubleshooti

Sometimes, through the normal operation of a hypervisor host, network interfaces (NICs) can become “overwhelmed” and be unable to respond to traffic fast enough as being served by an upstream network switch. This condition results in frames, which are subsequently being dropped from the receive buffer of the host, to go unprocessed or “missed”.Though it is generally not a good situation when a host (and/or a downstream element from the host, like a VM) “misses” network traffic, many applications are robust enough to handle a few misses from time to time. However, if this condition is frequent and/or perpetual, it can cause production issues and alarms would be exhibited from Prism accordingly.To combat a frequent/perpetual condition, there are several available options. For ESX and Hyper-V hosts, the receive buffer size of NICs can simply be increased. For AHV hosts, instead of increasing the receive buffer size, it is recommended to employ load-balancing across the available uplinks.

Hi,I'm looking up for a consolidated monitoring sheet if any available on Nutanix AOS and ESXI hyper-v integrated with some threshold applicable and severity will be really helpful. If someone has please share it.

When you are adding new features to your Nutanix environment, there might be a need to update the license or purchase a new one. If you do not update the license, there will be a red banner at the top of the Prism Page indicating that there are licensing violations.Clicking on “View Licensing Details” will show offenders of the current license. Using NCLI, you can see more details about features available in the current license level and features that are not allowed.For more information about disabling features, attaining the correct license, or temporarily removing the banner, look at KB 3443.

Question: Does nutanix use Self Signed Cert or CA-cert by default?  

Did you know that, as of Calm version 3.0 and later releases, all of your Calm data can now be backed-up? Previous versions of Calm, unfortunately, did not allow this functionality.All Calm data can now be backed-up to and restored from a single archive file. Taking a backup of the Calm data involves simply connecting to the Prism Central instance hosting the Calm service via CLI (i.e. via SSH), then connecting to the corresponding docker container (named “nucalm”) and executing a single command.More information regarding this procedure can be found within the “Taking Backup and Restoring Calm Data” section of the “Nutanix Calm Administration and Operations Guide”.

This article’s purpose is to provide assistance in determining the right amount of resources required to enable various services on Prism Central.Currently, there are multiple services available such as CALM, Karbon, Objects, Files, etc. Each of these services adds their own resource requirements to Prism Central hence, increasing the need for more resources for Prism Central in order to function properly. If enough resources are not configured, you might encounter the alert “Configured resource for the Prism Central VM is inadequate”. The NCC health check pc_vm_resource_resize_check verifies that the configured amount of memory and vCPU resources of the Prism Central VM is adequate. This check is introduced in NCC release version 3.10.0 and applies to Prism Central VMs (version 5.17.1 or higher) only (inapplicable to Controller VMs (CVMs) of Prism Elements clusters).Please refer to KB-8932 for more details.To learn more about the various services available and to determine the Prism C

Let’s face it, upgrades are daunting, and confusing, and frequent, and unavoidable, and, well, painful. Aiming to help with the preparation process and alleviate at least some of the worries Nutanix put together Acropolis upgrade (Upgrading AOS, Prism Central, Hypervisors, and Related Software Through The Web Console).Recommended Upgrade Order:Prism Central (PC): Upgrade and run NCC on Prism Central. PC: Upgrade Prism Central. PC: Run NCC. Prism Element clusters (PE): Upgrade and run NCC. PE: Upgrade Foundation. PE: Run and upgrade Life Cycle Manager (LCM): Perform an LCM inventory (also updates LCM framework). Do not upgrade any other software component except LCM in this step. PE: Upgrade AOS. PE: Run and upgrade Life Cycle Manager (LCM): Perform an LCM inventory (also updates LCM framework). Upgrade SATA DOM firmware (for hardware using SATA DOMs) as recommended by LCM. Upgrade all other firmware as recommended by LCM (BIOS / BMC / other). PE: Upgrade AHV for AHV clusters. PE: U

Have you been trying to keep your virtual machines (VMs) up to date with the latest versions of Nutanix Guest Tools (NGT), but face following hurdles:More difficult with seemingly ever-increasing release version activity? Tired of seeing alerts regarding a new version of NGT being available for a VM from within the Prism?Well, the good news is Prism Central is the answer for all of your NGT version management needs. Within Prism Central, NGT management has been greatly improved and simplified.For example, you can select particular VMs that are to receive an upgrade and employ this upgrade all at once (rather than one at a time and manually as through Prism Elements). Further, you can select the options to initiate the upgrade(s) immediately or, later, at scheduled time of your convenience.You can find more information about NGT management with Prism Central as per the “NGT MANAGEMENT IN PRISM CENTRAL” section of the Prism Central Guide.

First off, Prism Central RBAC SUCKS compared to vSphere. 11 years supporting VMware, which I am completely self taught, the the simple task of creating a role like the VMware 'VM Power User’ role has me ready to update my resume. This is maddening. I need a role that will be assigned to team members to manage all aspects of VMs except creation/deletion, and they need to be able to mount ISO images to the VMs. I have gone through the granular list of permissions, and I don’t see anything like that under VM or Images. Where is this hidden?

Hi Guys. Recently I made a S3 demo application which you can deploy in Nutanix Karbon using Calm, and it has web interface from where you can upload files directly to Nutanix S3 bucket. It's made using node.js platform. I tried to make everything streamlined using Nutanix Calm, so I made  Docker Host, Docker Registry blueprints (Docker registry seeded blueprint, doesn't work because it doesn't save the the registry's password properly), docker image build Runbook pipeline and blueprint for deploying S3 demo application to Karbon cluster.Whole project can be found here and includes full step-by-step instructions guide - https://github.com/vildek/s3demo Article about the demo - https://www.linkedin.com/pulse/building-s3-demo-application-nutanix-private-cloud-kaspars-vilde/?trackingId=7tjcuIKv0XKx95XEp%2FrD0w%3D%3D Hope this helps somebody understanding the platform better! Cheers.

This article aims to better explain the nuances of the alert “Disk space usage for one or more disks on controller VM” as seen in Prism and I have included some basic troubleshooting steps to identify the exact issue.  SSH into any CVM of the cluster and run the below command: ++ allssh “df -h”This will list the use % of all the disks controlled by the CVM. For ex: see screenshot belowNutanix reserves space in the SSD under /home for its infrastructure and is capped at 40GB, and sometimes it is possible to run low on the space usage and triggering an alert.   Check out this article for more information on how to reduce /home space usage.   The other disks /dev/sdx are the individual physical drives on each node. First we eliminate the possibility of a faulty drive by running a smartctl check.  ++ sudo smartctl -a /dev/sdx Replace x with the appropriate disk name from df -h output. If the smartctl test fails, then contact your vendor for disk replacement. Back to the df -h output, the s

Sometimes Kubernetes clusters are not listed when we try to open Karbon console from Karbon management GUI on Prism Central. This is due to Karbon (karbon_core & karbon_ui) containers being in an unhealthy state. This can be due to various reasons, one of them being docker containers are unable to get the proper resources or docker is unable to establish a connection with Karbon containers. Below is a screenshot of what one would notice when one would go to the Karbon console and no Kubernetes clusters are listed. Symptoms are container is unhealthy and here is the simple fix to try as a first step:  The Karbon container karbon_core is unhealthy. The Karbon container karbon_core is unhealthy.nutanix@NTNX-10-X-X-X-A-PCVM:~$ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 3cf0cb9a9dee karbon-core:v1.0.1 "/start.sh" 5 months ago Up 5 months (unhealthy)

How can we determine if Node is experiencing NIC issues and if so, then what? There are many reasons that can cause host NIC errors and troubleshooting this usually involves analysis to triangulate the issue to a certain part in the networking topology. Link flapping (interface continually goes up and down) Cable disconnect/connect Faulty external switch port Misconfiguration of the external switch port Faulty NIC port Faulty cable Faulty SFP+ module The two big error counters that we are concerned are rx_crc_errors and rx_over_errors (and in conjunction, rx_missed_errors/rx_fifo_errors).Run the following command on your host depending on the hypervisor:AHV:ethtool -S <eth> | egrep "rx_errors|rx_crc_errors|rx_missed_errors"ESXi:esxcli network nic stats get -n <vmnic> | egrep "Total receive errors|Receive CRC errors|Receive missed errors"Hyper-V:Get-NetAdapterStatistics -Name Ethernet*<interface number> | fl * Rx_crc_error:The sending host computes a cyclic

 Anecdote by @Kiran Surya  Nutanix users and administrator are often required to access the CVMs, Hypervisors and IPMIs through their IP addresses.  The most commonly used method to look at the IP addresses of these three entities is through Prism Element, and going through the Hardware tabs.  However, looking through the GUI can be time consuming since you are required to click on every Host to view the IP address details of all three components.  It’s all the more cumbersome if you have a huge infrastructure, like say 24 Hosts.  It would be ideal if there is an easier process to list all these pieces of vital information.   I happen to use the following combination of characters to list all these three pieces of data, and needless to say the customers are delighted when they see me run this command:  nutanix@NTNX-7B43JB2-A-CVM:10.171.154.127:~$ nutanix@NTNX-7B43JB2-A-CVM:10.171.154.127:~$ echo; set `ncli host list | grep Address | sed 's/^.*: //'`;echo -e "    IPMI\t    CVM\t\tHy

Hi All,   New here so I’m not sure this is the right place.  Is there playbook or way to automate disk expansion?  Can it expand the disk then trigger powershell to expand in the vm?    Thank you, Doug

Original plans do not always work out as intended. Goals change over time, new features are introduced. The modern world is an ever-changing one. You decide to migrate Prism Central VM (or even all three of them in case of a Scale Out PC deployment) to a different container but alas see no ‘migrate’ button. You can still do it.Find VM disk files Power off the VM/VMs Create a new Image for each VM disk  Create a new VM/VMs using acli or using Prism UI. Use image created in Step 3 as a source for the disk. Power on new VM/VMs and check functionality. If static IP address was used you may need to manually configure it one more time. Delete old VM if it is not needed anymore Delete Image if it is not needed anymoreThat’s right, these are the same steps used for migration of a regular (User VM) between containers.For a detailed process with commands and screenshots refer to KB-2663 AHV | How to move VMs between containers within an AHV cluster

Are there any recommendation for protecting Prism Central - I’m actually using PC Scaleout so have 3 PCVM’s and am curious if I should place these in a Metro or Async Protection Domain in the event that we perform a datacenter failover.I know that this cluster wouldn’t be critically needed in such a scenario but it would be nice to have. Any pro’s/con’s to using either PD type (Metro vs Async) for this?

Q.Which should be upgraded first BIOS, BMC or Host Boot Firmware?A. Please find below the upgrade order for all Nutanix firmware and software components. For hardware that has SATADOM, SATADOM should be upgraded first before the rest of the components. Always update BCM before the BIOS.Acropolis Upgrade Guide v5.17 Q. Can numerous upgrades be performed at the same time? For example, can I upgrade Boot Firmware on all five hosts in the cluster at once?A. You can select all components for all hosts to be upgraded. However, the upgrade will be done on each component of a host one by one. So only one component of one host will be upgraded at a time. When completed the LCM will proceed with the upgrade of the next component and so on. The non-critical and non-impacting components are upgraded first. Q. Should these upgrades be performed after hours? Is there an estimated timeframe for how long these upgrades take?A. No, the LCM upgrades are non-disruptive, so it does not require downtime.Th

Keeping track of hardware details across a cluster can be overwhelming. Some information could be found in Portal and other information might be found in IPMI.NCC provides a tool via the command line that allows for the accumulation of all the necessary details into a single output using the following command: ncc hardware_info show_hardware_info.You can use this command to see BIOS/BMC versions, DIMM Serial numbers, Disk firmware versions, etc. More information about the data collected can be found in KB 7084.This information is stored in the CVM for that specific node for access later and is collected by the sysstats utility every 24 hours. More information about the location can be found in the Support Portal Documentation.

Calm allows Nutanix Enterprise customers to seamlessly select, provision, deploy & manage their Business Apps across all their infrastructure, both private and public cloud. Nutanix Calm ties together an AppStore, App Lifecycle, Monitoring & Remediation by providing a single control-point for managing heterogeneous infrastructure, be it VMs or containers, or even bare metal servers. Calm will eventually support all the components required to manage a complete Software Defined Data Center.Calm is deployed alongside SSP in the Prism Central VM. Calm consumes multiple Nutanix internal services and is not a standalone component. By extension, users must have a Nutanix Prism Central VM to enable Calm functionalityUse Cases Automation for complex applications, not only provisioning, but also self service. Enable application self service for multiple teams. Setup complex applications with a single click. Quickly get dev/test environments ready to go. Prerequisites we need to con

Prism Central has Image management feature that allows pushing disk images to managed Prism Element instances based on configured image placement policies. The policies rely on categories that are associated with both images and clusters. Altogether this allows building a granular and intricate (if you chose to) set of image distribution policies.If for any reason using Image Management is not an option for you, you can push images from the PC to the clusters of choice manually either using PC CLI. Read further for details and instructions:Prism Central Guide v5.17: Image ManagementKB-4892 Prism Central: Manually pushing images uploaded to PC to multiple Prism ElementsKB-6205 Prism Central - How to create VMs on containers other than SelfServiceContainer

 I need to define a NIC for my windows substrate as a variable.current workflow:User selects subnet string from predefined variable >Pre-Create Parses and sets the subnet variable to the json from previous step. The NIC can be connected toone of n(x) subnets that are defined in the following escript:subnet1 = { "name": "name", "uuid": "redacted_uuid" }subnet2 = { "name": "name", "uuid": "redacted_uuid" }subnet3 = { "name": "name", "uuid": "redacted_uuid" }subnets = { "Preprod": subnet1, "Preprod": subnet2, "Prod-IaaS": subnet3 }print("{}".format(json.dumps(subnets.get("@@{appSubnet}@@"))))Output: @@{setSubnet}@@Profile Configuration:  This is where it breaks: 

Would you like to never see a PSOD screen again? I mean, who would? Elevated heart rate, restarted VMs, explaining the reason, searching for the reason, trying to distinguish a false positive from an actual alert. You definitely have better things to do.When looking at memory failures (DIMM is what I am referring to in this instance) uncorrectable memory errors is what brings most of the trouble and requires DIMM replacement but the majority of the errors are correctable.Each BIOS version brings improvement to handling hardware. That includes correcting memory errors. Meaning, you can avoid pesky hardware replacement windows for much longer if not forever.To keep your BIOS up-to-date use LCM. If you do opt for a manual process remember to always upgrade BMC first and only then proceed with the BIOS. BMC upgrades are non-disruptive. BIOS upgrade does require a host reboot.And as always, keep your NCC up to date (current NCC version is 3.10.0.1 and some false positive alerts in relation

I thought I’d put together a set of sources to help with troubleshooting some of the popular issues with the Prism.If you ever lock yourself out of Prism account do not fret. Local admin account can be unlocked by logging into a CVM with nutanix user and resetting the password for the admin user.KB-8130 Prism Admin Account Locked With "Account locked due to too many failed attempts." If you ever need to find a Prsim Leader node as part of any troubleshooting:KB-1841 Checking which node is the Prism Leader. To verfify connectivity status between Prism Central and Prism Element use NCC check cluster_connectivity statusKB-3379 NCC Health Check: cluster_connectivity_status. For troubleshootimg PE to PC connectivity follow KB-6970 PE-PC Connection Failure alerts.Expand the list in the comments, help the community!