How It works

Below are new knowledge base articles published on the week of September 20-26, 2020.KB 8911 - Alert - Flow Visualization Statistics Collector Service Restart Detected - ConntrackStatsCollectorServiceRestart KB 9622 - NCC Health Check: pc_fs_inconsistency_check KB 10023 - Push images from one PC cluster to another PC cluster KB 10036 - Windows Portable Foundation Application may fail to start without any error on Windows 10 1809 and above. KB 10048 - Metric-server cannot scrape metrics on K8s cluster KB 10049 - Prism Central throwing "BEARER_TOKEN_BAD_SIGNATURE" error KB 10056 - Troubleshooting common issues when discovering a node’s BMC information for out-of-band power management in X-RayNote: You may need to log in to the Support Portal to view some of these articles.

Nutanix infrastructure customers often suffer performance issues on their database, and call in to Nutanix Technical Support to help them resolve them.  One aspect I have observed with database customers, and which impacts their performance, is the placement of the VMDKs on SCSI Controllers.  Assuming that there are four VMDKs serving the database VM, they might have configured the four VMDKs as follows.  This is an extract from the ESXi Host which the VM is running on:[root@esxasrk63u35:/vmfs/volumes/9f254e88-e3c751d/pqrsd00][root@esxasrk63u35:/vmfs/volumes/9f254e88-e3c751d/pqrsd00] cat pqrsd00.vmx | grep vmdk -A3scsi0:0.fileName = "pqrsd00.vmdk"sched.scsi0:0.shares = "normal"sched.scsi0:0.throughputCap = "off"scsi0:0.present = "TRUE"--scsi0:1.fileName = "pqrsd00_1.vmdk"sched.scsi0:1.shares = "normal"sched.scsi0:1.throughputCap = "off"scsi0:1.present = "TRUE"--scsi0:2.fileName = "pqrsd00_2.vmdk"sched.scsi0:2.shares = "normal"sched.scsi0:2.throughputCap = "off"scsi0:2.present = "TRUE"-

If there was a gflag applied in the past on cluster and now you dont have a track on the details, the same can be found out by running this NCC check ncc health_checks system_checks gflags_diff_checkThe output will show you the list of non-default gflag currently on the system, more details here -NCC Health Check: gflags_diff_check

Very frequently we receive an alert just before an upgrade that cluster doesn't have enough space to download the binary files (). If that is happening to you, there are ways to clean the partition below the threshold (75%). Some of them include below  Cleaning Old ISOs and Software Binaries ( For old AOS, NCC, Foundation versions) Checking Removing Old Logs (Log files shared for support cases if not cleared later, occupy space)   Cleaned up files from the approved directories but still see high usage in /home?At this time, open a support case to identify any other underlying issues or deep cleaning on the nodes with the help of Nutanix Engineer. More details on Nutanix KB below https://portal.nutanix.com/page/documents/kbs/details?targetId=kA0600000008dpDCAQ

Nutanix has worked closely to build integrations between ServiceNOW’s SaaS offering andNutanix infrastructure with the following areas of focus:ServiceNOW CMDB integration for Nutanix hyper-converged infrastructure discovery and modeling Nutanix Alert Integration with ServiceNOW Events and Incidents and remediation via XPlay ServiceNOW Request Management and Fulfillment for Services Hosted on Nutanix via CALM plug-in For More Details: https://www.nutanix.com/blog/nutanix-integration-with-servicenow

Ever wondered about the underlying mechanism used by AHV hosts to facilitate network communications for themselves and hosted VMs across a network? This underlying mechanism is an open-source software platform called Open vSwitch which operates as a software-defined switch on each host.This software-defined switch operates very similarly to a traditional layer-2 hardware switch in that it learns and maintains MAC addresses and makes frame-forwarding decisions based upon that information. However, it also can be considered much more scalable and extendable.With AHV, each host maintains its own Open vSwitch instance and parameters are passed to these instances using an open-standard protocol called OpenFlow.You can find more information regarding the Open vSwitch platform within the AHV Administration Guide and from the associated Linux Foundation website.

When you on-ramp new team members they’re going to need access to all their tools. If those new teammates will be managing Nutanix systems they most likely need a login to the support portal for access to documentation, downloads, the knowledge base, and the ability to open a support case.   So how do we set up access for new users? There are a few simple steps covered in the article How to Gain User Access to Nutanix Support Portal .   You’ll need a block or software-only serial number and the user’s email address. The email address should be in the authorized domain for your customer account.   To find the block serial number log into Prism and navigate to Hardware > Diagram. The block serial number is displayed above each block in the diagram. If there are multiple blocks, pick one and note the serial number for later. For software-only licenses, The software asset serial number is mentioned in the acknowledgement email sent upon the fulfilment of your order.   Once you have the

Below are new knowledge base articles published on the week of September 13-19, 2020.KB 9982 - LCM: Operation failed. Reason: Multiple LAGs as uplinks in a VSwitch is not supported. KB 9987 - HPDX Platforms Cabling Guides KB 10004 - LCM inventory does not show available firmware updates for HPE nodes with iLO version 2019.03.01 KB 10008 - AHV | Uploading vhdx image may fail if the image is corrupted KB 10012 - Cloning an image and Updating disk size via Prism Element API V2 call KB 10014 - Unable to see option to edit existing disk settings for any VM in PC when logged in as Cluster/Prism Central Admin user KB 10026 - Move - Migration failed with Bad/incorrect request, AOS 5.18Note: You may need to log in to the Support Portal to view some of these articles.

Compression is one of the key features of the Nutanix Capacity Optimization Engine (COE) to perform data optimization. Data Storage Fabric provides both inline and offline flavors of compression to best suit the cluster’s needs and type of data. As of 5.1, offline compression is enabled by default.Inline compression will compress sequential streams of data or large I/O sizes (>64K) when written to the Extent Store (SSD + HDD). This includes data draining from OpLog as well as sequential data skipping it. There is no impact to random I/O, helps increase storage tier utilization and benefits large or sequential I/O performance by reducing data to replicate and read from disk.Offline compression will initially write the data as normal (in an un-compressed state) and then leverage the Curator framework to compress the data cluster wide. When inline compression is enabled but the I/Os are random in nature, the data will be written un-compressed in the OpLog, coalesced, and then compresse

Representational state transfer ( REST ) Application Programming Interface (API) 3.0 is based on an intentful API philosophy. According to the intentful API philosophy the machine should handle the programming instead of the user enabling the datacenter administrator able to focus on the other task There are three publicly available Nutanix APIs. Note that while you may see API v0.8 listed in the REST API Explorer in Nutanix Prism, it is strongly recommended to use v3 APIs wherever possible.   v1 (Prism Element only) v2.0 (Prism Element only) v3 (Prism Central only) cURL Command AnalysisAs an extra step, let’s take the v3 API request above and look at what each part of the command is doing. If you are familiar with using cURL to make API requests, this will look very familiar. curl -X POST – Run cURL and specify that we will be making an HTTP POST request (as opposed to HTTP GET) https://[prism_central_virtual_ip]:9440/api/nutanix/v3/vms/list – Specify the complete request URL

Nutanix Cluster Check (NCC) is cluster-resident software that can help diagnose cluster health and identify configurations qualified and recommended by Nutanix. Depending on the issue discovered, NCC raises an alert or automatically creates Nutanix Support cases. NCC can be run provided that the individual nodes are up, regardless of cluster state.NCC actions are grouped into plugins and modules:A Plugin is a purpose-specific or component-specific code block inside a module, commonly referred to as a check. A plugin can be a single check or one or more individual related checks. A Module is a logical group of common-purpose plugins. It can also be a logical group of common-purpose modulesNCC Output: Each NCC plugin is a test that completes independently of other plugins. Each test completes with one of these status types:PASS FAIL WARN INFO ERRNCC can be executed via:a) GUIb) CLINutanix Cluster Check (NCC) 3.10 Guide Reference LinkLogbay Tags Reference Guide for Nutanix Cluster Check (

Nutanix recommends that you use the Prism Life Cycle Manager to perform firmware updates. However, firmware must be updated manually if:The component is located in a single-node cluster, OR The component is located in a multi-node cluster, but the hypervisor the cluster is running does not support LCM firmware updates.Manual Upgrades:Manually Updating SATA DOM Firmware Manually Updating M.2 RAID Hypervisor Boot Drive Firmware Manually Updating the BMC and BIOS Manually Updating HBA Controller Firmware Manually Updating Data Drive Firmware Manually Updating NIC FirmwareFor more details: https://portal.nutanix.com/page/documents/details?targetId=Hardware-Admin-Ref-AOS-v5_18:bre-manual-firmware-update-c.html

When attempting to hit LCM via ‘Software Upgrade’ screen via a Cluster in Central, it links you to LCM on central, not on the cluster. Is this intended behavior or a bug? 

The Prism interface allows the investigation of the disk I/O latency. As a result, following questions are raised. Note: Nutanix recommends that maximum latency readings should not be used as a measure of cluster performance and health. Average latency is a useful measure of cluster performance and health. What should be the average latency on a production cluster? What should be the maximum latency? What point is the latency too high? How to investigate the high latency? Consider the following for latency investigations. The end-user impact for any performance investigation. If the impact is not measurable by the end-user, then any investigation of performance statistics is going to reveal normal and healthy cluster operations. VM combinations, traffic type at the time, write or read size, sequential versus non-sequential, read versus write factors on which investigations are dependent. Latency Variables in a Nutanix ClusterThe following points provide you with the information

We may have a requirement to have a RPO as less as 1 minute.  The NearSync feature provides you with the ability to protect your data with an RPO of as low as 1 minute.To implement the NearSync feature, Nutanix has introduced a technology called Lightweight Snapshots (LWS) to take snapshots that continuously replicates incoming data generated by workloads running on the active cluster. The LWS snapshots are created at the metadata level only. These snapshots are stored in the LWS store, which is allocated on the SSD tier. LWS store is automatically allocated when you configure NearSync for a Protection Domain.Some of the advantages of NearSync are as follows. Protection for the mission-critical applications. Securing your data with minimal data loss in case of a disaster, and providing you with more granular control during the restore process. No latency or distance requirements that are associated with fully synchronous replication feature. Allows resolution to a disaster event in

NTP i.e. Network Time Protocol is a critical component of the Nutanix cluster and is crucial in keeping up hardware, processes, services and applications running with time synchronization across one another.  Inconsistencies in time synchronization could lead to undesirable consequences, not to mention the potential disastrous impact it could have on databases and real time applications through operational failures.  The downside of a poor NTP scenario could be data loss, hard to detect security breaches, even leading to legal liabilities, and loss of credibility.  NTP alerts are generated when you run the NCC Health checks, and are typically shown as below:Detailed information for check_ntp:Node 172.24.0.143:FAIL: The hypervisor is not synchronizing with any NTP server. This might occur if none of the configured NTP servers are available or you are currently experiencing network instability determined by the high offset/high jitter.Node 172.24.0.144:FAIL: NTP leader is not synchronizi

Witness VM in Metro Availability Configuration:A "Witness" is a special VM that monitors the Metro Availability configuration health. The Witness resides in a separate failure domain to provide an outside view that can distinguish a site failure from a network interruption between the Metro Availability sites. It can only be configured on AHV and ESXi hypervisors.The main functions of a Witness include:· Making a failover decision in the event of a site or inter-site network failure.· Avoiding a split-brain condition where the same storage container is active on both sites due to (for example) a WAN failure.· Handling situations where a single storage or network domain fails.To learn about the different scenarios you might encounter and the requirements of a witness VM, click here. Did you know?Witness VM can also be deployed if you are using a 2-node cluster! To learn more about how it works in that environment, click here.

Getting this error when I try to use putty/plink for any ACLI-related command:bash: ACLI: command not foundDoesn't seem to matter what acli “command” i run, even just running “acli” by its self gives the same error. I am able to run commands like “hostname” etc using plink/putty on the CVM. Feel like im missing something very simple here.Example command:putty $cvmIP -l $username -pw $password “acli vm.disk_get $vmname disk_addr=scsi.0”

Below are new knowledge base articles published on the week of September 6-12, 2020.KB 9960 - AD user cannot manage PC and NCC upgrades after upgrading to PC 2020.8 version KB 9980 - Cluster outage may occur during an LCM upgrade when restoring block fault-tolerance KB 9981 - SSP | VM nic update task performed by SSP user may fail on Prism Central 5.17 and above versions KB 9985 - Intermittent Network Disconnects Due to MAC Learning Disabled When Using AHV Bond Mode Balance-SLB KB 9989 - LCM : Could not find any upgrading entities forgroup Group-Cluster Service-Foundation KB 9995 - Move - Guest preparation may fail when SSL is enabled on Windows VMNote: You may need to log in to the Support Portal to view some of these articles.

Do you know Support Cases can be Scheduled on Support Portal?Support Portal users that are working with Nutanix Support can elect to schedule a follow-up for a case. The instructions within the case page are designed to be self-explanatory on how to schedule. However, you may want to understand how a case is handled when scheduled prior to using the feature. Portal Scheduling Overview The scheduling feature is only presented under the following conditions: Support contract allows for 24x7 coverage Case priority is low - P3 or lower Additionally, when scheduling, the following limits apply: Can only schedule for the next business day - not today Cannot schedule on weekends or public holidays Can only schedule up to 1 week out Your Systems Reliability Engineer (SRE) is able to work with you to assist with scheduling in other circumstances. Scheduling is available both at case creation and during the life of a case by using th

While doing LifeCycle Manager (LCM) upgrades, Nutanix infrastructure customers could encounter failures on the SATADOM upgrade on a ESXi Host, with the message:"Operation failed. Reason: Lcm prechecks detected 2 issues that would cause upgrade failures.Check 'test_hypervisor_config' failed with 'Failure reason: APD/VMCP feature is enabled on cluster, Please disable, Please refer KB 6470'Check 'test_min_phoenix_version' failed with 'Failure reason: APD/VMCP feature is enabled on cluster, Please disable, Please refer KB 6470' You can try disabling the APD using the following command on the Host:# esxcfg-advcfg -s 0 /Misc/APDHandlingEnable From experience I can tell you that this does not always resolve the issue.  It may be necessary, on certain occasions, to access the VMware vCenter, through possibly the vSphere Web Client, and do the following:Access the Hosts pageClick on the Host in questionClick on ManageGo to Advanced System SettingsSearch for “APD”Modify Misc.APDHandlingEnable an

The CVMs that comprise a Nutanix cluster get their time by syncing to a single member which is known as the NTP Leader (aka Genesis Master). This node will be responsible for syncing with whatever NTP servers the customer added to Prism. When NTP is properly configured, the Leader CVM will set its own clock to the time provided by the server and then all other CVMs will sync with the Leader's time.  For AHV clusters, the same list of servers used by the NTP Leader will be configured on each host to sync with independently. All other hypervisors need to have NTP configured separately using their unique management tools. By default User VMs sync their time with the local hypervisor where they are running, but users can configure their machines to sync with local or internet-based NTP servers as well. Importance of NTP: If there is a power outage and cvm’s pick up wrong time it can prevent the cluster to start It’s very difficult if you're trying to RCA any issue Commands:To check ntp s

A Cluster can be licensed following these simple stepsStep 1 is to download a Cluster Summary File(CSF)On your Prism home screen, click on the Gear Wheel (top right of the screen) --> Licensing --> Update License -->Download.The CSF will be generatedStep 2 is to generate a License Summary File(LSF)Login to your support portal account and continue with the steps belowClick on the Menu Icon ---> Licenses --> Licenses--> Manage Licenses --> Upload the CSF. Based on the product for which you are applying the license(AOS, Files, Prism Pro, Flow, Buckets etc), select the License type from the cards visible and click on Next, then download which will generate the LSF.Step 3 is uploading LSF to the Prism Access the Prism Console, under the Licensing page, Upload the LSF and click on Apply License, the licenses will be updated now. Note : Refresh(logout & re-login) the Prism for the licensing changes get applied.

Helloafter reinstalling Prism Central (PC) from v.5.16.1.3 to v5.17.1.1, using the procedure for unregistering PCUnregistering cluster from PChttps://portal.nutanix.com/page/documents/details?targetId=Prism-Central-Guide-Prism-v5_5:mul-vcenter-server-unregister-pc-t.htmlwe have some strange issues:- A PC REST API query return 2 available cluster (one for PE and one for PC, with different uuid on metadata)It is normal for API 3.1? - On Nutaniy API Explorer a simle query of vms "POST /vms/list"With small payload for "get_entities_request"{ "kind": "vm", "length": 10, "offset": 0}returns { "api_version": "3.1", "code": 500, "message_list": [ { "message": "Internal Server Error. Failed to get schema for cluster 000597eb-aa13-1c72-0000-000000029cf0", "reason": "INTERNAL_ERROR" } ], "state": "ERROR"}Where the displayed uuid is the correct PE cluster uuid.This is a lab cluster, then we have just one culster with < 50 VmsAny suggestion?Thanks  

I don’t see anywhere to acknowledge or remove this alert, it is no longer an issue and we resolved what was causing the issue but it keeps showing that we have a critical alert for the 28th.