5 Essential Tips for Maximizing Your Experience at Nutanix .NEXT for Bloggers
Listing the recommended order of Upgrade for various components of Nutanix Cluster : 1. Start with Prism Central(PC) - (If Deployed) Upgrade NCC Upgrade PC Run NCC health checks. Check for impacted clusters. 2. Then Upgrade Prism Element (PE): Upgrade NCC on all your clusters Run NCC health checks Correct any issues found before moving forward. 3. Upgrade Foundation to the latest on each PE. 4. Update Lifecycle Manager (LCM) Perform LCM inventory (Which updates the LCM framework to the latest software) During this time, Do not upgrade anything else on that cluster. This step only updates the LCM framework itself and also performs inventory. No reboots or upgrades to the server are performed. 5. Upgrade AOS on each PE cluster. 6. Update Firmware versions using LCM This step should be performed only if all the previous steps are complete and health checked. 7. Upgrade Hypervisor. For AHV Clusters : Upgrade AHV to bundled version from the PE
Network segmentation enhances security, resilience, and cluster performance by isolating a subset of traffic to its own network.The four primary means of network segmentation are the following:Isolating Backplane Traffic by using VLANs (Logical Segmentation) Isolating Backplane Traffic Physically (Physical Segmentation) Isolating Service-Specific Traffic Isolating Stargate-to-Stargate traffic over RDMAWe will discuss on how to secure traffic associated with a service (for example, Nutanix Volumes) by confining its traffic to a separate vNIC on the CVM and using a dedicated virtual network that has its own physical NICs. This type of segmentation therefore offers true physical separation for service-specific traffic.You can use Prism to create the vNIC on the CVM and configure the service to communicate over the dedicated virtual network. However, you must first manually configure the virtual network on the hosts and associate it with the physical NICs that it requires for true traffic
BACKUP AND RECOVERY:For Nutanix Clusters on AWS Nutanix provides the following backup and recovery options.Disaster Recovery for Instance, AZ, and Region Failure Nutanix has native inbuilt replication capabilities to recover from complete cluster failure. Nutanix supports both near-synchronous (NearSync) and asynchronous replication on an AHV cluster. Integration with Third-Party Backup Solutions Deploying a single cluster in AWS is great for more ephemeral workloads where you want to take advantage of performance improvements and use the same automation pipelines you use on-prem.Disaster Recovery for Instance, AZ, and Region Failure:Nutanix has native inbuilt replication capabilities to recover from complete cluster failure. Nutanix supports both near-synchronous (NearSync) and asynchronous replication on an AHV cluster.You can set your Recovery Point Objective (RPO) to be as little as one minute with NearSync, one hour with asynchronous and instant with synchronous replications. You
NUTANIX CLUSTERS REGISTRATION To register for Nutanix Clusters, you must have a My Nutanix account. A My Nutanix account allows you to access, manage, and use Nutanix Clusters. Your My Nutanix account is your first point of access to Nutanix Clusters. After you create a My Nutanix account, you can onboard into Nutanix Clusters either by signing up for one of the paid plans and applying Nutanix licenses at the same time.Pro-tip:You can also start a 30-day free trial of Nutanix Clusters. Nutanix Clusters Hybrid Subscription Model:You can sign up for Nutanix Clusters with any one of the payment plans (Pay As You Go or Cloud Commit) and apply Nutanix software licenses (that you might already have purchased or planning to purchase) such as Prism Pro, Files, and more at the same time. If you choose to apply licenses, the licenses are consumed first and then your payment plan falls back to the one you selected. If you do not have licenses or do not plan to purchase licenses, you can simply se
Hardware failures are a part of any datacenter lifecycle. The Nutanix architecture was designed with this inevitability in mind. A cluster can tolerate one or two failures (depending on the replication factor of the cluster or container) of a variety of hardware components while still running guest VMs and responding to commands through the management console. Many of these failures also trigger an alert through that same management console in order to give the administrator a chance to respond to the situation.Nutanix provides the ability to tolerate rack failures for extended data availability, in addition to drive, node, block, and network link failure.Block fault tolerance lets a Nutanix cluster make redundant copies of data and metadata and place the copies on nodes in different blocks.A block is a rack-mountable enclosure that contains one to four Nutanix nodes. All nodes in a block share power supplies, front control panels (ears), backplane, and fans.Nutanix offers block fault
Failures are part of everything and Nutanix Clusters is not immune to it. But how we plan for failures determines the versatility of the product or a person for that matter!!Nutanix categorizes the type of failures into availability domains essentially based on type of failure. Nutanix provides the ability to tolerate rack failure for extended data availability, in addition to drive, node, block and network link failure. Node FailureA Nutanix Node comprises Physical host and a controller VM. Both these components can fail without any impact to the Nutanix cluster.CVM failureWhen a CVM fails, an alert is generated in Prism and another CVM redirects the storage path on the related host to another CVM. Read and writes will occur over the 10GbE network until the CVM comes back online.It is business as usual for the end customer with maybe a slight performance decrease.Controller VM FailurePhysical Host failureIf a node fails, all HA-protected VMs can be automatically restarted on other nod
A Storage Replication Adapters (SRA) allows VMware Site Recovery Manager (SRM) to integrate with 3rd party storage array technology. Nutanix SRA is one such software module construct that allows SRM to interact with Nutanix clusters. This allows SRM to perform “Array Based Replication” using Nutanix replication, Data Protection and disaster recovery.SRM is an orchestration tool that allows us to perform recovery plans and runbook functionality. You can set up, test and perform pre and post recovery steps in case of a failover, Set the order in which the VMs come up and decide to change IP address if needed. SRM depends on vCenter and is licensed separately. Before you start, ensure the following. SRM and vCenter versions are compatible. Please visit the VMWare website to confirm compatibility. AOS SRA and SRM versions are compatible. Please refer to the Nutanix portal Nutanix SRA for SRM compatibility matrix to confirm compatibility. You will need 2 clusters managed by 2 vCenter
There are many views to esxtop. To change between these views, please use the following legendc = cpum = memoryn = networki = interruptsd = disk adapteru = disk devicev = disk VMp = power mgmtx = vsan V = only show virtual machine worldse = Expand/Rollup CPU statistics, show details of all worlds associated with group (GID)k = kill world, for tech support purposes only!l = limit display to a single group (GID), enables you to focus on one VM# = limiting the number of entities, for instance the top 5 The Memory panel displays server-wide and group memory utilization statistics. As on the CPU panel, groups correspond to resource pools, running virtual machines, or other worlds that are consuming memory The first line, found at the top of the Memory panel displays the current time, time since last reboot, number of currently running worlds, and memory over commitment averages. The memory over commitment averages over the past one, five, and fifteen minutes appear. Memory over commitment
The network panel in ESXTOP has a few metrics that will enable you to determine issues such as loss of network on a VM or high dropped packet on specific nics. Below is simple overview on how to use esxtop for networking: How to access:Log into your ESXi host and type esxtopPress ‘n’ to show only the networking view Port ID - The port ID used on the virtual switch. This ID is used for example for network traces with pktcap-uw.Press 'N' to sort by PORT-ID column.USED-BY - Indicates what is connected to the virtual switch port. Connected devices can be virtual machine NICs, VMkernel ports (vmk#), physical NICs (vmnic#) or ports used for health checks (Shadow of vmnic#).TEAM-PNIC - The physical NIC that the corresponding device is actively using. This information is helpful for network troubleshooting.%DRPTX - Dropped packets transmitted, hardware overworked. Possible cause : very high network utilization.%DRPRX - Dropped packets received, hardware overworked. Possible cause : very high
ESXTOP can be very good to tool to see the live stats and usage of the Physical host and its components, but more often than not, we would like to use that data to plot graphs or even save the data to examine it later. This brings us to the most important use case of ESXTOP is to run ESXi in batch mode. Batch mode provides us a way to collect esxtop stats from a physical ESXi host and analyze it on a Windows machine using Perfmon. Capture the Data:First thing would be capture only the relevant info and omit all the metrics you don't need.Use ‘f’ to ADD/Remove FieldsUse ‘o’ to change the order of Metrics Use ‘W’ to save the settings that you just edited.Caution: If you don't input a file name, the default settings will be saved as such. We are ready to run the esxtop in batch mode. esxtop -b -d 2 -n 100 > esxtop.csv b=batch moded=delayn=No of Iterationsa=all Metrics Usually capturing esxtop for an extended period of time can result in a large file. In order to avoid this, we can zip
About Xi Frame:Xi Frame is a secure cloud platform that lets enterprises and independent software vendors (ISVs) deliver applications, desktops, and software-defined workspaces to users. Users only require a connected device with a modern web browser. There are no clients, downloads, or plugins to install.Please follow this KB article for a step-by-step procedure for setting up and configuring your Cloud Connector Appliance manually.Note: CCA and WCCA VMs are automatically created in PC versions above 5.11 when deployed. Requirements Nutanix cluster running AHV with Acropolis Operating System (AOS) with Prism Central 5.10 or newer Frame Agent and Frame Cloud Connector Appliance (which you can find here) A Xi Frame subscription (sign up through https://my.nutanix.com/) Review the Network Configuration Requirements documentation which outlines required network protocol and port configurations. Note:If the CCA is already deployed, then just follow the steps in the Frame Documentatio
Here I discuss the effects of Enabling or Disabling Deduplication on a container even if the Container has data already written to it. The benefit of Compression and Fingerprinting+Deduplication is to hold more data in the container, by reducing the stored size and avoiding duplicate data, respectively.Nutanix’s intelligent selection of dedupable candidates prevents deduplication being performed where the benefit would be low. Deduplication Best Practices: Enable deduplication Do not enable deduplication Full clones Physical-to-virtual (P2V) migration Persistent desktops Linked clones or Nutanix VAAI clones: Duplicate data is managed efficiently by DSF so deduplication has no additional benefit Server workloads: Redundant data is minimal so may not see significant benefit from deduplication Enabling Dedupe:Fingerprinting is method of creating signatures of the data in Metadata. Fingerprint-on-write (Cache-Ti
In many ways, identifying the problem is harder than solving it. At least in IT.In order to better understand the performance of Nodes in a cluster and User VMs, ESXTOP for ESXi hypervisors and TOP for Linux OS based machines provide us an immediate birds eye view of the performance of the host. It is important to understand the output as a tool towards problem-solving. Here I aim to discuss scenarios you might need to isolate and troubleshoot High CPU observed at the Hypervisor or User VMs.ESXTOP : This command when run lists live CPU statistics specific to the Node the command is run on.esxtop outputIf you press “M” it shows you memory metric and “N” for network etc. As Always ‘H’ is for help. We will focus on a few CPU statistics:The output will show you all the VMs and the following metrics corresponding to them.Some Important ones discussed below:%USED, %RDY, %CSTP , %MLMTD and %SWPWT Note:To convert CPU ready % value to ms(milliseconds)CPU ready % = ((CPU summation ready value i
Nutanix AHV is a bare metal Type-1 hypervisor developed by Nutanix. Nutanix AHV can be directly installed on any Nutanix certified OEM hardware server i.e SuperMicro NX, IBM CS , Lenovo HX , HPE ProLiant, Cisco UCS, Dell XC and many more being added along the way.AHV is built upon the CentOS KVM foundation and extends its base functionality to include features like HA and live migration, etc. Full hardware virtualization is used for guest VMs (HVM).In AHV deployments, the Controller VM (CVM) runs as a VM and disks are presented using PCI pass-through. This allows the full PCI controller (and attached devices) to be passed through directly to the CVM and bypass the hypervisor. AHV does not leverage a traditional storage stack like ESXi or Hyper-V. All disk(s) are passed to the VM(s) as raw SCSI block devices. This keeps the I/O path lightweight and optimized.KVM Architecture:Within KVM there are a few main components: KVM-kmod KVM kernel module Libvirtd An API, daemon and manag
Nutanix AHV uses Open vSwitch(OVS) to connect to the CVM, the hypervisor and the user VMs to each other and to the physical network on each node. The CVM manages the OVS inside the AHV host. Since the OVS is an open source software switch that behaves like a layer-2 learning switch, it maintains a MAC address table. Each AHV server maintains an OVS instance, managed as a single logical switch through Prism. Bonds:When multiple uplinks are used they are added to a bond acting as a single logical interface, to which the bridge is connected. Open vSwitch (OVS) does not support bonds with single uplink and as workaround we directly connect bridge to single uplink.WARNING:Avoid the use of a single uplink configuration and do not attempt to modify a single uplink configuration using manage_ovs if the version of AOS is 5.10.x prior to 5.10.4. Warning:Updating uplinks can cause short network disconnect. It is strongly recommended performing network changes on a single node at the time after ma
Nutanix presents us with many management interfaces like HTML5(Prism), REST API, acli and ncli for managing and troubleshooting and maintaining work infrastructure. We will look into how to access the Nutanix Command line Interface and what the capabilities and purpose of acli and ncli command-lets areaCLI: Acropolis Command Line Interface Utility to create, modify and manage VMs in AHV. Provides extra abilities(commands) to manage AHV host networking, manual snapshot etc Cannot manage Nutanix Cluster and so we have ncli nCLI: Nutanix Command Line Interface Utility to manage the entire Nutanix cluster operations. Ncli is more extensive and complex command set To access the CLI, You can install it on your local machine Check out how to install ncli on your local machine here. From any Controller VM SSH to any CVM as a nutanix user and type ncli and hit return to enter the ncli command shell and will be the same process for acli.Once inside the shell, you can view the list o
Cluster creation or initialization is a process of bootstrapping the cluster by configuring the unconfigured nodes, load some node information into Zeus configuration file and start the services. Let's break it down. What is an Unconfigured node? Node which are factory shipped. Node which was removed from an existing cluster. These nodes are typically Pre-installed with CVM/HypervisorNO IPv4 address is configured But IPv6 link local address configured on eth0. This will always remain on a host. Before you begin to create a cluster with your brand-new nodes,You Must Have IPv4 Address configuration IPMI IP address Hypervisor IP address CVM IP address DNS/NTP IP address (Required while creating cluster via foundation) Hypervisor and CVM should be installed. What are the methods of Cluster initialization? Foundation: One click process for cluster creation Re-Images multiple nodes Assigns IP address on each node Manual Manual Hypervisor inst
So you have decided to relocate your Nutanix cluster to a different data center. Here are a few things to consider and a brief overview of steps to follow for seamless transition. Caution: This information is only provided to serve as a guide to plan your move. Please engage Nutanix Support if you have any concerns or questions following this process. Before you decide to move: Consider the possibility of incorporating the existing IP address schema into the new infrastructure by reconfiguring the router and switches instead of Nutanix nodes and CVMs. If that is not possible, proceed with this guide. Before you unplug everything:Refer to these guides for the procedure.Doc 1 (CVMs) CHANGING THE CONTROLLER VM IP ADDRESSESDoc 2 (AHV hosts) CHANGING THE IP ADDRESS OF AN ACROPOLIS HOSTDoc 3 (IPMI) CHANGING AN IPMI IP ADDRESS A few things to note:1. Since the cluster is being relocated and the new network will not be able to communicate with the old network, you will need to run through so
There could be many reasons why an administrator would want to perform a factory reset of the IPMI/BMC module. Whatever the reason maybe, knowing the different options and its differences is important.Using ipmitool, we can perform 3 types of resets each with its own implication. Reset BMC Partial factory reset (IPMI and BMC reset but will preserve Network settings) Full factory Default reset (Potential to lose IPMI network settings)Before you execute any of these commands, note the IP_Addr, Net mask and GW IP_Addr for the IPMI. ipmitool lan printUsage note: On VMware ESXi hypervisor, place a leading "/" in front of "ipmitool", for example, "/ipmitool". The executable is located at the top-level root directory.Check out the KB-8123 for detailed instructions.
Can you imagine performing firmware upgrades manually?Checking compatibility matrix for each component, procuring ISOs and staging them, take days to perform firmware upgrade and one can only hope that the host/cluster is still functioning after everything is said and done.LCM simplifies Nutanix IT infrastructure life cycle operations by consolidating software and firmware component upgrades into a unified control plane. LCM does all the hard work of managing all the upgrade dependencies for the software and firmware components. You don't have to worry about a thing. LCM normalizes firmware upgrades across hardware platforms and provides a single unified process regardless of vendor. Based on the bundle of components -- software and firmware-- that are selected for upgrade, LCM creates a plan that ensures the updates are deployed with the minimal number of service or host restarts.Get Started: Use Prism Central or Prism Element to open LCM Perform the LCM inventory (default is set to
This article aims to better explain the nuances of the alert “Disk space usage for one or more disks on controller VM” as seen in Prism and I have included some basic troubleshooting steps to identify the exact issue. SSH into any CVM of the cluster and run the below command: ++ allssh “df -h”This will list the use % of all the disks controlled by the CVM. For ex: see screenshot belowNutanix reserves space in the SSD under /home for its infrastructure and is capped at 40GB, and sometimes it is possible to run low on the space usage and triggering an alert. Check out this article for more information on how to reduce /home space usage. The other disks /dev/sdx are the individual physical drives on each node. First we eliminate the possibility of a faulty drive by running a smartctl check. ++ sudo smartctl -a /dev/sdx Replace x with the appropriate disk name from df -h output. If the smartctl test fails, then contact your vendor for disk replacement. Back to the df -h output, the s
How can we determine if Node is experiencing NIC issues and if so, then what? There are many reasons that can cause host NIC errors and troubleshooting this usually involves analysis to triangulate the issue to a certain part in the networking topology. Link flapping (interface continually goes up and down) Cable disconnect/connect Faulty external switch port Misconfiguration of the external switch port Faulty NIC port Faulty cable Faulty SFP+ module The two big error counters that we are concerned are rx_crc_errors and rx_over_errors (and in conjunction, rx_missed_errors/rx_fifo_errors).Run the following command on your host depending on the hypervisor:AHV:ethtool -S <eth> | egrep "rx_errors|rx_crc_errors|rx_missed_errors"ESXi:esxcli network nic stats get -n <vmnic> | egrep "Total receive errors|Receive CRC errors|Receive missed errors"Hyper-V:Get-NetAdapterStatistics -Name Ethernet*<interface number> | fl * Rx_crc_error:The sending host computes a cyclic
What is Nutanix Collector ?It is a desktop application that collects cluster statistics from your VMware or Prism cluster. This output can then be viewed in Nutanix collector or can be downloaded as an Excel file. You can then upload these Excel files into Nutanix Sizer to size and predict future workloads or just to collect statistics on current usage.Nutanix collector enables you to collect data in following methods: You can choose between: Data center/Cluster or All entities registered to vCenter Single Cluster or all entities registered to Prism Central What you need in order to install:You can install Nutanix collector on your desktop and then connect to either vSphere or Prism to collect statistics. Log in to Nutanix portal and download Nutanix collector desktop application package from Downloads>Essential Tools Install the package on your desktop What you can see:The Clusters Summary tab displays the configuration details and performance metrics of the clusters includi
Cluster lockdown is the ability to disable password based CVM access and/or only allow key based access to users.Nutanix recommends that access including SSH directly to CVM and hypervisor should be restricted to as few entities as possible. In high security settings, Cluster lockdown can be very appropriate and should be implemented Cluster Lockdown does not effect any cluster communication between its components. Cluster will function as normal.It ultimately boils down to requirements. You can choose the level of security and give access to only a select few when the cluster is in lockdown mode. Below are few helpful links that can help you secure your Nutanix Clusters : KB 1895 - How to create a password-less SSH login to your Nutanix clusterKey Based SSH access to CVMSecurity Guide
Ever wondered what are some of the main services/components that make up Nutanix?The following is a simplified view of the main Nutanix cluster components.All components run on multiple nodes in the cluster and depend on connectivity between their peers that also run the same component. Cluster components ZeusKey Role: Access interface for Zookeeper· A key element of a distributed system, zeus is a method for all nodes to store and update the cluster's configuration. This zeus configuration includes details about the physical components in the cluster, such as hosts and disks, and logical components, like storage containers. The state of these components, including their IP addresses, capacities, and data replication rules, are also stored in the cluster configuration.· Zeus is the Nutanix library that all other components use to access the cluster configuration, which is currently implemented using Apache Zookeeper. MedusaKey role: Access interface for Cassandra· Distributed sy
Already have an account? Login
Enter your username or e-mail address. We'll send you an e-mail with instructions to reset your password.
Sorry, we're still checking this file's contents to make sure it's safe to download. Please try again in a few minutes.
Sorry, our virus scanner detected that this file isn't safe to download.
