Intelligent Operations | Nutanix Community
Skip to main content

Intelligent Operations

Streamline Your Infrastructure Management and IT Operations

  • 438 Topics
  • 902 Replies

Automatically Send Nutanix Alerts to ServiceNow

  • 1
  • 0

Prims Pro 5 17 - A Short Tutorial

  • 1
  • 0

Prism Pro 5 17 What's New

  • 1
  • 0
438 Topics
G
Nutanix Employee
GaretNutanix Employee
posted in Intelligent Operations
Network Interface (NIC) CRC Errors and Cut-Through Switching

 Is it really possible that a networking issue, which exists on the other side of a large/vast network, could manifest locally on a host as NIC CRC errors (rx_crc_errors)?Yes!The way that frames are moved across a network with cut-through switching (which is the model used by current/high-performant data center switches) differs from the traditional store-and-forward model.With store-and-forward switching, frames are entirely received (and error-checked) on a switch before being passed along to the next switch inline. If errors are found within a frame, the frame is not passed along to the next switch.However, with cut-through switching, a switch is simultaneously receiving a frame and already passing it along to the next switch inline! Error checking is then only completed on a frame after it has already left a switch. Accordingly, any errors within the frame simply get passed along to the next switch inline until the frame reaches its final destination.In this way, when troubleshooti

0
G
Nutanix Employee
4 years ago
G
Nutanix Employee
GaretNutanix Employee
posted in Intelligent Operations
Network Interface (NIC) Receive (RX) Missed Errors

Sometimes, through the normal operation of a hypervisor host, network interfaces (NICs) can become “overwhelmed” and be unable to respond to traffic fast enough as being served by an upstream network switch. This condition results in frames, which are subsequently being dropped from the receive buffer of the host, to go unprocessed or “missed”.Though it is generally not a good situation when a host (and/or a downstream element from the host, like a VM) “misses” network traffic, many applications are robust enough to handle a few misses from time to time. However, if this condition is frequent and/or perpetual, it can cause production issues and alarms would be exhibited from Prism accordingly.To combat a frequent/perpetual condition, there are several available options. For ESX and Hyper-V hosts, the receive buffer size of NICs can simply be increased. For AHV hosts, instead of increasing the receive buffer size, it is recommended to employ load-balancing across the available uplinks.

0
G
Nutanix Employee
4 years ago
V
Venkata SainathExplorer
asked in Intelligent Operations
Monitoring Parameters for AOS intergrated with ESXi & Hyper-v

Hi,I'm looking up for a consolidated monitoring sheet if any available on Nutanix AOS and ESXI hyper-v integrated with some threshold applicable and severity will be really helpful. If someone has please share it.

1
R
Nutanix Employee
4 years ago
N
Nutanix Employee
nirmal.ravirajan-57912Nutanix Employee
posted in Intelligent Operations
Check Features Violating License

When you are adding new features to your Nutanix environment, there might be a need to update the license or purchase a new one. If you do not update the license, there will be a red banner at the top of the Prism Page indicating that there are licensing violations.Clicking on “View Licensing Details” will show offenders of the current license. Using NCLI, you can see more details about features available in the current license level and features that are not allowed.For more information about disabling features, attaining the correct license, or temporarily removing the banner, look at KB 3443.

0
N
Nutanix Employee
4 years ago
C
CarloNXTrailblazer
asked in Intelligent Operations
Self-signed certificates or CA-Cert

Question: Does nutanix use Self Signed Cert or CA-cert by default?  

4
C
4 years ago
AarthiKam
Nutanix Employee
AarthiKamNutanix Employee
posted in Intelligent Operations
Resource requirements for Prism Central

This article’s purpose is to provide assistance in determining the right amount of resources required to enable various services on Prism Central.Currently, there are multiple services available such as CALM, Karbon, Objects, Files, etc. Each of these services adds their own resource requirements to Prism Central hence, increasing the need for more resources for Prism Central in order to function properly. If enough resources are not configured, you might encounter the alert “Configured resource for the Prism Central VM is inadequate”. The NCC health check pc_vm_resource_resize_check verifies that the configured amount of memory and vCPU resources of the Prism Central VM is adequate. This check is introduced in NCC release version 3.10.0 and applies to Prism Central VMs (version 5.17.1 or higher) only (inapplicable to Controller VMs (CVMs) of Prism Elements clusters).Please refer to KB-8932 for more details.To learn more about the various services available and to determine the Prism C

0
AarthiKam
Nutanix Employee
4 years ago
Alona
Nutanix Employee
AlonaNutanix Employee
posted in Intelligent Operations
All things considered — upgrade sequence and preaparation guidelines

Let’s face it, upgrades are daunting, and confusing, and frequent, and unavoidable, and, well, painful. Aiming to help with the preparation process and alleviate at least some of the worries Nutanix put together Acropolis upgrade (Upgrading AOS, Prism Central, Hypervisors, and Related Software Through The Web Console).Recommended Upgrade Order:Prism Central (PC): Upgrade and run NCC on Prism Central. PC: Upgrade Prism Central. PC: Run NCC. Prism Element clusters (PE): Upgrade and run NCC. PE: Upgrade Foundation. PE: Run and upgrade Life Cycle Manager (LCM): Perform an LCM inventory (also updates LCM framework). Do not upgrade any other software component except LCM in this step. PE: Upgrade AOS. PE: Run and upgrade Life Cycle Manager (LCM): Perform an LCM inventory (also updates LCM framework). Upgrade SATA DOM firmware (for hardware using SATA DOMs) as recommended by LCM. Upgrade all other firmware as recommended by LCM (BIOS / BMC / other). PE: Upgrade AHV for AHV clusters. PE: U

2
Alona
Nutanix Employee
4 years ago
G
Nutanix Employee
GaretNutanix Employee
posted in Intelligent Operations
Simplified NGT Management with Prism Central

Have you been trying to keep your virtual machines (VMs) up to date with the latest versions of Nutanix Guest Tools (NGT), but face following hurdles:More difficult with seemingly ever-increasing release version activity? Tired of seeing alerts regarding a new version of NGT being available for a VM from within the Prism?Well, the good news is Prism Central is the answer for all of your NGT version management needs. Within Prism Central, NGT management has been greatly improved and simplified.For example, you can select particular VMs that are to receive an upgrade and employ this upgrade all at once (rather than one at a time and manually as through Prism Elements). Further, you can select the options to initiate the upgrade(s) immediately or, later, at scheduled time of your convenience.You can find more information about NGT management with Prism Central as per the “NGT MANAGEMENT IN PRISM CENTRAL” section of the Prism Central Guide.

1
M
4 years ago
D
David GriswoldVoyager
asked in Intelligent Operations
Where is the 'Mount ISO' permission when creating custom roles?

First off, Prism Central RBAC SUCKS compared to vSphere. 11 years supporting VMware, which I am completely self taught, the the simple task of creating a role like the VMware 'VM Power User’ role has me ready to update my resume. This is maddening. I need a role that will be assigned to team members to manage all aspects of VMs except creation/deletion, and they need to be able to mount ISO images to the VMs. I have gone through the granular list of permissions, and I don’t see anything like that under VM or Images. Where is this hidden?

1
Alona
Nutanix Employee
4 years ago
N
Nutanix Employee
NutonianNutanix Employee
posted in Intelligent Operations
Disk Space Usage High alert in Prism

This article aims to better explain the nuances of the alert “Disk space usage for one or more disks on controller VM” as seen in Prism and I have included some basic troubleshooting steps to identify the exact issue.  SSH into any CVM of the cluster and run the below command: ++ allssh “df -h”This will list the use % of all the disks controlled by the CVM. For ex: see screenshot belowNutanix reserves space in the SSD under /home for its infrastructure and is capped at 40GB, and sometimes it is possible to run low on the space usage and triggering an alert.   Check out this article for more information on how to reduce /home space usage.   The other disks /dev/sdx are the individual physical drives on each node. First we eliminate the possibility of a faulty drive by running a smartctl check.  ++ sudo smartctl -a /dev/sdx Replace x with the appropriate disk name from df -h output. If the smartctl test fails, then contact your vendor for disk replacement. Back to the df -h output, the s

0
N
Nutanix Employee
4 years ago
Sateesh
Nutanix Employee
SateeshNutanix Employee
posted in Intelligent Operations
Kubernetes cluster not listed in Karbon management GUI

Sometimes Kubernetes clusters are not listed when we try to open Karbon console from Karbon management GUI on Prism Central. This is due to Karbon (karbon_core & karbon_ui) containers being in an unhealthy state. This can be due to various reasons, one of them being docker containers are unable to get the proper resources or docker is unable to establish a connection with Karbon containers. Below is a screenshot of what one would notice when one would go to the Karbon console and no Kubernetes clusters are listed. Symptoms are container is unhealthy and here is the simple fix to try as a first step:  The Karbon container karbon_core is unhealthy. The Karbon container karbon_core is unhealthy.nutanix@NTNX-10-X-X-X-A-PCVM:~$ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 3cf0cb9a9dee karbon-core:v1.0.1 "/start.sh" 5 months ago Up 5 months (unhealthy)

0
Sateesh
Nutanix Employee
4 years ago
N
Nutanix Employee
NutonianNutanix Employee
posted in Intelligent Operations
Host Nic Errors

How can we determine if Node is experiencing NIC issues and if so, then what? There are many reasons that can cause host NIC errors and troubleshooting this usually involves analysis to triangulate the issue to a certain part in the networking topology. Link flapping (interface continually goes up and down) Cable disconnect/connect Faulty external switch port Misconfiguration of the external switch port Faulty NIC port Faulty cable Faulty SFP+ module The two big error counters that we are concerned are rx_crc_errors and rx_over_errors (and in conjunction, rx_missed_errors/rx_fifo_errors).Run the following command on your host depending on the hypervisor:AHV:ethtool -S <eth> | egrep "rx_errors|rx_crc_errors|rx_missed_errors"ESXi:esxcli network nic stats get -n <vmnic> | egrep "Total receive errors|Receive CRC errors|Receive missed errors"Hyper-V:Get-NetAdapterStatistics -Name Ethernet*<interface number> | fl * Rx_crc_error:The sending host computes a cyclic

0
N
Nutanix Employee
4 years ago
R
Nutanix Employee
rahul.khandalkarNutanix Employee
posted in Intelligent Operations
Automated listing IPMI, CVM and Hypervisor IPs in CLI

 Anecdote by @Kiran Surya  Nutanix users and administrator are often required to access the CVMs, Hypervisors and IPMIs through their IP addresses.  The most commonly used method to look at the IP addresses of these three entities is through Prism Element, and going through the Hardware tabs.  However, looking through the GUI can be time consuming since you are required to click on every Host to view the IP address details of all three components.  It’s all the more cumbersome if you have a huge infrastructure, like say 24 Hosts.  It would be ideal if there is an easier process to list all these pieces of vital information.   I happen to use the following combination of characters to list all these three pieces of data, and needless to say the customers are delighted when they see me run this command:  nutanix@NTNX-7B43JB2-A-CVM:10.171.154.127:~$ nutanix@NTNX-7B43JB2-A-CVM:10.171.154.127:~$ echo; set `ncli host list | grep Address | sed 's/^.*: //'`;echo -e "    IPMI\t    CVM\t\tHy

0
R
Nutanix Employee
4 years ago
Alona
Nutanix Employee
AlonaNutanix Employee
posted in Intelligent Operations
Moving Prism Central between containers

Original plans do not always work out as intended. Goals change over time, new features are introduced. The modern world is an ever-changing one. You decide to migrate Prism Central VM (or even all three of them in case of a Scale Out PC deployment) to a different container but alas see no ‘migrate’ button. You can still do it.Find VM disk files Power off the VM/VMs Create a new Image for each VM disk  Create a new VM/VMs using acli or using Prism UI. Use image created in Step 3 as a source for the disk. Power on new VM/VMs and check functionality. If static IP address was used you may need to manually configure it one more time. Delete old VM if it is not needed anymore Delete Image if it is not needed anymoreThat’s right, these are the same steps used for migration of a regular (User VM) between containers.For a detailed process with commands and screenshots refer to KB-2663 AHV | How to move VMs between containers within an AHV cluster

0
Alona
Nutanix Employee
4 years ago
M
mandgTrailblazer
asked in Intelligent Operations
Protecting Prism Central

Are there any recommendation for protecting Prism Central - I’m actually using PC Scaleout so have 3 PCVM’s and am curious if I should place these in a Metro or Async Protection Domain in the event that we perform a datacenter failover.I know that this cluster wouldn’t be critically needed in such a scenario but it would be nice to have. Any pro’s/con’s to using either PD type (Metro vs Async) for this?

4
Alona
Nutanix Employee
4 years ago
Alona
Nutanix Employee
AlonaNutanix Employee
posted in Intelligent Operations
LCM Frequently Asked Qestions (FAQ)

Q.Which should be upgraded first BIOS, BMC or Host Boot Firmware?A. Please find below the upgrade order for all Nutanix firmware and software components. For hardware that has SATADOM, SATADOM should be upgraded first before the rest of the components. Always update BCM before the BIOS.Acropolis Upgrade Guide v5.17 Q. Can numerous upgrades be performed at the same time? For example, can I upgrade Boot Firmware on all five hosts in the cluster at once?A. You can select all components for all hosts to be upgraded. However, the upgrade will be done on each component of a host one by one. So only one component of one host will be upgraded at a time. When completed the LCM will proceed with the upgrade of the next component and so on. The non-critical and non-impacting components are upgraded first. Q. Should these upgrades be performed after hours? Is there an estimated timeframe for how long these upgrades take?A. No, the LCM upgrades are non-disruptive, so it does not require downtime.Th

1
R
4 years ago
N
Nutanix Employee
nirmal.ravirajan-57912Nutanix Employee
posted in Intelligent Operations
Collecting Hardware information

Keeping track of hardware details across a cluster can be overwhelming. Some information could be found in Portal and other information might be found in IPMI.NCC provides a tool via the command line that allows for the accumulation of all the necessary details into a single output using the following command: ncc hardware_info show_hardware_info.You can use this command to see BIOS/BMC versions, DIMM Serial numbers, Disk firmware versions, etc. More information about the data collected can be found in KB 7084.This information is stored in the CVM for that specific node for access later and is collected by the sysstats utility every 24 hours. More information about the location can be found in the Support Portal Documentation.

0
N
Nutanix Employee
4 years ago
Alona
Nutanix Employee
AlonaNutanix Employee
posted in Intelligent Operations
How to push images from Prism Central to the clusters manually

Prism Central has Image management feature that allows pushing disk images to managed Prism Element instances based on configured image placement policies. The policies rely on categories that are associated with both images and clusters. Altogether this allows building a granular and intricate (if you chose to) set of image distribution policies.If for any reason using Image Management is not an option for you, you can push images from the PC to the clusters of choice manually either using PC CLI. Read further for details and instructions:Prism Central Guide v5.17: Image ManagementKB-4892 Prism Central: Manually pushing images uploaded to PC to multiple Prism ElementsKB-6205 Prism Central - How to create VMs on containers other than SelfServiceContainer

0
Alona
Nutanix Employee
4 years ago
Alona
Nutanix Employee
AlonaNutanix Employee
posted in Intelligent Operations
No PSODs! Never say 'good bye' to your host

Would you like to never see a PSOD screen again? I mean, who would? Elevated heart rate, restarted VMs, explaining the reason, searching for the reason, trying to distinguish a false positive from an actual alert. You definitely have better things to do.When looking at memory failures (DIMM is what I am referring to in this instance) uncorrectable memory errors is what brings most of the trouble and requires DIMM replacement but the majority of the errors are correctable.Each BIOS version brings improvement to handling hardware. That includes correcting memory errors. Meaning, you can avoid pesky hardware replacement windows for much longer if not forever.To keep your BIOS up-to-date use LCM. If you do opt for a manual process remember to always upgrade BMC first and only then proceed with the BIOS. BMC upgrades are non-disruptive. BIOS upgrade does require a host reboot.And as always, keep your NCC up to date (current NCC version is 3.10.0.1 and some false positive alerts in relation

0
Alona
Nutanix Employee
4 years ago
Alona
Nutanix Employee
AlonaNutanix Employee
posted in Intelligent Operations
Prsim troubleshooting cheat sheet

I thought I’d put together a set of sources to help with troubleshooting some of the popular issues with the Prism.If you ever lock yourself out of Prism account do not fret. Local admin account can be unlocked by logging into a CVM with nutanix user and resetting the password for the admin user.KB-8130 Prism Admin Account Locked With "Account locked due to too many failed attempts." If you ever need to find a Prsim Leader node as part of any troubleshooting:KB-1841 Checking which node is the Prism Leader. To verfify connectivity status between Prism Central and Prism Element use NCC check cluster_connectivity statusKB-3379 NCC Health Check: cluster_connectivity_status. For troubleshootimg PE to PC connectivity follow KB-6970 PE-PC Connection Failure alerts.Expand the list in the comments, help the community!

0
Alona
Nutanix Employee
4 years ago
Alona
Nutanix Employee
AlonaNutanix Employee
posted in Intelligent Operations
VM creation date and time - quick tip

Did you know you can determine when a VM was created? All you need is a Prism Central instance.In PC UI go to Virtual Infrastructure –> VMs –> List –> in Focus dropdown choose +Add Custom  and search  for “Created Timestamp”.You can also find the time stamp from PC CLI using nuclei.For details refer to KB-9045 How to find a VM creation date and time in Prism Central.  Always remember to configure adblocking plugin (if you are using one) in the Web browser to exclude Prism from checks. KB-8762 Prism home page shows "loading" when Web browser has AdBlock extension installed

0
Alona
Nutanix Employee
4 years ago
Alona
Nutanix Employee
AlonaNutanix Employee
posted in Intelligent Operations
When the weird gets weirder. Performance Troubleshooting.

Performance issues usually begin with the scene of an administrator looking at statistics, scratching the back of the head and saying ‘Wait, what?’ The issue can manifest itself in so many ways and the main struggle is to find the entities of the environment that experience issues at the same time and look at the environment layer by layer. Whether the bottleneck lies on the network, compute or storage layer, troubleshooting performance issue is always a tedious and time-consuming task. That is why Nutanix published an article (or two) to help you with the process. Where to look, what to look for and how to interpret the numbers. KB-2345 [Performance] Troubleshooting high CPU in Nutanix environments KB-5012 [Performance] Interpreting vCenter Performance CPU Ready chart values Tech TopX: ESXI Performance Troubleshooting

3
Alona
Nutanix Employee
4 years ago
G
GabrielPaganAbarcanVoyager
asked in Intelligent Operations
How do I get a log of who has logged into Prism

HI, recently someone upgraded a VM in our Nutanix cluster without documenting it. I know when it happened from the Tasks view in Prism. I need to konw if there is a log of who has logged into Prism to correlate but can’t seem to find this. Any ideas? Thanks!

1
Neel Kotak
Nutanix Employee
4 years ago
AarthiKam
Nutanix Employee
AarthiKamNutanix Employee
posted in Intelligent Operations
Prism Pro - you can predict the future!

Prism Pro is a set of additional features that are unlocked when Prism Central is licensed. These features include capacity planning, custom dashboards, and advanced search capabilities. The one-click planning feature that lets you forecast future workload growth so you can expand accordingly to meet the demands.Capacity PlanningThis feature ensures that environment never runs out of capacity. You can forecast shortages of CPU, memory, and storage, defining how much time until a given resource is exhausted. Information is based on machine-learned consumption behavior based on current and past utilization. This also provides a recommendation of scale out options.Just-in-time ForecastingAllows an environment to scale without over or under provisioning. This allows you to define additional workloads to add to an environment and identify whether the environment has resources to handle such workloads. If sufficient resources do not exist, Just-in-time Forecasting can recommend resources to

0
AarthiKam
Nutanix Employee
4 years ago
M
mikemaggioTrailblazer
asked in Intelligent Operations
Nutanix Patching

I”m trying to find out if Nutanix has a scheduled patching regimen similar to Microsoft’s Patch Tuesday. How are admins informed when there is an emergency patch? Do these include patches to Prism, to AOS and firmware patches? Thank you.

4
M
4 years ago

Badges

Show all badges
Terms & Conditions