Solved

AOS manual upgrade / “pre-upgrade” task

  • 22 January 2023
  • 3 replies
  • 44 views

Badge +1

 tl;dr how to perform pre-upgrade checks from the command line?

Hello Guys,
I have a question related to the performing manual / cli upgrade (command line). Here’s the scenario:

  • AOS with ESXi as a hypervisor,
  • multiple ROBO offices / Dark Site (no access to the internet),
  • Dark Site server with LCM,
  • very limited broadband: 16Mbit with static shaping (4Mbit) for management purposes and 12Mbit for business

It is very difficult to upload via AOS GUI 4-5GB of  upgrade and .json file. 3-4% / 10minutes and upload task informing about a ‘server issue’. I’m 99% sure that it related to the broadband shaping.

Solution is simple:

  • upload tar.gz install/upgrade bundle,
  • ‘untar’,
  • install ( ./install/bin/cluster -i ./install upgrade )

My question is: is any magic command to start pre-upgrade tasks with is normally presented in GUI if upgrade package is available? It will be very helpfull to start pre-upgrade check in-advance the main upgrade.

I tried to find the command in lab:
-- uploaded the bundle in ‘normal GUI’ way,
-- started pre-upgrade check from GUI
-- ps aux from CVM:
 

It looks that install/bin folder is keeping script “pre-upgrade”. I tried with different ways run it from the ‘nutanix’ account: /usr/bin/python2.7 -B /home/nutanix/software_uncompressed/nos/6.5.1.6/install/bin/preupgrade

No luck...

 

icon

Best answer by Moustafa Hindawi 22 January 2023, 10:56

View original

This topic has been closed for comments

3 replies

Userlevel 4
Badge +5

Hello @Dominik.Siedlak 

LCM Pre-Checks

test_aos_afs_compatibility

Verifies that AOS and Nutanix Files are compatible.

test_cluster_status

Verifies that the cluster statuscommand returns a healthy state.

test_cassandra_status

Checks zeus_config_proto (run zeus_config_printer for equivalent) for the following:

  • Node cassandra_status is not Normal.
  • No nodes are in maintenance.
  • The NCC check described in KB 4301 returns a PASS.

test_nutanix_partition_space

Checks that all CVMs have sufficient space on /home/nutanix.

test_network

Verifies that CVM IPs are reachable.

test_hypervisor_tests

Verifies that SSH can connect to the hypervisor.

  • AHV: verifies libvirtd on AHV hosts.
  • ESXi: checks if there is a VmkNic for all ESXi hosts in the same subnet as eth0/eth2. Verifies pyVim connectivity to ESXi hypervisor.

test_zk_status

Verifies that at least three zk hosts are available, on a cluster having three or more nodes.

test_replication_factor

Verifies that no containers in the cluster have RF=1.

test_under_replication

Checks for under-replication in the cluster.

test_prism_central_prism_element_compatibility

Checks that the registered Prism Central is compatible with Prism Element by reading prism_central_compatibility.json.

test_prism_central_ova_compatibility

Checks that Prism Central has boot and data partitions on different disks.

test_degraded_node

Verifies that no nodes are degraded.

test_vlan_detection

Verifies that LCM can fetch VLAN information from all nodes using NodeManager.

test_upgrade_in_progress

Checks to see if a non-LCM upgrade is in progress.

test_foundation_service_status_in_cluster

Checks to see if the Foundation service is running on any node in the cluster.

test_vmotion_support/test_live_migration_support

Checks that an ESXi or Hyper-V host has vMotion enabled.

test_cluster_config

Verifies that the cluster has vSphere HA enabled on vCenter mapped to a Nutanix cluster. (Make sure that you disable admission control.)

test_hosts_managed_by_same_mgmt_server

Verifies that the same management server (vCenter) manages all hosts.

test_hypervisor_config

  • All hosts: verifies that the host has sufficient space.
  • ESXi:
    • Verifies that the host does not have APD/VMCP enabled. (If yes, disable it.)
    • Verifies that no CD-ROMs (ISO image stored in local storage) or other physical devices are attached to the VM. (If yes, remove the physical device from UVM.)
  • AHV: verifies that the host can enter maintenance mode.
  • Hyper-V: none.

test_check_revoke_shutdown_token

Checks that the CVM can revoke the shutdown token after an LCM update completes on a node in a cluster.

test_ntp_server_reachability

Checks that the configured NTP servers are reachable before starting LCM updates.

test_prism_central_minimum_resource_check

Check that Prism Central is configured with the correct license and has enough memory to support all licensed features.

test_two_node_cluster_checks

Check that the configuration on a two-node cluster allows LCM updates.

  • Check that both nodes are up and the cluster is functional.
  • Check that the cluster has selected an LCM leader.
  • Check that the cluster has configured a witness VM.
  • Check that there are at least two meta disks available on each CVM.

test_zk_migration

Check to see if there is a Zookeeper migration in progress.

test_catalog_workflows

Check that the most commonly used catalog workflows for LCM are currently performable.

test_foundation_workflows

Check whether the current version of Foundation on the cluster is on the LCM denylist.

test_uniform_foundation_across_cluster

Check to make sure that every node in the cluster is running the same version of Foundation.

test_esxi_bootbank_space

Check that the ESXi bootbank has enough space for LCM upgrades that involve restarting into Phoenix.

test_esx_entering_mm_pinned_vms

Check to see if any node in the cluster contains a VM that cannot be live migrated to another node.

Note: For inventory scans, LCM skips the aos_afs_compatibility and under_replication tests, because inventory scans are not disruptive.

Badge +1

Hi Moustafa,
Thank you for answering me in such a short time 🙂 I am asking about a pre-checks related to the new release of the AOS and upgrade. I’ve downloaded upgrade package tar xzf to it with is keeping installer in the ‘install’ folder.

Install is very simple: start the command: “./install/bin/cluster -i ./install upgrade”

Process will start pre-check and if all is fine upgrade.
I  am asking how to start ‘Pre-upgrade’ from the command line instead of from GUI.

 

Command-line way of installation: it is showing progress in CLI and in GUI including health checks related to the upgrade:
 

 

I am asking if is any single command to process in-advance  pre-checks with no installation. Thanks!
Dominik

Userlevel 4
Badge +5

Hello @Dominik.Siedlak 

All the above commands used before upgrade or update, if you want only to check the compatibility then refer to compatibility matrix as below, also you need to check the hardware itself and Hypervisor version supported or not