Connect

Nutanix XCP VM Flash Mode – Enable SSD performance in a Hybrid System

  • 27 October 2015
  • 2 replies
  • 21204 views
Nutanix XCP VM Flash Mode – Enable SSD performance in a Hybrid System
Userlevel 7
Badge +35
The Nutanix Xtreme Computing Platform includes an information lifecycle management (ILM) service that automatically moves data between the hot and cold tiers. However, some users prefer all-flash performance for specific workloads within a hybrid system. In this situation, data for business-critical and latency-sensitive applications does not migrate from the SSD-backed, hot data tier to the cold data tier, which uses HDD (spinning) disks.

How it works

No more than 25 percent of the available DSF SSD capacity can be used for the VM Flash Mode feature. There will be no specific partition in the software set aside for this purpose and thus space not used by VM Flash Mode is used for other purposes. VM Flash Mode uses the SSD part of the Nutanix extent store capacity and is completely controlled by software.

VM Flash Mode works on a per-virtual machine virtual disk level. The virtual disks and VM Flash Mode can be configured on existing virtual machine virtual disks or on newly created virtual disks. If enabling VM Flash Mode on an existing virtual disk with data residing on the cold tier, the ILM mechanism migrates the data to the hot tier, rather than the VM Flash Mode feature.

VM Flash Mode can be configured for the entire virtual disk (or disks) or a portion of the virtual disk, specified using GB. DSF assigns data to the hot tier based on a first-in first-out (FIFO) basis when using the portion configuration option. If SSD space is available, data in excess of the minimum VM Flash Mode configuration requirements can reside in the hot tier. ILM does not migrate data for powered off virtual machines using VM Flash Mode to the cold tier.

The Figure Below shows VM Flash Mode virtual disk options for two virtual machines and four virtual disks (D1, D2, D3, and D4):

  • Virtual disk D1 is not using VM Flash Mode.
  • The VM Flash Mode feature places virtual disks D2 and D4 entirely in the hot tier.
  • The VM Flash Mode feature points a portion of virtual disk D3’s capacity to the hot tier.




For shared virtual disks, the VM Flash Mode configuration is visible via all the virtual machines sharing the respective virtual disks.

When using the virtual disk portion configuration option, it is critical to understand that data classified as hot initially might not be classified as hot in subsequent days—and thus ILM may migrate the data to the cold tier.

The following example illustrates what could happen when using the VM Flash Mode portion virtual disk configuration.




Nutanix stores two or three identical blocks of the same data, depending on Nutanix container configuration. For new writes, either two or three blocks are included in VM Flash Mode. However, when data is migrated from the cold to hot tier, only the local data copy is moved, which saves additional hot tier capacity.


Nutanix node failure scenario

During a node failure scenario, the available SSD capacity is reduced. This can lead to a situation where more than 25 percent of the SSD tier is pinned to SSD by VM Flash Mode, as in the following example:

  • 4 Nutanix nodes provide 600 GB VM Flash Mode SSD capacity.
  • 500 GB are used by VM Flash Mode.
  • 1 Nutanix node fails and only 450 XG is available for VM Flash Mode.

When the failure scenario appears, the soft limit increases to 50% (900 GB instead of 450 GB) of total SSD capacity and thus data will not be migrated from the hot tier to the cold tier. DSF migrates data from the hot to cold tier during a node failure when VM Flash Mode uses more than 50% (900 GB) of the SSD space.


Configuration

As mentioned previously, VM Flash Mode works on a virtual machine virtual disk level. There are two different options to configure VM Flash Mode and both include virtual disk GB specification:

  • Option 1: VM Flash Mode places the entire virtual disk in hot tier.
  • Option 2: VM Flash Mode places a portion of the virtual disk in hot tier.

The virtual machine virtual disk configuration is managed via nCLI using the following command:

ncli virtual-disk update-pinning id= tier_name=SSD-SATA pinned-space=Note: If the entire disk is configured for VM Flash Mode, there is no need to run the command again if the virtual disk is extended; the new size is included automatically.

Use the following command to find the virtual-disk-id required for the configuration:

The configuration can be verified via the two following options:

  • The command "vdisk_config_printer"
  • This output shows a virtual disk that is fully pinned to SSD by VM Flash Mode:

params { total_reserved_capacity: 42949672960 random_io_tier_preference: "SSD-SATA" sequential_io_tier_preference: "SSD-SATA"}
  • This output shows a virtual disk with a portion (4 GB) pinned to SSD by VM Flash Mode:
params { tier_params { tier_name: "SSD-SATA" min_pinned_usage_bytes: 4294967296 }}
  • Viewing the log file stargate.INFO available in the CVM directory /home/nutanix/data/logs. Use the following command, from one CVM, to view the stargate.INFO log file on all CVMs.
allssh cat data/logs/stargate.INFO | grep config_vdisk_op.cc

Interoperability

The VM Flash Mode feature is hypervisor agnostic. You can take advantage of the feature when running any of the supported hypervisors:

  • Acropolis
  • ESXi
  • Hyper-V

This section outlines the interoperability for VM Flash Mode and some of the many features available in the different supported hypervisors and in DSF:

  • Block awareness – No effect in homogenous Nutanix clusters. In heterogeneous Nutanix clusters, when VM Flash Mode places a virtual machine virtual disk entirely in hot tier it could fail a write I/O due to unavailable SSD space.
  • Clone – A newly cloned virtual disk does not preserve the VM Flash Mode configuration; the VM Flash Mode configuration applies only to the original virtual disk. Configure VM Flash Mode for the newly cloned virtual disk to take the advantage of the feature.
  • Compression – Does not affect VM Flash Mode. The DSF compression feature reduces the amount of SSD space used.
  • Deduplication – Does not affect VM Flash Mode. Deduplication reduces the amount of SSD space used.
  • Snapshot - A DSF VM snapshot preserves the virtual disk VM Flash Mode configuration. This configuration is not preserved when restoring a DSF VM snapshot, however, which means you must configure VM Flash Mode for the restored virtual disk to take advantage of the feature.
  • Replication - Remote copies will not have VM Flash Mode enabled. The original virtual disk keeps its VM Flash Mode configuration.
  • Disaster Recovery – Remote copies will not have VM Flash Mode enabled. The original virtual disk keep its VM Flash Mode configuration. A restore operation does not preserve the VM Flash Mode configuration.
  • Live Migration/vMotion - Does not affect VM Flash Mode configuration.
  • Storage Migration/Storage vMotion – The new virtual disk created on the target datastore will not have VM Flash Mode enabled. Configure VM Flash Mode for the newly created virtual disk to take the advantage of the feature
  • Metro availability - VM Flash Mode configuration will not be propagated to the VM running in the second Nutanix cluster.

Monitoring

The Nutanix Xtreme Platform has built-in monitoring associated with VM Flash Mode in both Nutanix PRISM and the Nutanix Cluster Check (NCC) tool.

The following alerts are built in to PRISM and NCC:

  • When 25 percent of the total SSD capacity is used.
  • When VM Flash Mode is activated on powered off virtual machines.
Additional monitoring is available via cli and two useful commands are listed below:

  • arithmos_cli master_get_entities entity_type=vmfilter_criteria=controller.total_pinned_vdisks=gt=0
  • The command output is plain text that can be searched for the information needed. plain text that can be sent to a file and searched for required information.
  • stats_tool --stats_type=vdisk_usage --stats_key=
  • The output will print ties usage for the virtual disk.

Recommendation

Before enabling VM Flash Mode, consider using Nutanix data reduction technologies such as compression, deduplication and EC-X, as they increase the usable SSD tier. This allows ILM to maintain more data in the hot tier, giving all flash performance to a larger working set.

Nutanix support may recommend using VM Flash Mode to increase performance, since hot tier capacity might be allocated for workloads that do not require hot tier performance.

Following are a set of useful guidelines for implementing VM Flash Mode:

  • Use for latency sensitive applications.
  • Use for workloads that run on a schedule that initiates data migration to the cold tier between jobs.
  • Activate the entire virtual disk if possible.
  • When VM Flash Mode is used to place a portion of the virtual disk in hot tier, data you want to remain hot might be migrated to cold tier. This happens when other non-critical data is written to the virtual disk and uses the VM Flash Mode space.
  • Activate for read workloads. Write-intensive workloads will hit SSD either via random writes that go to SSD because of oplog or sequential writes that go to hot tier in extent store.
  • Make sure the correct alert monitoring is activated in PRISM according to section https://#_Monitoring_1 and that actions are defined for the teams receiving the alerts.
  • It is always very important to monitor the hot tier usage in a Nutanix Xtreme Computing Platform environment to make sure the active working set fits in the hot tier. When using VM Flash Mode, it is even more important.

VM Flash Mode should be used as a last resort, as it reduces the ability of the DSF to manage workloads in a dynamic manner.

This post was authored by Magnus Andersson, solutions and consulting architect performance senior consultant.

We strive to ensure that the information contained in this blog post is accurate at time of publication and up to date - the info is subject to change.

2 replies

Badge +1
How to disable it back?
Pinning cant be used with starter license now
Badge +6
Hi, mfauzan have you upgraded to AOS 5.0 and if so did you follow the below outlined procedure in the Flash Mode Best Practices Guide:
  • Before Nutanix introduced the Flash Mode feature in AOS 5.0, Nutanix offered a similar feature, called Pin to SSD, beginning in AOS 4.5. If you have Pin to SSD enabled in a Nutanix cluster that you are upgrading to AOS 5.0 or later versions, follow the following procedure:
  • Disable Curator scan.
  • Disable Pin to SSD (now called Flash Mode).
  • Upgrade the Nutanix cluster.
  • Enable Flash Mode.
  • Enable Curator scan
If you don't have Ultimate license you should not enable Flash Mode meaning skipping the last two steps.

Thanks
Magnus

Reply