Last year Intel and Nutanix announced two innovations that unleashed the performance of NVMe storage on Nutanix AOS. To get some background on these, visit our blog here.
Today we’ll be discussing further enhancements under development focused on our amazing Intel® Optane™ technology. It’s been a year since Intel® Optane™ SSDs become officially available in platforms such as the NX8170-G7, but we’re still hard at work to bring you every bit of performance we can. By default, Nutanix AOS runs in a flat storage tier when using most drive configurations. This storage layout was not taking advantage of some of the best qualities Intel Optane SSDs had to offer and I’ll explain why.
Nutanix currently categorizes drives in one of three types: DAS-SATA, SSD-SATA, and SSD-PCIe. The SSD-PCIe category includes drives like our Intel P4510 and Intel Optane SSD. Nutanix uses a flat tier architecture with SSDs – and assuming all drives have sufficient space available, the IO is distributed in a round robin fashion.
If Intel Optane SSDs are only receiving 1/4th of the writes and being read from ~1/4th of the time, then the latency benefits are masked.
So how can we get more of the IO to come from our reliably fast and high endurance Optane SSDs? 1 To answer this, we adopted Nutanix’s tiered storage architecture currently used with SSD plus HDD hybrid configurations. Our initial tests were promising, but more in-depth testing revealed some issues with data locality.
When using an SSD plus HDD hybrid configuration, it was often more beneficial to read/write most data from an SSD, even over the network. However, early benchmarks showed this isn’t always the case for tiered Intel Optane SSD plus SSD configurations. Reads from remote Intel Optane SSDs would have a higher latency than data read from other local SSDs. The result was performance regressions our joint innovation lab discovered. Once spotted, our engineer hats went on and the tiering code was updated to include strict data locality practices where UVM (user virtual machine) data would only be migrated between tiers on the same host node. For a graphical representation of the behavior differences, see the images below:
Hybrid SSDs plus HDD:
Tiered Intel Optane SSDs plus 3D NAND SSD:
The logic for up-migration (or horizontal) follows the same as that defined for egroup locality.
- Multiple reads every 10 second sampling count as a single touch
- Number of touches are monitored over a 10-minute window. Data is up-migrated if:
- 3 touches for random
- 10 touches for sequential
With this new version of tiering, we’ve gained substantial performance increases for read IO 2 (see below), which vary dependent on block size and working set size (how much frequently accessed data you have).
The takeaway? Exciting things continue to come from our joint innovation lab with Nutanix and AOS with Intel® Optane™ SSDs just keeps getting better. Keep an eye out for more upcoming news if you want to know what else we have in store for you!
For more details on the performance improvement Intel Optane SSDs can provide to a tiered storage architecture, please check out our .Next virtual exhibit booth.
This post was authored by Michael Velasco, Intel
Notices & Disclaimers
Performance varies by use, configuration and other factors. Learn more at www.Intel.com/PerformanceIndex .
Performance results are based on testing as of dates shown in configurations and may not reflect all publicly available updates. See backup for configuration details. No product or component can be absolutely secure.
Your costs and results may vary.
Intel technologies may require enabled hardware, software or service activation.
© Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.
[1] Intel® Optane™ SSD DC P4800X Series delivers 10µs/12µs read/write latencies and is rated for 60 DWPD (drive writes per day) compared to TLC NAND SSDs at ~3 DWPD.
Product Brief: Intel® Optane™ SSD DC P4800X/P4801X Series
Product Brief: Intel® SSD DC P4610 Series
[2] Intel tested, June 8th, 2021. Workload: 16K 100% Random Read & 70/30 Random R/W, using XRay 3.70.
4 VMs, 1 VDisk, QD 4 to 128. See configurations below:
Configuration 1: 1-node, 2x Intel Xeon 6248R, 24c, 3.0GHz Base, 4.0GHz Turbo, 192GB (12 slots/16GB/2933MHz total 192GB DDR4 memory, 8x Intel® SSD DC P4510 (2TB), 1x Intel® SSD D3-S4510 M.2 (240GB) Intel® X710 (10GbE) & Mellanox MT27710 (25GbE), AOS version 5.20 development build, BIOS SE5C620.86B.02.01.0008.CNX01
Configuration 2: 1-node, 2x Intel Xeon 6248R, 24c, 3.0GHz Base, 4.0GHz Turbo, 192GB (12 slots/16GB/2933MHz total 192GB DDR4 memory, 2x Intel® Optane™ DC P4800X (750GB), 8x Intel® SSD DC P4510 (2TB), 1x Intel® SSD D3-S4510 M.2 (240GB) Intel® X710 (10GbE) & Mellanox MT27710 (25GbE), AOS version 5.20 development build, BIOS SE5C620.86B.02.01.0008.CNX01