Five secrets traditional storage array vendors don’t want you to know about NVMe
This blog was authored by Paul Updike, Sr. Manager, Technical Marketing Engineering at Nutanix, Andy Daniel, Sr. Technical Marketing Engineer at Nutanix, & Marc Trouard-Riolle, Sr. Product Marketing Manager at Nutanix.
Non-Volatile Memory Express (NVMe) breaks the way traditional storage arrays use mass storage devices and requires significant product investment to enable it. Why? Because NVMe is not a traditional storage architecture. In fact, the first iterations of flash storage (SSD) weren’t either, but to bring products to the market faster and to foster wider adoption, interfaces and protocols to drives didn’t change. Initial SSDs were designed to enable flash in traditional storage architectures. To meet this requirement, SSDs emulate traditional spinning hard drives. The data in and out flows over hard drive interfaces that pretend there’s a spinning platter on the other side, and are therefore bound by their limitations. With this in mind, here is what traditional storage array vendors don’t want you to know about NVMe:
NVMe was delivered first as a server technology
As is the case with most new technology, the first drives were found inside generic desktops and server systems. Today, you can purchase NVMe technology inside servers from all the major vendors. You can even buy individual units from Best Buy. Gamers put them in desktops and if you’ve purchased a laptop in the last year, it’s likely that NVMe is in there too. Unfortunately, NVMe and other mass market technologies designed for use inside servers take a long time to adopt for storage array use. (Nutanix Hyperconverged Infrastructures are server based technology)
High availability requires a new architecture
Traditional array designs are built on SAS and SATA interfaces. SAS is natively dual ported to allow more than one controller at a time to connect and array vendors use specialized transposers to allow the same for SATA. NVMe architecture doesn’t use SAS or SATA components and requires a whole new design. The first NVMe 1.1 spec, U.2 dual port NVMe drives were announced in early 2016, but there is no major storage vendor shipping this dual port capability. Note that dual port U.2 must also halve available PCIe lanes (2 for each controller), potentially limiting throughput. (Nutanix Hyperconverged Infrastructures don't have to dual-attach drives for High Availability)
NVMe is harder to hot-swap
NVMe runs directly over PCIe and requires full platform support for PCIe hot-swap. There are NVMe drives sold as hot-plug ready, but that is only with a specific physical platform, CPU, operating system, and driver. The current drivers are written for a server platform and OS. Again, vendor specialization is required for support. (Nutanix can migrate work and enable node level repairs without interruption)
RDMA is required to take full advantage of remote NVMe devices
Outside of exotic PCIe switched designs, remote NVMe access over traditional fabrics (Ethernet and fibre-channel) requires remote direct memory access (RDMA) and new protocols like NVMe over Fabric (NVMeF). Simple use of the TCP/IP stack negates the significant latency reduction of NVMe without RDMA to bypass the stack. Like all industry standards, NVMeF is slow in development and not currently shipping from any major storage vendor. (Nutanix doesn’t need NVMeF to take advantage of RDMA)
NVMe on an array encourages even more aggregation of risk
Since they thrive on the growth and consolidation of data in data centers, this is actually something many storage array vendors would encourage. The goal of many vendors is the constant replacement of older generation hardware. To do this with faster, higher capacity storage you must pile your data into the same system while increasing your "eggs in one basket" risk as you do. (A Nutanix Enterprise Cloud grows in units of nodes and risk is spread rather than concentrated)
How to take advantage of NVME right now.
Nutanix is adding a new all flash node to the NX-9030 series platform that exploits a combination of SATA SSDs and NVMe. To review how we do this:
► Our cluster nodes are standardized server architectures. ► We can easily handle a fault and replacement at the node level, should an NVMe device fail. ► We have customized our software to enhance performance by thoughtful use of the NVMe. ► We innovate quickly using existing RDMA technology that isn't further reliant on a new industry specification. ► Our Enterprise Clouds spreads, rather than aggregates, risk.
Ultimately, Nutanix produces innovative software. Our fail-fast webscale design, based around hyperconverged infrastructure principles enables rapid adoption of new hardware technology from innovative hardware vendors. Riding the server wave of innovation allows us to help our customers deliver positive business outcomes first.