Deploying Hadoop on Nutanix Enterprise Cloud and AHV

  • 30 May 2018
  • 0 replies
Deploying Hadoop on Nutanix Enterprise Cloud and AHV
Userlevel 7
Badge +35
This post was authored by Gabe Contreras, Consulting Architect, Nutanix

Should you run Big Data applications such as Hadoop on bare metal or virtualize them? This question, along with a number of myths and misconceptions, continues to persist. The Nutanix consulting team thinks that virtualizing Big Data using an efficient hypervisor such as Nutanix AHV offers similar benefits to what you’d expect for any application environment:
  • Faster provisioning
  • Simplified infrastructure management
  • Improved availability
  • Greater flexibility
  • Increased efficiency and decreased cost
In most real-world Big Data environments, these advantages outweigh any potential benefits you might get from bare metal. For example, for one Nutanix customer the ability to manage Big Data and general server virtualization together—including all Production, Development, and Test environments—was a major factor, eliminating the need for multiple different silos of infrastructure.

A Nutanix Consulting Services team had multiple sessions with the customer’s Big Data team to convince them that it would be beneficial to virtualize Hadoop. They quickly saw that having one management pane, Prism Central, to see all of their environments, share images, and compare performance between Test and Production environments made management much easier. They also learned that Nutanix is very efficient when it comes to storage space utilization even when running with a replication factor of 2 (RF2) on both HDFS and Nutanix Distributed Storage Fabric (DSF).

Virtualizing Hadoop on Enterprise Cloud

Here is a summary of the customer’s Hadoop environment on Nutanix Enterprise Cloud. Production and Test environments are separate but identical. Each environment consists of:
  • Three (3) 32-node Nutanix clusters (each cluster is in a single rack)
  • Each 32-node cluster provides 1.32PB RAW; 662TB useable (RF2)
  • Total storage capacity: ~1.96PB usable with RF2
  • 24 cores and 512GB RAM per compute node
  • RF2 on both Hadoop and DSF
  • Hadoop Cluster spans all three racks
  • Each rack is both a Nutanix and Hadoop failure domain

The overall design follows the Nutanix reference architecture for Hadoop on AHV. This customer environment runs the entire Hadoop stack services with two Name Nodes, 75 Data Nodes, Kafka, Ambari, Grafana, MySQL, Zookeeper etc. All services are distributed evenly across the three racks. With even distribution, the Hadoop environment can withstand a simultaneous failure of one entire rack along with a single node in each of the other two racks.

Benefits of Virtualizing Hadoop

The customer experienced a number of significant benefits from virtualizing Hadoop on Nutanix:
  • Fast recovery from node failures
  • Simplified management
  • Efficient space usage
Fast recovery. Failure testing shows that it took less than 60 seconds after the complete failure of a single node for the VMs from the failed node to be up and running on another node in the Nutanix cluster. Because this recovery is so efficient, we tweaked the rebuild setting on the Hadoop side from “immediate” to “10 minutes.” This allows Hadoop to see if a node comes back up after a failure and avoids kicking off a rebuild for nodes that are able to recover.

Simplified management. The infrastructure team saw greatly improved manageability with Prism Central. It gives them one place to see their entire environment spanning almost 800 nodes—of which Hadoop is just one piece.

Hadoop admins gain the visibility to see not only their entire Test and Production environments from one pane, but easily filter down to an individual failure domain using Prism Central tags. Tags were created for all Hadoop VMs in each failure domain. Tags were also created for all the Hadoop environment VMs in both Test and Production.

The Prism Central analysis page helps the Hadoop admins correlate any issues in the environment. They can see VM performance, not just down to the hypervisor level but down to the level of each individual disk. This allows them to easily compare performance statistics on nodes over a period of months and also compare performance between Test and Production. This makes it easy to see the performance impact of changes during Test—before those changes are promoted to Production.

Efficient space usage. Useable capacity can be an important metric for a Hadoop environment; it can become especially critical depending on how long data is kept. The Nutanix reference architecture I mentioned above provides a detailed comparison of the Nutanix approach versus a bare metal Hadoop environment with RF3.

This customer keeps data for a long period of time. To achieve maximum efficiency, on the Nutanix-side we enabled both compression and Nutanix EC-X erasure coding on the data container. The running Nutanix software version, AOS 5.5, uses both the LZ4 and LZ4HC algorithms. Normal data is compressed with LZ4 for optimal performance while LZ4HC is used on cold data for an improved compression ratio.

The default settings were used for EC-X with a 4/1 stripe size. With these settings, EC-X decreases storage overhead to 1.25 vs the standard 2 for RF2. The combination of compression and erasure coding yielded significant space savings as shown in the following figure:

With Nutanix space savings features this customer sees effective capacity equivalent to the full RAW storage capacity. When you factor in the benefits of superior high availability and simplified management for this entire environment, that’s a big advantage. While we were onsite, the Nutanix team saw space savings numbers ranging from 2.02 to 2.05. These numbers were still rising as software updates were done and new efficiencies in algorithms were taking effect. Given the size of this cluster, it would also be possible to decrease the overhead further by increasing the EC-X stripe size to 18/1. That would bring down the EC-X overhead to 1.06, increasing useable space even further.

To find out more about deploying Big Data on Nutanix Enterprise Cloud, check out our Big Data eBook. If you’d like to learn more about Nutanix Consulting Services visit To set up a consultation, contact us at

©️ 2018 Nutanix, Inc. All rights reserved. Nutanix, the Nutanix logo and the other Nutanix products and features mentioned herein are registered trademarks or trademarks of Nutanix, Inc. in the United States and other countries. All other brand names mentioned herein are for identification purposes only and may be the trademarks of their respective holder(s).

This topic has been closed for comments