Independent industry reports tell us that the Global Datasphere [1] – the summation of all globally stored digital data – will grow to 175 zettabytes (ZB) by 2025. To give an idea of that kind of scale, the recent increase in interconnectedness of all things means the average household already fills over 300 iPhones (32 GB) in a year! At that kind of scale and growth, the usual file and block storage paradigms become too unwieldy in terms of management and performance. Object storage is built for hyper-scale and can handle the ever-growing unstructured data mix of social media, video, audio, log files, sensor data, and emails.

• Objects are stored as constituent pieces of data, essentially a flat list of objects that make up the file. Any relevant metadata is stored with the object and a custom identifier (UUID) is provided, rather than a filename.
• Objects use the Amazon S3 storage APIs. This is the de-facto standard for object storage data transfer.  Access objects through “GET” and “PUT” actions in API calls without worrying about complex directory structures.

The added benefit of using the Nutanix Cloud Platform infrastructure is that you inherit all core data path efficiencies trusted by thousands of customers – for example, compression and erasure coding – and much more. It drastically reduces the time to buy, build, manage, and deploy what can often be complex infrastructure.

Global Namespace 

The Nutanix solution uses S3 APIs for Objects. It provides a single namespace and storage fabric across Nutanix clusters and public clouds. Additionally, applications that write object data to a Nutanix cluster can replicate/tier across clouds. This enables developers to write applications to a single namespace without having to rewrite them if applications move across cloud boundaries.

Built for Scale 

This solution easily scales and reduces upfront costs to minimize unused resources. When the Nutanix solution is projected to run out of compute or storage resources, simply add more compute/capacity non-disruptively to scale the cluster and redistribute resources. In this way, the Nutanix solution is able to provide the same kind of elasticity typically associated with the public cloud.

One-click Simplicity

To span multiple clouds, management must be seamless and easy. Nutanix Objects provides a simple yet powerful way to scale and handle large amounts of unstructured data. Application developers can quickly and easily consume the storage using single-click operations. The Nutanix Cloud Platform not only simplifies operations, but it also uses machine intelligence and automation to reduce many clicks to one click via Prism Central. It’s designed to learn from large volumes of system data to automate common day-to-day tasks and generate actionable insights for optimizing virtualization, as well as deliver single-pane-of-glass management of applications and workloads across both private and public clouds.

Exploring Additional Use Cases

As part of our use-case analysis we are developing a Nutanix Objects document that explains how to configure your Hadoop environment to be able to run basic file operations, distcp, and MapReduce jobs on Nutanix Objects clusters.

See examples below:

Basic file operations

$ hdfs dfs -ls s3a://test-bucket/
$ hdfs dfs -copyFromLocal /home/ray/hosts /user/ray
$ hadoop fs -cp /user/ray/hosts s3a://test-bucket/
$ hadoop fs -cat s3a://test-bucket/hosts

Distcp operations

$ hadoop distcp hdfs://<NN IP>:8020/user/ray/hosts s3a://test-bucket/
Where <NN IP> : NameNode IP-Address or DNS name

Running MapReduce jobs on Objects

time /usr/bin/yarn jar /opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar teragen -Dmapreduce.job.maps=144 1000000000 s3a://mrtesting-bucket/teragen-test

If you want to read more about configuring your environment to use Nutanix Objects features, see the S3 configuration details for various use cases in our solutions document: Nutanix Objects: Additional Use Cases. If you’re working with S3 compatible Objects stores that you wish to run as part of a hybrid cloud environment, we would love to talk more and share experiences. Please get in touch with us via our social media channels or the Nutanix Community Forums.

h1] IDC Data Age 2025 : https://www.seagate.com/files/www-content/our-story/trends/files/idc-seagate-dataage-whitepaper.pdf

2022 Nutanix, Inc. All rights reserved. Nutanix, the Nutanix logo and all Nutanix product, feature and service names mentioned herein are registered trademarks or trademarks of Nutanix, Inc. in the United States and other countries. Other brand names mentioned herein are for identification purposes only and may be the trademarks of their respective holder(s). This post may contain links to external websites that are not part of Nutanix.com. Nutanix does not control these sites and disclaims all responsibility for the content or accuracy of any external site. Our decision to link to an external site should not be considered an endorsement of any content on such a site. Certain information contained in this post may relate to or be based on studies, publications, surveys and other data obtained from third-party sources and our own internal estimates and research. While we believe these third-party studies, publications, surveys and other data are reliable as of the date of this post, they have not independently verified, and we make no representation as to the adequacy, fairness, accuracy, or completeness of any information obtained from third-party sources.

This post may contain express and implied forward-looking statements, which are not historical facts and are instead based on our current expectations, estimates and beliefs. The accuracy of such statements involves risks and uncertainties and depends upon future events, including those that may be beyond our control, and actual results may differ materially and adversely from those anticipated or implied by such statements. Any forward-looking statements included herein speak only as of the date hereof and, except as required by law, we assume no obligation to update or otherwise revise any of such forward-looking statements to reflect subsequent events or circumstances.