Blog

Part I: How to setup a three-node NUC Nutanix CE cluster

Part I: How to setup a three-node NUC Nutanix CE cluster
Userlevel 3
Badge +12
This blog was authored by MarcNutanix Sr. Systems Engineer at Nutanix.

The following is my experience in successfully setting up a three-node Nutanix Community Edition (CE) cluster for my home lab. During this process I did a lot of research and reviewed the CE forums, [go].

Since much of the information can be found in many locations, I thought it was a good idea to put them together in a single post that consolidates the details on how to get your first Nutanix CE cluster up and running.

To build a single NUC Node you need the following
► (1) Intel NUC Kit NUC6i7KYK Mini PC [go]
► (1) Transcend 32GB JetFlash 710 USB 3.1/3.0 Flash Drive [go]
► (2) Sandisk X400 Solid State Drive - Internal (SD8SN8U-512G-1122) [go]
► (1) Crucial 32GB Kit (16GBx2) DDR4 2133 MT/s (PC4-17000) SODIMM 260-Pin Memory - CT2K16G4SFD8213 [go]

A single NUC using the ingredients above will cost you around $1,000.

NOTE: Nutanix CE can run on 1, 3 or 4 Node Configurations. I specifically selected a (3) Node configuration in my home lab since this is also the minimum config for our Nutanix Customers in the real world.

The hardware install is very easy, just flip the NUC over, loosen 4 screws at each corner, remove the bottom cover and insert memory and the SSD drives.


Now that you have your hardware setup, the next step is to register and download the Nutanix CE software.

Register to join the Community

► Start here http://www.nutanix.com/products/community-edition/
► Scroll down the page and click on Get Access


► Fill in the info and click on submit.

Now that you have access to the Community
► Download the Nutanix CE Image file [go]
► Next, you have to get the image onto a USB Drive which your Node will boot from. I used Rufus on Windows, https://rufus.akeo.ie/
► Verify you have the following files downloaded.

Create a bootable Nutanix CE image on your USB Flash Drive

► Insert your USB Flash Drive.
► Run Rufus and click on the Image Icon.



► Make sure to change below to “All files” instead of ISO file, find the Nutanix CE Image file and click Open.



► Just click on Start and the Nutanix CE Image will be “burned” onto the USB Flash Drive.



► IMPORTANT: Each NUC needs its own USB flash drive to boot up and run Nutanix CE.
► Now plug in the USB flash drives into each one of your NUCs, power on the NUCs and install Nutanix CE.

Planning my (3) Node NUC Nutanix CE Cluster



I wanted to take some time and explain how my home network setup is configured and how my Nutanix CE home lab will interface with it. The Nutanix CE configuration will be sitting on my bookshelf in my home office, ready for use!
So my cable modem and wireless access point (with 4 Ports) sits on the main floor. All my laptops, tablets, phones, devices connect to the main floor WAP.

Since I wanted to have my Nutanix CE home lab sit in the bookshelf of my home office which is upstairs, I did the following:
► Purchased a gigabit ethernet switch for my lab
► Purchased a WAP to extend my home wireless
► Simply plugged in each NUC’s ethernet port into the gigabit ethernet switch
► Since the WAP is extending my home wireless, anything I plug into the bookshelf switch is connected to the internet

NOTE: You need to have internet connectivity when you log into the CVM or Cluster IP address via Prism to manage and run your Nutanix CE Cluster because this is tied to your Community email/password on my.nutanix.com
My configuration
Host (Physical NUC) IP Address: 192.168.1.150-152
Host Subnet Mask: 255.255.255.0
Host Gateway: 192.168.1.1

CVM (Each Host runs a Nutanix Controller Virtual Machine) IP Address: 192.168.1.160-162
CVM Subnet Mask: 255.255.255.0
CVM Gateway: 192.168.1.1

Things to do after Nutanix CE Installation and before you create your Cluster

After the installation, the first thing I did was to change the CVM memory which by default takes up 16GB. The NUC has a total of 32GB of memory, I wanted to reduce the amount of memory the CVM default takes so I can run plenty of virtual machines in my Nutanix CE home lab.

I decided to reduce the CVM memory to 8GB. After a couple weeks of using my Nutanix CE Cluster with plenty of virtual machines, I have not experienced any problems or concerns at all running each CVM with 8GB of memory.
To reduce the CVM memory to 8GB you can either (1) log directly into the Nutanix CE node or (2) SSH into the Nutanix CE node. Either way make sure to log in as “root” and not “nutanix”.

NOTE: Remember anytime you log into the NUC or Node you use “root” and anytime you log into the CVM you use “nutanix”. The password by default is nutanix/4u for either “root” or “nutanix”.

To set the CVM memory to 8GB
► virsh list --all
► virsh shutdown
► virsh setmem 8G --config
► virsh setmaxmem 8G --config
► virsh start
► virsh list –all, to make sure CVM is back up and running
Confirm CVM memory is set to 8GB
► virsh dominfo


Creating the Cluster
Now that you have installed each NUC node and configured each CVM with 8GB of memory, now is the exciting step of setting up a Cluster with your NUC Nodes.
► SSH into one of the CVMs, I opened up terminal on my Mac and typed “ssh nutanix@192.168.1.160” and used the password “nutanix/4u”
► Type “cluster –s cvmip,cvmip,cvmip create (no spaces between the commas)
► Example, to create my cluster I used “cluster –s 192.168.1.160,192.168.1.161,192.168.1.162 create”


► After completion, hopefully everything is up and running
► You can type “cluster status” and each CVM will display its status


Logging into your super cool new Nutanix CE Cluster for the first time
► Open your web browser of choice and connect to one of your CVM IP Addresses
► Use “admin” for the username and the password
► You will then be asked to change the password
NOTE: You need to make sure your laptop, tablet, etc. that you are using to access Prism also has internet access since Prism will ask for your email/password that you used to log into the Community site (my.nutanix.com).

Congrats! You are now logged into your Nutanix CE Cluster
Confirm that CVMs are set to 8GB of memory
- Click on Home – VM



- Click on Table, check the Include Controller VMs box and verify your CVMs are running at 8GB of Memory


Now let the fun begin!
Here is a pic of my Nutanix CE Home Lab Bookshelf Cluster


Disclaimer: This blog may contain links to external websites that are not part of Nutanix.com. Nutanix does not control these sites, and disclaims all responsibility for the content or accuracy of any external site. Our decision to link to an external site should not be considered an endorsement of any content on such site

This topic has been closed for comments

37 replies

Userlevel 2
Badge +5
Excellent writeup Valeria! Overall the process looks simple, but it's great to have a nice, clear write-up like this! Thanks!
Badge +2
Hi,
Great writeup ! just one question, after tconfiguring the host and CVm ip address, therer is teh set up single node cluster question (sowmthing like that anyway, do u chck that or leave it un checked?
Badge +6
Happy Friday JasonMills, since I was setting up (3) Nodes to place into a Cluster, I left the Single Node Cluster unchecked during the setup of each of these Nodes.
Badge +3
JasonMills: Even when I am setting up a one-node cluster I still leave that checkbox unchecked and I manually create the cluster from CLI:
cluster -s ,, create
Badge +3
valeriavong: Thank you for this great article!
Just one clarification: In your screenshots seems like you used the .gz (compressed image) file to burn to the USB thumb drive but if I'm not mistaken, it should be the uncompressed image file (.img). Is this correct?
Badge +2
Excellent write up. I have just one question about hardware configuration. Here in my country I didn't found Intel NUC for sell, and import from US is not an viable option. My question is, the Nutanix CE cluster might runing with the hardware configuration listed below?
Lenovo ThinkCentre M710 Tiny:- Intel Core i7-7700T Processor (8MB Cache, 2.90 GHz)- 32GB ( 16+16 ) DDR4 2400 SODIMM- 500GB Hard Drive, 7200RPM, 2.5", SATA3- 512GB SolidState Drive M.2 PCIe Opal- Integrated Gigabit Ethernet Port- Intel Dual Band Wireless-AC 3165, 802.11ac, 1x1 Wi-Fi + Bluetooth 4.0
I think I can found that hardware here in Brazil.
Thanks.
Badge +3
freed_m_lima : Yes. That configuration looks good. I would increase RAM to 64G if you can but even 32G will do Ok.
You don't need the WiFi adapter but I assume it is built-in.
Badge +2
Thanks Gus. 32 GB is the maximum memory supported. But what about the 500GB Hard Drive, 7200RPM, 2.5", SATA3? You think it will works fine?
Badge +3
Ok. 32G RAM will be OK for one node.
Yes, drives look fine. You want to have one SSD + one HDD minimum. Both your drives look good to me. Even your SSD can be a little smaller if you like (250G for example) and the HDD can be as big as you like.
Badge +2
Actually I´m planning to create a three node cluster and decrease the CVM memory from 16GB to 8 GB as I read at this post.
Badge
Hey one question .. is it possible to install Esxi Hypervisor or are we limited to Acropolis ? You didnt mentioned nothing about the HyperVisor.



Userlevel 4
Badge +20
Nutanix CE is limited to AHV for the base hypervisor. Some have tried nesting ESX with varied results.
Badge
Does any one know if intel Optane's works well with Nutanix CE? I'd like to set up a test lab at home. i am thinking of a 3 node NUC build.

x3 Intel BOXNUC7i7BNH NUC
x6 G.Skill Ripjaws 16GB (1x 16GB) DDR4 2133MHz
x3 Intel Optane 32GB M.2 Memory Module
x3 WD WD10JPLX 1TB Black 2.5" 7200RPM 9.5mm SATA3 Hard Drive

Be interested if this has been done before and anyones thoughts.

Ta,

H
Badge +2
Great Write-up!
I have one question about logging into Prism in a disconnected state. I plan on traveling with my 3 node cluster and am wondering what I do when I don't have connectivity to the internet. Looks like I may need to figure out a way to tether my phone if I'm Oscar Mike.

Thanks,
-Keith
Userlevel 4
Badge +20
Prism will have to be able to reach the internet. No way around it. How that's provided, either by cabled in hot spot or by guest wifi is fine.

Badge +4
Be wary of which drives you choose. I went with Intell SSD 600p Series nvme drives. They quickly went into read-only state. Its a safety feature built into the drive, apparently. I guess Nutanix was too much for them.

http://www.tomshardware.com/reviews/intel-600p-series-ssd-review,4738.html

Intel clarified the nature of the read-only feature, which is not based upon the endurance limit. All SSDs have spare area that is dedicated to replacing failed cells. The Intel 600p only switches into a read-only mode when the spare area is exhausted. Intel also noted users can copy the data from a read-only SSD by installing it as a secondary drive in another computer.
Badge
Can the CE edition be setup on AMF FX processors as well? Is working for my i7 however for a FX8150 I receive an error "invalid arch-independentELF magic" - Cool error BTW.
Badge
I've setup a cluster with 8GB CVM memory and AHV 5.1. It worked but after two weeks the CVMs have run low on Memory despite the cluster is running no load at all (no VMs beneath the CVMs).
:~# ncli cluster get-timezone## There is insufficient memory for the Java Runtime Environment to continue.# Native memory allocation (mmap) failed to map 258998272 bytes for committing reserved memory.# An error report file with more information is saved as:# /home/nutanix/prism/cli/hs_err_pid14868.log

One of the CVMs wasn't reachable via PRISM anymore. I've noticed that there's an option to configure the CVM memory in PRISM to increase memory. However, that doesn't work showing:

"Memory update cannot run, since all nodes are low memoryInvalid parameters passed" 🙂.

So while it seems a little bit odd the CVMs need more than 8GB on an idle cluster, sticking to the recommendations seems to be the better decision here.

Badge +2
Hi all. I'm trying to install the Nutanix CE but when I'm close to the end of my installation I saw the message below:

Wating for the Nutanix Controller VM to start up.....................
"A problem was encountred. Please review the contents of /home/install/firstboot.out for details, and refer to the documentation or the Nutanix NEXT community for nets steps."

Please, somebody can help me?
Thanks!
Badge +2
WhenI´m trying to create a cluster I see this message....

[root@NTNX-1aec6bf5-A ~]# cluster -s 192.168.1.160,192.168.1.161,192.168.1.162 createoption -s unrecognized - ignoredUsage: cluster graphfile-C k - generate no more than k clusters (0)0 : no limit-c k - use clustering method k (0)0 : use modularity1 : use modularity quality-o - output file (stdout)-v - verbose mode-? - print usage

Any suggestions?!
Badge +3
This is likely because you are logged in to the host (NUC) and not the CVM. Be sure to ssh to the CVM first and run the command from there.
You should login to the CVM as nutanix.
Badge +2
Perfect Gus, Thanks!
Now, what are the IP addresses or hostaname I should use to DNS and NTP Server?
Badge +3
It depends. If you have a domain in your local network or lab, you can use the domain controller as both DNS and NTP. Or you can simply use external ones such as Google's public DNS (8.8.8.8) and Public NTP: pool.ntp.org

Badge +2
Perfect Gus, thanks.

Now I reboot my cluster and I´m not able to conect. Prism doesn't work and SSH neither.

When I connect a monitor and keyboard direct on the physical node I can access the CVM, but just like root, the nutanix user doesn't work.
Any idea what may happen?
Badge +2



That the message what I see...