Blog

Part I: How to setup a three-node NUC Nutanix CE cluster

  • 23 November 2016
  • 44 replies
  • 619281 views
Part I: How to setup a three-node NUC Nutanix CE cluster
Userlevel 4
Badge +12
This blog was authored by MarcNutanix Sr. Systems Engineer at Nutanix.

The following is my experience in successfully setting up a three-node Nutanix Community Edition (CE) cluster for my home lab. During this process I did a lot of research and reviewed the CE forums, [go].

Since much of the information can be found in many locations, I thought it was a good idea to put them together in a single post that consolidates the details on how to get your first Nutanix CE cluster up and running.

To build a single NUC Node you need the following
► (1) Intel NUC Kit NUC6i7KYK Mini PC [go]
► (1) Transcend 32GB JetFlash 710 USB 3.1/3.0 Flash Drive [go]
► (2) Sandisk X400 Solid State Drive - Internal (SD8SN8U-512G-1122) [go]
► (1) Crucial 32GB Kit (16GBx2) DDR4 2133 MT/s (PC4-17000) SODIMM 260-Pin Memory - CT2K16G4SFD8213 [go]

A single NUC using the ingredients above will cost you around $1,000.

NOTE: Nutanix CE can run on 1, 3 or 4 Node Configurations. I specifically selected a (3) Node configuration in my home lab since this is also the minimum config for our Nutanix Customers in the real world.

The hardware install is very easy, just flip the NUC over, loosen 4 screws at each corner, remove the bottom cover and insert memory and the SSD drives.


Now that you have your hardware setup, the next step is to register and download the Nutanix CE software.

Register to join the Community

► Start here http://www.nutanix.com/products/community-edition/
► Scroll down the page and click on Get Access


► Fill in the info and click on submit.

Now that you have access to the Community
► Download the Nutanix CE Image file [go]
► Next, you have to get the image onto a USB Drive which your Node will boot from. I used Rufus on Windows, https://rufus.akeo.ie/
► Verify you have the following files downloaded.

Create a bootable Nutanix CE image on your USB Flash Drive

► Insert your USB Flash Drive.
► Run Rufus and click on the Image Icon.



► Make sure to change below to “All files” instead of ISO file, find the Nutanix CE Image file and click Open.



► Just click on Start and the Nutanix CE Image will be “burned” onto the USB Flash Drive.



► IMPORTANT: Each NUC needs its own USB flash drive to boot up and run Nutanix CE.
► Now plug in the USB flash drives into each one of your NUCs, power on the NUCs and install Nutanix CE.

Planning my (3) Node NUC Nutanix CE Cluster



I wanted to take some time and explain how my home network setup is configured and how my Nutanix CE home lab will interface with it. The Nutanix CE configuration will be sitting on my bookshelf in my home office, ready for use!
So my cable modem and wireless access point (with 4 Ports) sits on the main floor. All my laptops, tablets, phones, devices connect to the main floor WAP.

Since I wanted to have my Nutanix CE home lab sit in the bookshelf of my home office which is upstairs, I did the following:
► Purchased a gigabit ethernet switch for my lab
► Purchased a WAP to extend my home wireless
► Simply plugged in each NUC’s ethernet port into the gigabit ethernet switch
► Since the WAP is extending my home wireless, anything I plug into the bookshelf switch is connected to the internet

NOTE: You need to have internet connectivity when you log into the CVM or Cluster IP address via Prism to manage and run your Nutanix CE Cluster because this is tied to your Community email/password on my.nutanix.com
My configuration
Host (Physical NUC) IP Address: 192.168.1.150-152
Host Subnet Mask: 255.255.255.0
Host Gateway: 192.168.1.1

CVM (Each Host runs a Nutanix Controller Virtual Machine) IP Address: 192.168.1.160-162
CVM Subnet Mask: 255.255.255.0
CVM Gateway: 192.168.1.1

Things to do after Nutanix CE Installation and before you create your Cluster

After the installation, the first thing I did was to change the CVM memory which by default takes up 16GB. The NUC has a total of 32GB of memory, I wanted to reduce the amount of memory the CVM default takes so I can run plenty of virtual machines in my Nutanix CE home lab.

I decided to reduce the CVM memory to 8GB. After a couple weeks of using my Nutanix CE Cluster with plenty of virtual machines, I have not experienced any problems or concerns at all running each CVM with 8GB of memory.
To reduce the CVM memory to 8GB you can either (1) log directly into the Nutanix CE node or (2) SSH into the Nutanix CE node. Either way make sure to log in as “root” and not “nutanix”.

NOTE: Remember anytime you log into the NUC or Node you use “root” and anytime you log into the CVM you use “nutanix”. The password by default is nutanix/4u for either “root” or “nutanix”.

To set the CVM memory to 8GB
► virsh list --all
► virsh shutdown
► virsh setmem 8G --config
► virsh setmaxmem 8G --config
► virsh start
► virsh list –all, to make sure CVM is back up and running
Confirm CVM memory is set to 8GB
► virsh dominfo


Creating the Cluster
Now that you have installed each NUC node and configured each CVM with 8GB of memory, now is the exciting step of setting up a Cluster with your NUC Nodes.
► SSH into one of the CVMs, I opened up terminal on my Mac and typed “ssh nutanix@192.168.1.160” and used the password “nutanix/4u”
► Type “cluster –s cvmip,cvmip,cvmip create (no spaces between the commas)
► Example, to create my cluster I used “cluster –s 192.168.1.160,192.168.1.161,192.168.1.162 create”


► After completion, hopefully everything is up and running
► You can type “cluster status” and each CVM will display its status


Logging into your super cool new Nutanix CE Cluster for the first time
► Open your web browser of choice and connect to one of your CVM IP Addresses
► Use “admin” for the username and the password
► You will then be asked to change the password
NOTE: You need to make sure your laptop, tablet, etc. that you are using to access Prism also has internet access since Prism will ask for your email/password that you used to log into the Community site (my.nutanix.com).

Congrats! You are now logged into your Nutanix CE Cluster
Confirm that CVMs are set to 8GB of memory
- Click on Home – VM



- Click on Table, check the Include Controller VMs box and verify your CVMs are running at 8GB of Memory


Now let the fun begin!
Here is a pic of my Nutanix CE Home Lab Bookshelf Cluster


Disclaimer: This blog may contain links to external websites that are not part of Nutanix.com. Nutanix does not control these sites, and disclaims all responsibility for the content or accuracy of any external site. Our decision to link to an external site should not be considered an endorsement of any content on such site

This topic has been closed for comments

44 replies

Userlevel 7
Badge +35
Hi @Pooja

I suggest you get more RAM for your CE box. Check out the homelab section on the community for some ideas. 👍
Badge +2
I have 16GB RAM total for Nutanix CE box
Out of that 12GB has taken by CVM
And now I am not able to create any new VM on Nutanix CE.

Pointers will be helpful
Hi suporter,
I really like you writing and now i am deploying a lab with three Cisco UCS M4. My question is:
Can you suggest me a hardware ( CPU, RAM, RAID, HDD, SDD ) to deploy a nutanix software. Thank you
Badge +1
It is very helpful!
How is the performance over gigabit ethernet? Do you think a test setup is feasible over gigabit ethernet and 15k SAS drives?
Badge +5
@Gus I managed to installed CE as a single node on my other Intel i7-3820 machine and made sure I could logon to Prism. The Host IP is 192.168.1.3 and CVM = 192.168.1.6 and my Router Gateway is 192.168.1.1. I then shutdown the node and transfered the SSD/HDD to my Supermicro Atom system. It powers up perfectly and I can log on to AHV as root and do SSH to the CVM.

But the Host NIC is is not accessible by the CVM because AHV does not have access to the Intel NICs (Interl I354). The original machine on which I had installed CE has a different model of Intel NIC.

I can ping the above IPs from the AHV and CVM consoles I am led to believe in the interal networking is working but the external facing connection to the physical NIC is broken and I need to re-initialize the Host networking. Can someone please help me out here?

Badge
Thank you Gus. I thought so. I don't have the NUCs yet. Therefore I hesitate to buy my NUCs. It's 4k$ for something that might not last a year, given the fact our product is evolving so fast ^^Would love to see a microblock for demos able to support 64GB RAM per node 🙂 I looked to create my own cluster with MicroATX board, but then it's not that much transportable anymore. An alternative I thought of was to create a cluster in the cloud. Then it's also pricy.Well, for now, I'll do with hpocs. And perhaps the cloud solution might get cheaper in time.
Badge +3
Akim - A three-node NUC cluster is a great project/lab and it works extremely well.
Unfortunately memory utilization will continue to be an issue as the NUCs support up to 32G. However, it still works great and you can run a few VMs on the cluster and demostrate all Nutanix features including 1 node failure.
I've always set my CVMs with 16G to avoid issues. Yes, AOS 5.1.x requires 20G CVMs but please install the latest CE (2017.07.20) and you try with 16G CVMs.
Badge +5
Gus Sorry for the confusion, you are right this is at the AHV stage, the installer does not find the drives so it does not proceed further. So the screenshots are after logging on as root.
Badge
jas5.1 has been known to require more memory. On a the commercial version, 16GB CVMs give the same issue after a while. We now use to deploy 32GB CVM all the time.

On the CE, I wonder if 12GB would make it for a while. have you tried ?

I'm thinking of doing my own 3 node NUC cluster. But 16GB overhead on a NUC is quite a lot.
Badge +3
Looking at your screenshot, it seems like you are connected to the AHV host (the NUC) and not the CVM as you stated. Login to the AHV Host as root instead of nutanix.
If login is sucesful, then you can ssh from the AHV host to the CVM either by using the CVM real IP or the CVM's internal Nutanix IP (the backdoor) 192.168.5.254
Badge +5
While editing my post I accidently deleted the details. So just added again.
Below is the screenshot of the issue I am having when installing the CE on my Supermicro C2750 system fitted with 32GB RAM, 1 x 1TB SATA and 1 x 250GB SSD drive. I have tried installing several times but no luck. This screen shot is from the Installer routine. The installer is not able to detect any drives. For some reason it doesn't even find my 8GB USB flash drive which the system is being booted up with.




Logged on as root after the installer routine reverts to the installation screen. The FDISK does indicate the disks are attached to the system and the AHV/Centos can detect them but for some reason the requirements detection script thinks the drives are no good.
Badge +2



That the message what I see...
Badge +2
Perfect Gus, thanks.

Now I reboot my cluster and I´m not able to conect. Prism doesn't work and SSH neither.

When I connect a monitor and keyboard direct on the physical node I can access the CVM, but just like root, the nutanix user doesn't work.
Any idea what may happen?
Badge +3
It depends. If you have a domain in your local network or lab, you can use the domain controller as both DNS and NTP. Or you can simply use external ones such as Google's public DNS (8.8.8.8) and Public NTP: pool.ntp.org
Badge +2
Perfect Gus, Thanks!
Now, what are the IP addresses or hostaname I should use to DNS and NTP Server?
Badge +3
This is likely because you are logged in to the host (NUC) and not the CVM. Be sure to ssh to the CVM first and run the command from there.
You should login to the CVM as nutanix.
Badge +2
WhenI´m trying to create a cluster I see this message....

[root@NTNX-1aec6bf5-A ~]# cluster -s 192.168.1.160,192.168.1.161,192.168.1.162 createoption -s unrecognized - ignoredUsage: cluster graphfile-C k - generate no more than k clusters (0)0 : no limit-c k - use clustering method k (0)0 : use modularity1 : use modularity quality-o - output file (stdout)-v - verbose mode-? - print usage

Any suggestions?!
Badge +2
Hi all. I'm trying to install the Nutanix CE but when I'm close to the end of my installation I saw the message below:

Wating for the Nutanix Controller VM to start up.....................
"A problem was encountred. Please review the contents of /home/install/firstboot.out for details, and refer to the documentation or the Nutanix NEXT community for nets steps."

Please, somebody can help me?
Thanks!
Badge
I've setup a cluster with 8GB CVM memory and AHV 5.1. It worked but after two weeks the CVMs have run low on Memory despite the cluster is running no load at all (no VMs beneath the CVMs).
:~# ncli cluster get-timezone## There is insufficient memory for the Java Runtime Environment to continue.# Native memory allocation (mmap) failed to map 258998272 bytes for committing reserved memory.# An error report file with more information is saved as:# /home/nutanix/prism/cli/hs_err_pid14868.log

One of the CVMs wasn't reachable via PRISM anymore. I've noticed that there's an option to configure the CVM memory in PRISM to increase memory. However, that doesn't work showing:

"Memory update cannot run, since all nodes are low memoryInvalid parameters passed" 🙂.

So while it seems a little bit odd the CVMs need more than 8GB on an idle cluster, sticking to the recommendations seems to be the better decision here.
Badge
Can the CE edition be setup on AMF FX processors as well? Is working for my i7 however for a FX8150 I receive an error "invalid arch-independentELF magic" - Cool error BTW.
Badge +4
Be wary of which drives you choose. I went with Intell SSD 600p Series nvme drives. They quickly went into read-only state. Its a safety feature built into the drive, apparently. I guess Nutanix was too much for them.

http://www.tomshardware.com/reviews/intel-600p-series-ssd-review,4738.html

Intel clarified the nature of the read-only feature, which is not based upon the endurance limit. All SSDs have spare area that is dedicated to replacing failed cells. The Intel 600p only switches into a read-only mode when the spare area is exhausted. Intel also noted users can copy the data from a read-only SSD by installing it as a secondary drive in another computer.
Userlevel 4
Badge +20
Prism will have to be able to reach the internet. No way around it. How that's provided, either by cabled in hot spot or by guest wifi is fine.
Badge +2
Great Write-up!
I have one question about logging into Prism in a disconnected state. I plan on traveling with my 3 node cluster and am wondering what I do when I don't have connectivity to the internet. Looks like I may need to figure out a way to tether my phone if I'm Oscar Mike.

Thanks,
-Keith
Badge
Does any one know if intel Optane's works well with Nutanix CE? I'd like to set up a test lab at home. i am thinking of a 3 node NUC build.

x3 Intel BOXNUC7i7BNH NUC
x6 G.Skill Ripjaws 16GB (1x 16GB) DDR4 2133MHz
x3 Intel Optane 32GB M.2 Memory Module
x3 WD WD10JPLX 1TB Black 2.5" 7200RPM 9.5mm SATA3 Hard Drive

Be interested if this has been done before and anyones thoughts.

Ta,

H