HOW TO: Deploying Nutanix CE to a local disk (no USB)

  • 14 October 2016
  • 21 replies
  • 250 views

Userlevel 1
Badge +9
Here are the steps I'm using for deploying Nutanix CE nodes for testing. Using this method seems to be the most consistent. I have found bits and pieces on this forum, but not a solid end-to-end guide. You will only need the USB drive long enough to perform these steps, it's no longer required for booting.
  1. Remove all but 2 drives (1 SSD and the SATA disk you'll be using)
  2. Download the latest image from Nutanix Community Portal It should look something like "ce-****.**.**-stable.img.gz"
  3. Download Kali Linux light x64 ISO and use something like Rufus to make a bootable USB drive
  4. Copy the ce image to the root of the USB drive after applying the Kali image with Rufus
  5. Boot the server using the Kali USB stick you created
  6. Open a terminal window
  7. Navigate to the directory where the root volume is mounted (/lib/live/mount/medium if you use Kali) and run the following command(s): cd /lib/live/mount/medium gunzip -c ce-2016.06.30-stable.img.gz | dd of=/dev/*** bs=1M Note: Replace *** with the drive you wish to overwrite. Use `cat /proc/partitions` to determine the name of the SATA drive. Use fdisk /dev/*** to clear the drive of existing partitions. This process should only take a few minutes and it'll return you to prompt. Reboot and remove the USB. On the first boot, choose the Nutanix recovery console. Login with root (default credentials). The initramfs needs to be rebuilt using the following commands: KERNEL=`ls /lib/modules`dracut -f /boot/initramfs-${KERNEL}.img ${KERNEL}
  8. Reboot again and this time let Nutanix CE boot into the default environment. You'll now be able to proceed with installation.
  9. After installation is complete on the node(s), reinstall all other drives and use Foundation to create/add the node to a cluster.

This topic has been closed for comments

21 replies

Badge +6
This worked well for me. Couple differences in my procedure:

  1. I didn't need to remove additional drives as in Step 1. for this procedure to work
  2. I used Ubuntu 16.0.4
  3. Step 7. required use of sudo gunzip...... | sudo dd.....
Thanks for writing this up!
Userlevel 6
Badge +16
I made similar write up if anyone is interested:

Install CE

- Install Nutanix CE on server from USB.
- If node is already in cluster then stop it.
- If CE is already installed and cluster is stopped then shut down CVM with: cvm_shutdown –P now, turn off host and insert new SSD, which you will be using for boot image. Newly added SSD must be empty. Otherwise format it.

Copy Image from USB to SSD

- Copy contents from USB to SSD with: dd if=/dev/sdX of=/dev/SDY
- If - Input folder (Your USB)
- Of – Output folder(Your SSD)
Copying will take long time, even though its only 7GB.

Remove USB

- Once copying is finished, turn off host and remove USB from it.

Dracut

- Turn host back on and login as root nutanix/4u
- Make sure you use back aphostrophes! Like this: `
- Run: KERNEL=`ls /lib/modules`
- Run: dracut -f /boot/initramfs-${KERNEL}.img ${KERNEL}
- Reboot and your done!
Userlevel 2
Badge +9

Don’t worry about dracut (more on dracut can be found here|: https://dracut.wiki.kernel.org/index.php/Main_Page). Just follow the guide to move the host installation from USB to some disk like described above.

In general Nutanix needs at least 3 disks to function:

 - Host: 1x disk (when installing Nutanix CE, the USB you install from will become the host installation disk - after the host installation is complete you can migrate from USB to some other disk)​​​​​​​

 - CVM: 2x disk (at least 1x SSD/NVMe)

As you wrote above the USB sticks are unreliable so the best option is to move the host installation from USB to some other disk (not the CVM disks) after you setup the Nutanix cluster host.

Userlevel 7
Badge +25

And dracut is the tool you use to prep the initial ramdisk

https://www.kernel.org/doc/html/latest/admin-guide/initrd.html

The ramdisk included w CE used to only have a USB device kernel module so you needed to regenerate initrd to allow the kernel to connect to the root block device.

Userlevel 2
Badge +9

Don’t worry about dracut (more on dracut can be found here|: https://dracut.wiki.kernel.org/index.php/Main_Page). Just follow the guide to move the host installation from USB to some disk like described above.

In general Nutanix needs at least 3 disks to function:

 - Host: 1x disk (when installing Nutanix CE, the USB you install from will become the host installation disk - after the host installation is complete you can migrate from USB to some other disk)

 - CVM: 2x disk (at least 1x SSD/NVMe)

As you wrote above the USB sticks are unreliable so the best option is to move the host installation from USB to some other disk (not the CVM disks) after you setup the Nutanix cluster host.


Hi @izy 

do I need 3 physical disks or can I use also partitions instead? Right now I only have 1 NVMe and 1 SSD per server.

BTW: Is the installation of Nutanix the same for CE and the “real stuff” you put into production?

Kind regards

Christian

The easiest way (at least for me) to approach this was to install Nutanix CE with an USB and use the disks in the host for the CVM. Have in mind that all the disks that you will have in the host during the Nutanix CE installation will be used for the CVM.

After the installation is done you will add an additional disk into the host and use this one to migrate the USB content to this new disk.

Regarding performance, the difference between the commercial version and Nutanix CE is how the disks are being accessed. In the case of Nutanix commercial version the disks will be directly accessed (passtrough) so CVM will have direct access to all of the disks. In the case of Nutanix CE the disk IO will go trough and additional layer (virtio) which will add some performance penalty. However, in case NVMe is used, both the commercial and CE version will use passtrough. The thing with NVMe is that there are issues with some of the NVMe disk models, but all NVMe disks that are supported in the commercial version will work also with Nutanix CE. 

Userlevel 2
Badge +9

Nutanix is not just a simple OS or just a hypervisor. It is a hyperconverged platform that consists of an hypervisor + a controller VM which handles all the stuff. Please read the Nutanix bible for more info: https://nutanixbible.com/.

Maybe it is not clear enough, but the USB key acts as an installation media and also as a boot drive for the host once the installation is done.

What you will need to do is the following:

  1. Install Nutanix CE with an imaged USB key (download the image from the download section) on the host. The host will need to have at least 1x200GB or more of fast storage (e.g. SSD or NVMe) and at least 1x500GB of slow storage (e.g. HDD). So leave the 512GB NVMe  and the HDD inside. Nutanix will consume all of your disk storage. Don’t create any partitions on your own on the disks. In fact the best is that the disks are clean before performing the installation.
  2. After the installation is done and Nutanix is basically running you will follow the procedure described here to move the USB key content to some additional disk  (it can be a small one, e.g. 100GB) that you will place in the host after the installation is done and everything works : https://next.nutanix.com/discussion-forum-14/how-to-deploying-nutanix-ce-to-a-local-disk-no-usb-13836?postid=30257#post30257
  3. After you are done the USB key is not needed any more since the host will boot and run from this additional disk.

Note that the second step is optional. Nutanix CE is meant to be setup as a playground and typically running the host from the USB key is enough to do that. You will want to migrate from USB key to an additional disk only if you want to run the cluster for an extensive time period.

After you follow the procedure above (steps 1 and 2) you will end up with a host that will have 3 disks inside and will run/boot from the disk that was added additionally as a replacement for the USB key. The other 2 disks that were initially placed in the host are used for Nutanix storage. 

Userlevel 2
Badge +9

Hi @cfrancke,

Nutanix storage requires at least 2 disks for the it to function (1x200GB hot storage + 1x500GB cold storage). Nutanix also needs a separate disk where the host is going to be installed (in case of CE this is the USB key by default, in the commercial edition this is a SATADOM disk).

See here for CE requirements:

https://portal.nutanix.com/page/documents/details/?targetId=Nutanix-Community-Edition-Getting-Started:Nutanix-Community-Edition-Getting-Started

During CE installation from USB key you won’t be able to proceed if at least 2 disks with correct specs (size + iops) that will be used for Nutanix storage are not discovered. The disks are imaged during installation so you cannot modify them afterwards (at least I have never seen this being done before). Also the USB key gets imaged (the host is installed on it) and it becomes the host boot disk.

To achieve what you want to do (getting rid of the USB key) you will need at least 3 disks (2 for Nutanix storage and 1 to replace the host USB disk).

Userlevel 2
Badge +9

No, unfortunately UEFI is still not supported yet.

Userlevel 3
Badge +11

Hi all,

I must admit I have trouble understanding the installation mechanics of Nutanix CE. Can’t I just mount the IMG file via my KVM (technically I can) and boot from it like I would from a XenServer or Windows Server ISO and install it onto my NVMe drive? (I also got a 2TB SSD installed, so mixed is given).

I don’t understand why I have to go so complicated through SATADOM or USB…

Kind regards

Christian

Userlevel 2
Badge +9

Nutanix is typically installed on bare metal HW. Nutanix CE follows the same principle, but it allows also to be installed nested inside an already existing hypervisor (e.g. KVM, ESX or Hyper-V). 

When installed on bare metal Nutanix CE will first image a hypervisor node and then create the required CVM on top of it. For that to happen Nutanix CE IMG includes the AHV hypervisor (which is based on KVM). The AHV hypervisor is installed and a CVM virtual machine is created that is running inside the AHV hypervisor. During the installation the bootable USB is used as a destination for the AHV hypervisor host installation. This way you don’t actually need an additional disk for AHV itself and all disks can be used for the CVM. Since USB disks often fail (particularly if you would want to run Nutanix for a longer time period) it is advised to move the host installation to some local disk like described in the first post.

If you are installing nested (e.g. on an already existing hypervisor like KVM) you could technically mount the IMG to the VM which will actually become the nested Nutanix AHV host inside which another VM (the CVM) will run.

Hope this helps.

Userlevel 3
Badge +11

@izy 

Thanks a lot for your fast feedback.

I installed Nutanix before on my 3x SM E200-8d machines but they were running of USB sticks, which constantly broke down.

For some time now I try to figure out how to “normally” install Nutanix (bare metal) onto my server (1 got 1x NVMe and 1x SSD per server installed).  I must admit I may have not yet invested enough time into the issue.

The downloadable IMG file does not seem to boot via IPMI KVM...My knowledge in this area is actually is a bit rusty :D I tested Windows Server and XenServer ISOs though which both booted fine.

Any proposals from your side how I can best install Nutanix on the given HW?

Kind regards

 

Christian

Userlevel 3
Badge +11

I made similar write up if anyone is interested:
 

Install CE

- Install Nutanix CE on server from USB.
- If node is already in cluster then stop it.
- If CE is already installed and cluster is stopped then shut down CVM with: cvm_shutdown –P now, turn off host and insert new SSD, which you will be using for boot image. Newly added SSD must be empty. Otherwise format it.

Copy Image from USB to SSD

- Copy contents from USB to SSD with: dd if=/dev/sdX of=/dev/SDY
- If - Input folder (Your USB)
- Of – Output folder(Your SSD)
Copying will take long time, even though its only 7GB.
 

Remove USB

- Once copying is finished, turn off host and remove USB from it.
 

Dracut

- Turn host back on and login as root nutanix/4u
- Make sure you use back aphostrophes! Like this: `
- Run: KERNEL=`ls /lib/modules`
- Run: dracut -f /boot/initramfs-$.img $
- Reboot and your done!


Hi @Primzy 

what does “Dracut” mean?

kind regards

Christian

Userlevel 3
Badge +11

Don’t worry about dracut (more on dracut can be found here|: https://dracut.wiki.kernel.org/index.php/Main_Page). Just follow the guide to move the host installation from USB to some disk like described above.

In general Nutanix needs at least 3 disks to function:

 - Host: 1x disk (when installing Nutanix CE, the USB you install from will become the host installation disk - after the host installation is complete you can migrate from USB to some other disk)​​​​​​​

 - CVM: 2x disk (at least 1x SSD/NVMe)

As you wrote above the USB sticks are unreliable so the best option is to move the host installation from USB to some other disk (not the CVM disks) after you setup the Nutanix cluster host.


Hi @izy 

do I need 3 physical disks or can I use also partitions instead? Right now I only have 1 NVMe and 1 SSD per server.

BTW: Is the installation of Nutanix the same for CE and the “real stuff” you put into production?

Kind regards

Christian

Userlevel 3
Badge +11

And dracut is the tool you use to prep the initial ramdisk

https://www.kernel.org/doc/html/latest/admin-guide/initrd.html

The ramdisk included w CE used to only have a USB device kernel module so you needed to regenerate initrd to allow the kernel to connect to the root block device.


Hi @jrack 

 

thanks a lot for pointing that out. I don’t fully understand that now but I will get back to that topic I guess.

Kind regards

Christian

Userlevel 3
Badge +11

Don’t worry about dracut (more on dracut can be found here|: https://dracut.wiki.kernel.org/index.php/Main_Page). Just follow the guide to move the host installation from USB to some disk like described above.

In general Nutanix needs at least 3 disks to function:

 - Host: 1x disk (when installing Nutanix CE, the USB you install from will become the host installation disk - after the host installation is complete you can migrate from USB to some other disk)

 - CVM: 2x disk (at least 1x SSD/NVMe)

As you wrote above the USB sticks are unreliable so the best option is to move the host installation from USB to some other disk (not the CVM disks) after you setup the Nutanix cluster host.


Hi @izy 

do I need 3 physical disks or can I use also partitions instead? Right now I only have 1 NVMe and 1 SSD per server.

BTW: Is the installation of Nutanix the same for CE and the “real stuff” you put into production?

Kind regards

Christian

The easiest way (at least for me) to approach this was to install Nutanix CE with an USB and use the disks in the host for the CVM. Have in mind that all the disks that you will have in the host during the Nutanix CE installation will be used for the CVM.

After the installation is done you will add an additional disk into the host and use this one to migrate the USB content to this new disk.

Regarding performance, the difference between the commercial version and Nutanix CE is how the disks are being accessed. In the case of Nutanix commercial version the disks will be directly accessed (passtrough) so CVM will have direct access to all of the disks. In the case of Nutanix CE the disk IO will go trough and additional layer (virtio) which will add some performance penalty. However, in case NVMe is used, both the commercial and CE version will use passtrough. The thing with NVMe is that there are issues with some of the NVMe disk models, but all NVMe disks that are supported in the commercial version will work also with Nutanix CE. 


Hi ​​​@izy 

 

wow, ok that’s a lot to chew and to be honest (towards Nutanix) user friendly is something else. I can insert a XenServer ISO just like that and have a cluster running in no-time without the need for USB...really something they need to improve.

 

Ok, so the plan is:

 

1. remove my NVMe

  1. install Nutanix via USB
  2. put the NVMe back
  3. run DD command to transfer the USB content to the NVMe
  4. Done

As far I observed though, my NVMe is 512GB and when I run the DD command it will create a partition around 7GB in size. What’s happening to the rest of the storage? Can I add that to the Cluster somehow?

 

Kind regards 

Christian

 

 

 

Userlevel 3
Badge +11

Nutanix is not just a simple OS or just a hypervisor. It is a hyperconverged platform that consists of an hypervisor + a controller VM which handles all the stuff. Please read the Nutanix bible for more info: https://nutanixbible.com/.

Maybe it is not clear enough, but the USB key acts as an installation media and also as a boot drive for the host once the installation is done.

What you will need to do is the following:

  1. Install Nutanix CE with an imaged USB key (download the image from the download section) on the host. The host will need to have at least 1x200GB or more of fast storage (e.g. SSD or NVMe) and at least 1x500GB of slow storage (e.g. HDD). So leave the 512GB NVMe  and the HDD inside. Nutanix will consume all of your disk storage. Don’t create any partitions on your own on the disks. In fact the best is that the disks are clean before performing the installation.
  2. After the installation is done and Nutanix is basically running you will follow the procedure described here to move the USB key content to some additional disk  (it can be a small one, e.g. 100GB) that you will place in the host after the installation is done and everything works : https://next.nutanix.com/discussion-forum-14/how-to-deploying-nutanix-ce-to-a-local-disk-no-usb-13836?postid=30257#post30257
  3. After you are done the USB key is not needed any more since the host will boot and run from this additional disk.

Note that the second step is optional. Nutanix CE is meant to be setup as a playground and typically running the host from the USB key is enough to do that. You will want to migrate from USB key to an additional disk only if you want to run the cluster for an extensive time period.

After you follow the procedure above (steps 1 and 2) you will end up with a host that will have 3 disks inside and will run/boot from the disk that was added additionally as a replacement for the USB key. The other 2 disks that were initially placed in the host are used for Nutanix storage. 


Hi @izy

really appreciate your feedback! I really want to move away from USB because these sticks tend to fail after 3 month. I don’t want to rebuild my cluster every 3 month. I’d like to have consistent LAB environment that also works after a year.
So, my challenge now is that I have 2 drives in my server and I cannot fit any other drive in there - no space.

Shall I just remove the NVMe, install Nutanix on the 2TB SSD and then plug-in the NVMe to DD the USB content there?
You keep emphasizing that Nutanix needs 2 disks. For what exactly? Can’t it just run of 1 disk?

Kind regards


Christian

Userlevel 3
Badge +11

Hi @cfrancke,

Nutanix storage requires at least 2 disks for the it to function (1x200GB hot storage + 1x500GB cold storage). Nutanix also needs a separate disk where the host is going to be installed (in case of CE this is the USB key by default, in the commercial edition this is a SATADOM disk).

See here for CE requirements:

https://portal.nutanix.com/page/documents/details/?targetId=Nutanix-Community-Edition-Getting-Started:Nutanix-Community-Edition-Getting-Started

During CE installation from USB key you won’t be able to proceed if at least 2 disks with correct specs (size + iops) that will be used for Nutanix storage are not discovered. The disks are imaged during installation so you cannot modify them afterwards (at least I have never seen this being done before). Also the USB key gets imaged (the host is installed on it) and it becomes the host boot disk.

To achieve what you want to do (getting rid of the USB key) you will need at least 3 disks (2 for Nutanix storage and 1 to replace the host USB disk).

 

Hi @izy 

OK, clear now. So, I’ll need to find a SATADOM that fits into the small server :D Thanks a lot for clearing this up!

Kind regards

Christian

Userlevel 3
Badge +11

Hi @izy 
 

I figured out a way to remount my internal SSD so the Supermicro SATADOM finally will fit (still waiting on delivery). Finally 3 drives then :D

 

I got one more question on the deployment though:

I got my Ubuntu USB boot stick prepped to perform a temp boot and would DD the nutanix.IMG file then to the SATADOM. From my memories from past tries I don’t recall an EFI system partition was created with that step. 
In order to boot from that SATADOM do I need to create another partition (EFI system partition) on the SATADOM?

If yes, would I copy some EFI related content from the Nutanix partition to it?

Kind regards

Christian

 

Userlevel 3
Badge +11

Hi @izy ,

 

I did some research :)
Do you know if UEFI support for Nutanix installations is default now or do I still have to perform these specific steps outlined here: https://next.nutanix.com/discussion-forum-14/uefi-bootable-community-edition-37296

kind
regards

Christian

Userlevel 3
Badge +11

@izy Thanks again for coming back so quick! Much appreciated!

Note for anyone working on getting Nutanix CE onto an HPE Gen10 system. Instead of the dracut -f {blah}, merely run

  • dracut --force --no-hostonly
This will cause the initramfs to be rebuilt with ALL the drivers and appears to get things booting up.