Setup my first CE (3) Node HP DL360p G8 Cluster


Very happy to share that with the help of this forum I was able to setup my first CE (3) node cluster on some recently acquired HP DL360p G8 gear. Whether is it was enabling HBA on the P420i controller, or accounting for duplicate drive serial numbers, or troubleshooting the cluster creation process and how to cleanup and start over; ultimately, there wasn’t a single issue that I ran into that I couldn’t find the answer for right here within this community and I LOVE IT! *Also, HP is offering FREE Advanced iLO Licenses through the end of 2020 :o

  • (3) HP DL360p G8
  • (4) Intel(R) Xeon(R) CPU E5-2640 0 @ 2.50GHz
  • (2) Intel(R) Xeon(R) CPU E5-2660 0 @ 2.20GHz
  • (24) DDR3 1333Mhz 4GB (96GB) RAM
  • (1) 16GB Sandisk Cruzer USB
  • (1) 240GB SSD
  • (1) 500GB SSD
  • (1) 1TB SSD (Pending Addition)
  • (5) 300GB 10K SAS (Pending Addition)

 


23 replies

Userlevel 7
Badge +35

Hi @Jason Hendren 

Homelab looks awesome! :thumbsup:

keep us posted on any developments with the lab. Ping me if there is anything I can help with!

Thanks @aluciani , and great timing!

QUICK UPDATE:

  1. Deployed a couple of test Windows VM’s
  2. Backed them up with a separate Unitrends Virtual Appliance.
  3. Destroyed the cluster and redeployed it with the additional 1TB SSD’s
  4. Restored the test Windows VM’s successfully
  5. Used MOVE to migrate a few workloads from ESXi to AHV - consisting of W7, W10, W2K8.

I have to say, the MOVE migrations were the smoothest migrations I've ever done. Even the in-place MOVE upgrade from 3.5.0 to 3.5.1 was buttery smooth. It didn't matter whether I waited 14hrs to complete the W7 migration - Or if I stayed on top of the W2K8 migration - the cutover was slick and I only needed to deal with some minor NIC static IP assignments afterwards... a truly impressive utility!

*Any idea if FILES will be released with CE?

 

 

Hi @Jason Hendren,

we tried to create with CE 3 node cluster. Hardware is similar to yours:

each DL380p Gen8 with:

2 x 480 GB SSD

2 x 600 GB SAS

2 x E5-2630 v2

192 GB RAM 

Installer starts without errors, prechecks were fine but during installation we get strange messages (timeout, I/O Errors on disks,...)

We set the P420i Smartarray Controller in HBA mode too.

Do you have any hints for troubleshooting?

Thanks in advance and kind regards

Thomas

I/O Error
I/O Offline
Timeouts

 

@ThomasM, I'm happy to help! The first thing that comes to mind are the (2) 480 SSD. I would first check to see if they are presenting identically to the system (same s/n, etc). If this is the case the easiest troubleshooting method would be to remove (1) 480 and re-test the installation. There are ways to modify the s/n if necessary.

@Jason Hendren

Hi Jason,

I’ve removed one SSD and one SAS Drive, there left one SSD and one SAS drive in the server.

But no luck:

 

@ThomasM, I installed a 4th DL360 G8 this weekend, but I also got my hands on a DL380 G8 for testing. I'll follow-up tonight after I try an install. I'll be using a 500GB SSD and a 10k 600GB SAS harddisk.

~Jason

@ThomasM , I successfully installed CE on the DL380 G8 that I borrowed for testing and mentioned earlier. I experienced a few failed installs initially, and I also ran into the timeout issue you posted about, followed by the various I/O errors.

I attempted a couple of cleanups with re-installs but they also did not complete.

Tonight I started over with both a clean USB Key and a clean SSD and everything went smoothly. I believe if you do the same you’ll be successful too. 

Here are the primary issues that I addressed before starting with a clean install.

  1. Cleared NVRAM using DIP Switch 6 ON/OFF on mainboard.
  2. Ensured BIOS Date and Time were set correctly.
  3. Updated all firmware using the following iso from HP (P03093_001_spp-Gen8.1-SPPGen81.4) - this includes the firmware for the SAS 420i
  4. Cleared the SAS controller configuration and set to Pass-Through
  5. Used IMAGEUSB from http://www.passmark.com to write and post verify the written image on the USB key.
  6. Used a SATA to USB adapter to connect the SSD to my laptop and then used DISKPART to CLEAN the disk of any old configuration data before re-installing.

  

 

Let me know if this helps - I’m anxious to hear how it goes, and happy to provide additional assistance.

@Jason Hendren 

Hi, didn’t got it running.

We’ve observed strange output during install:

bootdevice vs. datadisk

In our testsystem is 1 x 480GB SATA SSD and 1 x 600GB SAS HDD, bootdevice is an internal SD card with ce image

What does it mean:

Chose boot disk /dev/sda

Formatting all data disks sda, sdb

My understanding is, that CE edition will only boot from removable media (SD card or USB). this device is /dev/sdc…

After failed installation fdisk -l shows only the SC card and no more disks, these will be visible only after a reboot:

 

kind regards, Thomas

 

Hi Thomas,

Sounds like we’re a bit closer! :o)

What type of internal SD Card (make/model/size) are you using? and what utility did you use to image it? I’d like to test this as well.

I’ve only ever successfully used Sandisk Cruzer 16GB USB 2.0 externally - with BIOS set to boot external drives first. So check that option with your SD Card to ensure boot internal devices first is selected instead.

Also, your mileage may vary, but if you do decide to try a USB thumb drive - I found that they are not created equally. I tried using new Kingston 32GB USB 3.0 devices and new PNY 16GB USB 2.0 devices - both with failures using Rufus to image them. (*see note). I ultimately settled on using the 16GB USB 2.0 Sandisk Cruzer. I also began using this freeware utility to to first check the health of the thumb drives in addition to including a verification step during creation. Check Flash 1.17.0

*note: I haven’t had the need to confirm yet, but I believe my earlier Kingston and PNY USB failures could have been attributed to various combinations of multiple issues in general - be it the creation method, antivirus, bios settings, hba pass-through, image verification, etc.. 

 

~Jason

Hi Jason,

thanks for quick response. I’ve tried these devices:

USB drive Transcend 32GB and SANDisk 16GB

SD Card HPE labeled 32GB and SANDisk Ultra 32GB

P420 is in HBA mode, latest firmware of all components (BIOS, Raid, iLO...) is installed.

Bootorder is set to USB drive (means also internal SD Card) as first boot device.

Image was put on the USB and SD Devices via dd if=<ce-image.img> of=/dev/<device>

This method is stated in the install guide: https://portal.nutanix.com/page/documents/details?targetId=Nutanix-Community-Edition-Getting-Started:Nutanix-Community-Edition-Getting-Started

Is the choice of the USB or SD device really critical??

But I’m confused about device numbering (see previous screenshot, bootdevice and datadisk are the same??)

Kind regards, Thomas

 

 

Hi Thomas,

I will test with an SD Card as soon as possible. I did note that the installation documentation does specify USB or SATADOM - without expressly stating if an SD Card is suitable.

As for myself, I chose to use External USB over internal devices due to the fact that at some point they may fail and require replacing and since my servers were stacked without rails - external would be easier to replace if ever necessary ;o)

As for the disk imaging process, are you using Linux or a Mac? My daily driver is Windows, but I have access to both Linux and Mac to test the creation as well, and would like to try the same method as yourself to see if anything pops up in that area.

*there is also a note regarding the Mac method: (OS X only) For improved performance when imaging, specify a block size of 1 MB. For example:dd if=ce.img of=/dev/rdisk2 bs=1m

~Jason 

 

@ThomasM, I'm happy to help! The first thing that comes to mind are the (2) 480 SSD. I would first check to see if they are presenting identically to the system (same s/n, etc). If this is the case the easiest troubleshooting method would be to remove (1) 480 and re-test the installation. There are ways to modify the s/n if necessary.

Hi, I’m also having the same issue. During install, drives are all presented with the same S/N and I’m getting the time-out issue… how do I resolve?

 

unfortunately I haven’t got an answer or helpful hint to my question above…

We’ll start from scratch with our tests soon, with standard hardware. 

 

 

@ThomasM and @mooseman

Hi, and I apologize for the delay in getting back to you ThomasM, but I was able to successfully install 5.17 onto a 32GB SD Card and everything worked as expected. I also used balenaEtcher to flash the image onto the SD Card since I didn’t have time to test otherwise with a Mac. I chose to create the cluster after the installation, and used the following command to add additional disks post cluster creation:

nutanix@cvm$: ce_add_disk

That said, please know that CE 5.18 has now been officially released, and the installation process has been changed. AFAIK, there is no longer a requirement for installing onto, or via, a USB drive and/or removable media. In addition, there’s now an ISO installer, and the initial disk requirements have changed. I hope all of these good changes help to smooth the installation process for you both. 

I have not performed this new install on my own HP DL380p G8 yet, but plan to do so before the end of the month, if not sooner.

https://next.nutanix.com/discussion-forum-14/download-community-edition-5-18-38417

@Jason

Great, We’ll try the new version asap.

Thanks for your advice

Kind regards, Thomas

 

@Jason:

it looks much better. Currently I Haven’got no hardware and I started a test in our virtual environment (ESX 6.5)

There crash the installer wirh “cannot launch installer vm” I think that’s caused by the virtualization, isn’t it?

Thanks.

 

@Jason:

it looks much better. Currently I Haven’got no hardware and I started a test in our virtual environment (ESX 6.5)

There crash the installer wirh “cannot launch installer vm” I think that’s caused by the virtualization, isn’t it?

Thanks.

 

I’m not sure of the new requirements, but I notice that the stderr is “cannot set up guest memory ‘pc.ram’ : Cannot allocate memory”. How much memory do you have overall?

Hi, I gave to the vm:

4 vCPU

16G vRAM

1 x vmdk 50GB

2 x vmdk 200GB

Kind regards, Thomas

p.s.

The whole cluster has even more memory…

 

@Jason Hendren 

Hi, after I gave the vm 32GB of memory the installer ran without issues and was finished…

Regards, Thomas

@Jason Hendren 

Hi, after creating a cluster I’ll try to first time login to web GUI.

In the documentation is stated:

  1. Open a web browser and navigate tohttp://management_ip_addr. Replace management_ip_addr with the IP address of any Nutanix Controller VM in the cluster.

    The browser will redirect to the encrypted port (9440) and may display an SSL certificate warning. Acknowledge the warning and proceed to the site.

  2. First time logon: Do this step if this is the first time you have logged on to your Community Edition cluster.
    1. Log on with the default admin account user name and password ofNutanix/4u).
    2. When prompted, change the default admin password, then log on with the admin account user name and new password.
    3. Type your Nutanix Next Community account user name and password and press Enter or click the right arrow icon.

    Click +Create Account if you have not joined the Next Community. Community Edition checks to ensure that the Community Edition role associated with your account is enabled.

  3. After logging on for the first time: Log on with the admin account user name and new password.”

What’s the default admin account login??

Thanks and regards, Thomas

 

Userlevel 1
Badge +1

The default login account is “admin” 😉

Hi, How do I see what’s causing the timeout during the install? where are the logs?

Hi, How do I see what’s causing the timeout during the install? where are the logs?

Is this 5.18 or 5.17? At what point does the timeout occur? What’s the end result being displayed?

Reply