4 node R710 cluster


Badge +15
I'm currently on a 4 node CE cluster of R710's. Don't have the Xeon specs handy at the moment.

I have the following memory in them:
96GB
128GB
144GB
144GB
Kind of strange memory setup, but it's how the chips went in.

I bought a 256GB consumer SSD for each and either RAID0'd a pair of 300GB HDD's, or bought a 1TB laptop drive. It's ugly, but it works.

I have 14 VM's at the moment. It's a mix of house things and experimental things. Plex, Unifi controller, Grafana, Netatmo data collection, Node playground, Chromecast server, Unifi Video, Wink API server(non-functional).

Outside of the Nutanix, I have a Nortel 48 port GE switch, Unifi UAP-AC-LR AP's(2), Unifi cameras(3). I also run a 18TB Unraid server for the bulk storage. The top R710 is a Windows Server I use for running Handbrake. 🙂

Bonus Grafana dashboard pic at the bottom:

14 replies

Userlevel 7
Badge +35
WOW - WOW - WOW, really imrpessed, I'm guessing you have that in your basement - How did you get that down there? WOW - I need to up my game. Great job sbrooke
Badge +15
I've been in the business for about 25 years. Over that time I've made a point of offering to take equipment to the dumpster/recycler whenever it's been ethical to do so. That cabinet was a company that was closing down and was literally going to throw it out. Got a friend with a pickup and we loaded it up. It was a bit challenging to get it in the basement, but when you take the sides and doors off it's not too heavy. Now that it's down there, it's probably never moving. My current problem (pun intended) is the power. The older, smaller UPSs aren't up to the task. If I get a power blip at all, they bounce. UPSs cost money, though. It's fine for the house, even though it's not great for the WAF when Plex is offline. I've been able to test how CE recovers from abrupt power outages! No problems, so far...
Userlevel 1
Badge +13
Nice setup and thanks for sharing?

A couple of questions on your grafana setup.

what backend DB ar you using for grafana? Influx? prometheus? how are you getting the nutanix metrics in the DB?

I've just started evaluating prometheus here at work, and pretty impressed with it so far and this was one of my next steps was to start trackign nutanix metrics.

Regards,

Stephen
Badge +15
Thanks! I'm using InfluxDB and Telegraf for the collection. Basically, I'm using Telegraf to do an SNMP get.

https://github.com/sbrooke/GrafanaSetup

The telegraf.conf has the OIDs I'm using for Nutanix.
Userlevel 1
Badge +13
thanks Scott
I'm having issues getting CE to see the drives connected in my R610.. did you do anything special to get it to see the drives behind the RAID controller?
Badge +15
zippyfear wrote:

I'm having issues getting CE to see the drives connected in my R610.. did you do anything special to get it to see the drives behind the RAID controller?


Yes! Sorry, I didn't get an alert on this thread being updated. I'm running the drives in RAID0 configuration behind an H700 controller. Because of that, CE can't detect the type of drive. You need to manually set the "rotational" drive and then it works fine.

Get to the initial install login, but instead of logging in as install, login as root.

Then, "dmesg | grep sda" to find the HDD. You may have to go through sdb, sdc... to find the HDD. Once you know which drive is the HDD and not the SSD, run the following:

"echo 0 > /sys/bloc/sd?/queue/rotational" with the correct sd letter in place of the ?

logout, then login as install. It should discover the disks correctly from there.
Badge +15
It looks like my original rack pic was removed, so here's a better looking one. More messy than I'd like, but that's the life of a homelab:

Badge
sbrooke wrote:


zippyfear wrote:

I'm having issues getting CE to see the drives connected in my R610.. did you do anything special to get it to see the drives behind the RAID controller?

Yes! Sorry, I didn't get an alert on this thread being updated. I'm running the drives in RAID0 configuration behind an H700 controller. Because of that, CE can't detect the type of drive. You need to manually set the "rotational" drive and then it works fine.

Get to the initial install login, but instead of logging in as install, login as root.

Then, "dmesg | grep sda" to find the HDD. You may have to go through sdb, sdc... to find the HDD. Once you know which drive is the HDD and not the SSD, run the following:

"echo 0 > /sys/bloc/sd?/queue/rotational" with the correct sd letter in place of the ?

logout, then login as install. It should discover the disks correctly from there.




Just wanted to say thanks for this. I've been banging my head trying to get CE installed on my Dell R710 Perc 6 SSD / SAS setup. This allowed the installer to find the drives correctly and fully install.
Badge
Slight typo figured I would mention it, just in case someone comes across this and wonders why the command fails. "bloc = block"

ie: " echo 0 > /sys/block/sd?/queue/rotational "
Badge +15
Nox, I just rediscovered that trying to fix a related issue. Check out my post in the main CE forum. Thanks!
Badge
sbrooke wrote:


zippyfear wrote:

I'm having issues getting CE to see the drives connected in my R610.. did you do anything special to get it to see the drives behind the RAID controller?

Yes! Sorry, I didn't get an alert on this thread being updated. I'm running the drives in RAID0 configuration behind an H700 controller. Because of that, CE can't detect the type of drive. You need to manually set the "rotational" drive and then it works fine.

Get to the initial install login, but instead of logging in as install, login as root.

Then, "dmesg | grep sda" to find the HDD. You may have to go through sdb, sdc... to find the HDD. Once you know which drive is the HDD and not the SSD, run the following:

"echo 0 > /sys/block/sd?/queue/rotational" with the correct sd letter in place of the ?

logout, then login as install. It should discover the disks correctly from there.



Did the steps above with the version ce-2018.05.01-stable and I'm still getting the error, "disks matching the minimum requirements was not found. I have a 400GB SSD and a 1.12 TB HDD attached to a perc 6/i both raid 0
Badge +15
y1der, I just did the same thing last weekend and didn't have any problems. I'm not sure if the perc 6/i might treat the disks differently than the H700. Maybe Nox will chime in.
Badge
sbrooke wrote:

y1der, I just did the same thing last weekend and didn't have any problems. I'm not sure if the perc 6/i might treat the disks differently than the H700. Maybe Nox will chime in.



Did you reboot before running the install

Reply