Mixing Hybrid with All Flash Nodes


Badge +7
With 5.1, is it possibel to mix 2 x Hybrid Nodes with 2 x All Flash Nodes or do we still need to have 3 x Hybrid Nodes and 2 x All Flash as a minimum?

13 replies

Userlevel 1
Badge +9
When adding any different node type it's preferred to add them in pairs, this is a must if adding nodes with larger capacity than existing ones.

In your example 2x hybrid and 2x AF nodes would be fine. Just use matching hybrid nodes and matching AF nodes.
Badge +7
briansuhr wrote:

When adding any different node type it's preferred to add them in pairs, this is a must if adding nodes with larger capacity than existing ones.

In your example 2x hybrid and 2x AF nodes would be fine. Just use matching hybrid nodes and matching AF nodes.


Thanks for the quick response. In this scenario is it still possible to enable Erasure Coding if having 2 x Hybrid and 2 x All Flash?
Userlevel 1
Badge +9
Hello,

Yes erasure coding will work normal with this config, min number of 4 nodes is met.
Badge +7
Would there be an issue if only 1 All flash node was added to a 3 node hybrid cluster? I can understand that performance may drop when doing live migration or reading from node having data on the cold tier but wouldn't that be the same regardless if having 2 all flash nodes, that the RF Data could sit anywhere in the cluster and not just on the other all flash node?

Userlevel 1
Badge +9
While this should be technically possible, it would not be a recommended configuration. When adding a different node type to a cluster it's recommended to add a pair the first time. By having a pair you are ensured there is rebuild capacity in the cluster for each node type.

Your example of 2 hybrid and 2 AF nodes to start a cluster. After that you can add either node types 1 at a time.

Badge +7
Is that only capacity constraint? Considering that an all flash node would most likely be smaller storage than a hybrid, I was wondeirng where the bottleneck would be.

i am thinking of adding an all flash node with this kind of configuration:
Current Cluster 3 nodes:
4 x 400GB SSD Per Node
6 x 2TB Sata Per Node

Additional Node:
10 x 400GB SSD
Userlevel 4
Badge +20
It's definitely a concern. In you're current cluster you should have around 1.84TiB of SSD of extent store (see designbrewz image) and the new node is bringing in 1.13TiB of SSD for workload that you are expecting SSD performance for everything that runs on that node. If the node fails, unless the other workload has used more than .71TiB, you are going to be overflowing the SSD. Buy starting in pairs and using affinity rules you'll be able to ensure that the VM's that expect SSD performance are going to get it if you experience a node failure.




Userlevel 1
Badge +9
The best practice of adding in pairs for different nodes types is both for capacity rebuilds and predictable performance. Having a node of matching capacity will ensure there is space to rebuild should a failure occur. This is of biggest concern when adding nodes with increased capacity to a cluster, not of concern in your case.

The second thought is on predictable performance. Should a failure occur much like the rebuild there is resources to provide same matching performance experience. If you added a single AF node and it was the one that failed you would be reverting back to your current performance experience. Which is likely not an issue for your environment, just guidance to protect the general situations.

I would also discuss with your local account team as they should know more about your environment and workloads.
Badge +7
Yes in our case capacity is not an issue. As for performance, even if there were a pair of all flash, there is no guarantee that the RF data for the AF node would sit in the other AF node right? Does VM host affinity controls where RF data goes?I don't mind losing performance if the AF node fails as even the Hybrid Nodes are quite decent specs for our workloads. An all flash would at least guarantee performance even for the cold tier for the most intensive VM sitting on that particular host if I got it right.
Userlevel 4
Badge +20
The affinity rules would not help with RF-placement as that data is spread throughout the cluster.
Badge +7
So to summarise, RF data woudl be spread to any hosts regardless if I have 1 or 2 AF nodes. In case I have 2 AF nodes, and One node were to fail, the vm would then migrate to the other AF node (If VM Affinity is set) but performance would still be based on where the data data resides until data locality kicks in? and only after data has been transferred over, which could be from SATA drives on other nodes.

So if space is not an issue as the 3 nodes are already accounting for more than we can chew,
what would be the reason to add an additional node to form a cluster of 4 nodes each with:

4 x 400GB SSD
6x 2TB HDD

Compared to having a cluster of 3 nodes each with:
4 x 400GB SSD6 x 2TB HDD
and a forth node with :
10 x 400GB SSD

From my logic, Option 1 will give me a small boost for all VM regardless of locations (Additional 4 x 400GB SSD, Option 2 will give a bigger boost to all VMS as they will have a bigger pool of SSDs to manage hot data. (Unless I am wrong) in cale of failure, then the data will be read either from the other nodes SSDs or HDDs depning on where the data resides and then data locality should kick in.

Now, where i am trying to understand the data allocation from an AF:
1: Will the cluster makes all the SSDs available for hot data or only the first 4 x would be used as hot data and the other 6 x SSD used as Cold?
2: If it is all of them, what happens when all 3 hybrid nodes SSDs goes above 75% ? will the data move to the cold tier of each nodes or will the new data be written to the all flash node as the SSD storage there would be less than 75%
3: If the data is written to the SSD of the AF, would read and write be slower as they will need to be done over the network instead of being local?
Userlevel 1
Badge +9
Hey Alex,

Since I know essentially nothing about your environment or use case, I'm providing answers based on general behaviors.

Having the AF node in the cluster does not directly relate to increasing the performance of all VMs in the cluser. Certainly VMs running on the AF node have access to more flash for predictable and lower latency performance. The AF node does further increase the SSD global tier of the cluster for spill over if hybrid nodes become full.

1. In AF nodes their is not the same tiering as hybrid nodes. So all SSD's are used. There is still a subset reserved for the write log (oplog) that is available locally and remotely if needed.

2. If the SSD tier on hybrid nodes reach that limit they will utilize flash from the global cluster pool before going to HDD's.

3. Any time you access data from a remote node regardless of it being hybrid or AF there is a slight difference in performance. This would be similar to accessing a SAN during that small period where locality is not available.
Badge +1
Hi,

I have some other questions about this subject:
  • If RF-data from AF node is written to ssd's of hybrid nodes, will this RF-data always stay in the flash tier of that node?
  • Other case if RF-data from hybrid node is written to AF node, where will this data migrate to if it has become cold?
  • Am I correct that you can only configure the affinity of a vm in AHV to run on an AF-node if you add at least 2 AF-nodes? Other words, there is no "should run on this node", when AF-node is down run on Hybrid nodes

Kind Regards,
Thibaut Noben

Reply