The expansion of cluster fails with an error log.
Could anyone tell me how to solve it?
<Detail>
When I try to add a node to a cluster in the prism element,
it fails with the error log below.
Failure in pre expand-cluster tests. Errors: Failed to start foundation service to do network validation SSD tier on nodes [u'xxx.xxx.xxx.xx'] is not sized to accomodate expected metadata growth on cluster. Refer KB-8842 for more details.
<Environment>
As shown in the diagram below, the Nutanix nodes are created
as one of the virtual machines on the ESXi.
used version : ce-2020.09.16.iso
<What I have tried>
・Improving the spec of SSDs in a virtual machine for the additional Nutanix node
SSD1 : 32GB → 64GB
SSD2 : 200GB → 400GB
SSD3 : 500GB → 1TB
・Confirming the result of KB-8842
-->The result of issuing commands are attached as "kb-8842.log"
in the bottom of this page.
<Re-production procedure>
1) Creating Nutanix nodes as one of the virtual machines on ESXi
2) Composing a cluster with 3 nodes
3) Expanding the cluster in the prism fails with the error log
Also, this issue occurs in the procedure below
1) Creating Nutanix nodes as one of the virtual machines on ESXi
2) Composing a cluster with 4 nodes
3) Deleting one node from the cluster
4) Try to add the deleted node to this cluster in the prism
5) This also fails with the same error log
<Background>
I would like to build an environment which enables engineers to test
some functions of Nutanix cluster for promoting sales without configuring every time.
For this purpose, we built the Nutanix nodes as one of the virtual machines
and saved them as templates on ESXi.
From these templates, the Nutanix nodes are supposed to be created, tested and deleted so that engineers can test them easily without configuring.
However, I faced the problem during the verification of this environment. Currently, this environment works except the expansion of the cluster.
It would be appreciated if anyone of you tell me how to solve this issue or advise me to
troubleshoot this problem.
Best regards,
kb-8842.log----------------------------------------------------------------------
nutanix@NTNX-a8ecfc4f-A-CVM:192.168.105.12:~$ allssh 'sudo find /home/nutanix/datta/stargate-storage/disks -maxdepth 3 -name cassandra -exec df -h {} \;|sort -ru''ncli sp ls
odetool -h 0 ringcli sp lsodetool -h 0 ring
Address Status State Load Owns Token
V0000000lFlVAK9hw68iPp9NIuMmNe4Eu4HcfyYJZf40dEyFegsKnLNddhpE
192.168.105.14 Up Normal 12.21 MB 50.00% 00000000aBzpuU8f32XGj7umFvcKovngyx1sWTHGmDIHAMwa9eb8aYRZpdkN
192.168.105.12 Up Normal 13.43 MB 25.00% FV00000082pry4lRFAL9iz6EJMW4eGLChDO535CuplCIiFk23h0k4v5MOkqE
192.168.105.13 Up Normal 10.84 MB 25.00% V0000000lFlVAK9hw68iPp9NIuMmNe4Eu4HcfyYJZf40dEyFegsKnLNddhpE
nutanix@NTNX-a8ecfc4f-A-CVM:192.168.105.12:~$
nutanix@NTNX-a8ecfc4f-A-CVM:192.168.105.12:~$
nutanix@NTNX-a8ecfc4f-A-CVM:192.168.105.12:~$
nutanix@NTNX-a8ecfc4f-A-CVM:192.168.105.12:~$ nodetool -h 0 ringallssh 'sudo find /home/nutanix/datta/stargate-storage/disks -maxdepth 3 -name cassandra -exec df -h {} \;|sort -ru''ncli sp lsId : 0005dd9e-e3f2-9280-07a5-005056aea63b::11
Uuid : 8966bf5a-79d3-4cb3-bc3a-607a87e53008
Name : default-storage-pool-80877259936142
Disk Count : 6
Disk Ids : 0005dd9e-e3f2-9280-07a5-005056aea63b::66, 0005dd9e-e3f2-9280-07a5-005056aea63b::71, 0005dd9e-e3f2-9280-07a5-005056aea63b::72, 0005dd9e-e3f2-9280-07a5-005056aea63b::74, 0005dd9e-e3f2-9280-07a5-005056aea63b::75, 0005dd9e-e3f2-9280-07a5-005056aea63b::65
Capacity (Physical) : 1.07 TiB (1,174,938,692,199 bytes)
Used (Physical) : 6.11 GiB (6,560,948,224 bytes)
Free (Physical) : 1.06 TiB (1,168,377,743,975 bytes)
nutanix@NTNX-a8ecfc4f-A-CVM:192.168.105.12:~$
nutanix@NTNX-a8ecfc4f-A-CVM:192.168.105.12:~$
nutanix@NTNX-a8ecfc4f-A-CVM:192.168.105.12:~$
nutanix@NTNX-a8ecfc4f-A-CVM:192.168.105.12:~$ ncli sp lsodetool -h 0 ringallssh 'sudo find /home/nutanix/datta/stargate-storage/disks -maxdepth 3 -name cassandra -exec df -h {} \;|sort -ru''
================== 192.168.105.13 =================
Filesystem Size Used Avail Use% Mounted on
/dev/sdb1 196G 1.2G 193G 1% /home/nutanix/data/stargate-storage/disks/0200000000000000
/dev/sda4 433G 1.3G 427G 1% /home/nutanix/data/stargate-storage/disks/0300000000000000
================== 192.168.105.14 =================
Filesystem Size Used Avail Use% Mounted on
/dev/sdb1 197G 1.1G 194G 1% /home/nutanix/data/stargate-storage/disks/0200000000000000
/dev/sda4 433G 1.1G 428G 1% /home/nutanix/data/stargate-storage/disks/0300000000000000
================== 192.168.105.12 =================
Filesystem Size Used Avail Use% Mounted on
/dev/sdb1 196G 1.3G 193G 1% /home/nutanix/data/stargate-storage/disks/0200000000000000
/dev/sda4 433G 1.3G 427G 1% /home/nutanix/data/stargate-storage/disks/0300000000000000
nutanix@NTNX-a8ecfc4f-A-CVM:192.168.105.12:~$