Disk Replacement is too slow and Cassandra metadata repair required


Badge +6
Hello!

Several days ago I've tried to replace two of my disks. The process stucked for several days on 72% and 95% for each disk. Time to time Prism displays the error:
Disk replacement for node 192.168.159.254 is progressing slowly. Total elapsed time: 9328 min(s). Node performing the operation: 192.168.159.252
After ncc healthckeck there are problems with one node Cassandra Metadata.

Detailed information for cvm_services_status:Node 192.168.159.252:FAIL: Components core dumped in last 15 mins: ['dynamic_ring_changer']Refer to KB 2472 (http://portal.nutanix.com/kb/2472) for details on cvm_services_status or Recheck with: ncc health_checks system_checks cvm_services_status --cvm_list=192.168.159.252Detailed information for incomplete_disk_removal_check:Node 192.168.159.254:FAIL: Disk 1051 (/home/nutanix/data/stargate-storage/disks/BTWL33950409240N) disk marked for removal but data not migratedFAIL: Disk 1053 (/home/nutanix/data/stargate-storage/disks/BTWL33950518240N) disk marked for removal but data not migratedRefer to KB 1545 (http://portal.nutanix.com/kb/1545) for details on incomplete_disk_removal_check or Recheck with: ncc health_checks hardware_checks disk_checks incomplete_disk_removal_checkDetailed information for recovery_point_availability_check:Node 192.168.159.254:WARN: Protection domain RD does not have a local backup to recover fromRefer to KB 1879 (http://portal.nutanix.com/kb/1879) for details on recovery_point_availability_check or Recheck with: ncc health_checks data_protection_checks protection_domain_checks recovery_point_availability_checkDetailed information for cluster_services_status:Node 192.168.159.254:FAIL: Components core dumped in last 24 hours: ['alert_manager', 'pithos', 'stargate', 'curator']Refer to KB 3378 (http://portal.nutanix.com/kb/3378) for details on cluster_services_status or Recheck with: ncc health_checks system_checks cluster_services_statusDetailed information for cassandra_status_check:Node 192.168.159.254:FAIL: CVM id: 1042 IP: 192.168.159.254 cassandra status is kMetadataRepairRequired, cassandra_auto_add_disabled: 0, casandra_auto_detach_disabled: 0Refer to KB 1547 (http://portal.nutanix.com/kb/1547) for details on cassandra_status_check or Recheck with: ncc health_checks cassandra_checks cassandra_status_checkDetailed information for curator_scan_time_elapsed_check:Node 192.168.159.254:FAIL: Time since last Full scan is beyond thresholdRefer to KB 1668 (http://portal.nutanix.com/kb/1668) for details on curator_scan_time_elapsed_check or Recheck with: ncc health_checks system_checks curator_scan_time_elapsed_checkDetailed information for orphan_vm_snapshot_check:Node 192.168.159.254:ERR : Acropolis client error when checking for orphan VM snapshotRefer to KB 3752 (http://portal.nutanix.com/kb/3752) for details on orphan_vm_snapshot_check or Recheck with: ncc health_checks hypervisor_checks orphan_vm_snapshot_check
Also allssh "source /etc/profile; nodetool -h localhost ring" output:

xecuting source /etc/profile; nodetool -h localhost ring on the cluster================== 192.168.159.251 =================Address Status State Load Owns Token zzzzzzzzg4vfzDd9Ue0iGPpRR84stizOlY6QZri1C77fvOqRMkp5VTWTt9o8192.168.159.254 Up Forwarding 4.76 GB 20.83% CupfKfKe0000000000000000000000000000000000000000000000000000192.168.159.251 Up Normal 7.28 GB 20.83% PpfKfKfK0000000000000000000000000000000000000000000000000000192.168.159.253 Up Normal 6.65 GB 25.00% fKfKfKfK8TyMTed6vi90avp1sQ11JzY7b73YogHlEcUnREq0fnvPR10zdEqn192.168.159.252 Up Normal 7.25 GB 33.33% zzzzzzzzg4vfzDd9Ue0iGPpRR84stizOlY6QZri1C77fvOqRMkp5VTWTt9o8Connection to 192.168.159.251 closed.================== 192.168.159.252 =================Address Status State Load Owns Token zzzzzzzzg4vfzDd9Ue0iGPpRR84stizOlY6QZri1C77fvOqRMkp5VTWTt9o8192.168.159.254 Up Forwarding 4.76 GB 20.83% CupfKfKe0000000000000000000000000000000000000000000000000000192.168.159.251 Up Normal 7.28 GB 20.83% PpfKfKfK0000000000000000000000000000000000000000000000000000192.168.159.253 Up Normal 6.65 GB 25.00% fKfKfKfK8TyMTed6vi90avp1sQ11JzY7b73YogHlEcUnREq0fnvPR10zdEqn192.168.159.252 Up Normal 7.25 GB 33.33% zzzzzzzzg4vfzDd9Ue0iGPpRR84stizOlY6QZri1C77fvOqRMkp5VTWTt9o8Connection to 192.168.159.252 closed.================== 192.168.159.253 =================Address Status State Load Owns Token zzzzzzzzg4vfzDd9Ue0iGPpRR84stizOlY6QZri1C77fvOqRMkp5VTWTt9o8192.168.159.254 Up Forwarding 4.76 GB 20.83% CupfKfKe0000000000000000000000000000000000000000000000000000192.168.159.251 Up Normal 7.28 GB 20.83% PpfKfKfK0000000000000000000000000000000000000000000000000000192.168.159.253 Up Normal 6.65 GB 25.00% fKfKfKfK8TyMTed6vi90avp1sQ11JzY7b73YogHlEcUnREq0fnvPR10zdEqn192.168.159.252 Up Normal 7.25 GB 33.33% zzzzzzzzg4vfzDd9Ue0iGPpRR84stizOlY6QZri1C77fvOqRMkp5VTWTt9o8Connection to 192.168.159.253 closed.================== 192.168.159.254 =================Address Status State Load Owns Token zzzzzzzzg4vfzDd9Ue0iGPpRR84stizOlY6QZri1C77fvOqRMkp5VTWTt9o8192.168.159.254 Up Forwarding 4.76 GB 20.83% CupfKfKe0000000000000000000000000000000000000000000000000000192.168.159.251 Up Normal 7.28 GB 20.83% PpfKfKfK0000000000000000000000000000000000000000000000000000192.168.159.253 Up Normal 6.65 GB 25.00% fKfKfKfK8TyMTed6vi90avp1sQ11JzY7b73YogHlEcUnREq0fnvPR10zdEqn192.168.159.252 Up Normal 7.25 GB 33.33% zzzzzzzzg4vfzDd9Ue0iGPpRR84stizOlY6QZri1C77fvOqRMkp5VTWTt9o8Connection to 192.168.159.254 closed.
What is the way to repair corrupted Cassandra Metadata and get rid of the stuck disks replacement?

This topic has been closed for comments

4 replies

Userlevel 3
Badge +20
Nutanix CE is a test environment which deals with disks in a completely different way then production nutanix. Most of CE users neither have RAID card or RAID card with JBOD or HBA option therefore CE uses LUN passthru instead of PCI passthru as in production nodes. In my experience, testing CE for the last 2 years when you get hard disks problems it is always better to rebuild the node then to add and remove disks before another node starts giving problem and you loose quorum.
I have 3 single node clusters and setup Production Domains with replication to the other 2 clusters. When I have problem with any node I simply activate the PD and rebuild the node and migrate the PD back to the rebuilt node. This whole process from start to finish takes about 3 hours.
Thanks
mswasif
Userlevel 3
Badge +20
As I explained earlier this problem is unique to CE. In production adding, removing and replacing disks is straight forward. If a problem is unique to CE and cannot be resolved quickly, easily, and efficiently I will rather rebuild the node. I encounter it with version upgrades sometimes and usb failure before moving ce boot to ssd. In the beginning I wasted days before deciding on rebuilding the node.Thanksmswasif
Badge +6
Thanks, mswasif!
As a final approach - I will rebuild entire cluster. But I am still looking for the possibility to force remove node and disk in Community Edition.
As for adding disk in CE - it seems that I have the solution.
Userlevel 3
Badge +20
You can try to force remove node using this command
ncli cluster remove-start id={node-id} force=true skip-space-check=true
Thanks
mswasif