Cluster not fully starting | Nutanix Community
Skip to main content

HP DL380P Gen 8

2x Xeon E5 2667

128GB RAM

boot: 128GB SSD

hypervisor: 1TB SSD

storage: 1TB HDD

=============================

Im attempting to create a single node cluster. Everything appears to install fine. I selected the “create single node cluster” option in the installer, and entered my local DNS server. I waited the ‘15 minutes’ after install for the cluster to start itself. I do see ports opening on the server, but not 8000:

 

Starting Nmap 7.94 ( https://nmap.org ) at 2024-08-15 16:04 Atlantic Daylight Time
Nmap scan report for 10.110.10.101
Host is up (0.0022s latency).
Not shown: 960 filtered tcp ports (no-response), 33 closed tcp ports (reset)
PORT     STATE SERVICE
22/tcp   open  ssh
111/tcp  open  rpcbind
2099/tcp open  h2250-annex-g
2100/tcp open  amiganetfs
2103/tcp open  zephyr-clt
9876/tcp open  sd
9877/tcp open  x510
MAC Address: 50:6B:8D:F7:85:3A (Nutanix)

==============================================

If I run a `cluster status` I see this:

nutanix@NTNX-cbf1e3ab-A-CVM:10.110.10.101:~$ cluster status
2024-08-15 12:06:08,460Z INFO MainThread zookeeper_session.py:191 cluster is attempting to connect to Zookeeper
2024-08-15 12:06:08,469Z INFO Dummy-1 zookeeper_session.py:625 ZK session establishment complete, sessionId=0x19155ecd3200009, negotiated timeout=20 secs
2024-08-15 12:06:08,478Z INFO MainThread cluster:2943 Executing action status on SVMs 10.110.10.101
The state of the cluster: start
Lockdown mode: Disabled

        CVM: 10.110.10.101 Up, ZeusLeader
                                Zeus   UP       4378, 4431, 4432, 4435, 4445, 4462]
                    SysStatCollector DOWN       3]
                           IkatProxy DOWN       ]
                    IkatControlPlane DOWN       ]
                       SSLTerminator DOWN       N]
                      SecureFileSync DOWN       ]
                              Medusa DOWN       ]
                  DynamicRingChanger DOWN       ]
                              Pithos DOWN       ]
                          InsightsDB DOWN       ]
                              Athena DOWN       D]
                             Mercury DOWN       ]
                              Mantle DOWN       ]
                            Stargate DOWN       s]
                InsightsDataTransfer DOWN       ]
                               Ergon DOWN       /]
                             GoErgon DOWN       ]
                             Cerebro DOWN       ]
                             Chronos DOWN       ]
                             Curator DOWN       ]
                               Prism DOWN       ]
                                Hera DOWN       e]
                                 CIM DOWN       ]
                        AlertManager DOWN       ]
                            Arithmos DOWN       o]
                             Catalog DOWN       ]
                           Acropolis DOWN       <]
                               Uhura DOWN       ]
                   NutanixGuestTools DOWN       ]
                          MinervaCVM DOWN       ]
                       ClusterConfig DOWN       ]
                         APLOSEngine DOWN       ]
                               APLOS DOWN       ]
                     PlacementSolver DOWN       ]
                               Lazan DOWN       ]
                             Polaris DOWN       ]
                              Delphi DOWN       ]
                            Security DOWN       O]
                                Flow DOWN       ]
                             Anduril DOWN       >]
                               XTrim DOWN       ]
                       ClusterHealth DOWN       ]
2024-08-15 12:06:10,756Z INFO MainThread cluster:3104 Success!

=========================================================

when I try to run `cluster start` this just posts ad nauseum:

nutanix@NTNX-cbf1e3ab-A-CVM:10.110.10.101:~$ cluster start
2024-08-15 12:15:38,980Z INFO MainThread zookeeper_session.py:191 cluster is attempting to connect to Zookeeper
2024-08-15 12:15:38,988Z INFO Dummy-1 zookeeper_session.py:625 ZK session establishment complete, sessionId=0x19155ecd3200014, negotiated timeout=20 secs
2024-08-15 12:15:38,989Z INFO MainThread cluster:2943 Executing action start on SVMs 10.110.10.101
Waiting on 10.110.10.101 (Up, ZeusLeader) to start:  SysStatCollector IkatProxy IkatControlPlane SSLTerminator SecureFileSync Medusa DynamicRingChanger Pithos InsightsDB Athena Mercury Mantle Stargate InsightsDataTransfer Ergon GoErgon Cerebro Chronos Curator Prism Hera CIM AlertManager Arithmos Catalog Acropolis Uhura NutanixGuestTools MinervaCVM ClusterConfig APLOSEngine APLOS PlacementSolver Lazan Polaris Delphi Security Flow Anduril XTrim ClusterHealth

Waiting on 10.110.10.101 (Up, ZeusLeader) to start:  SysStatCollector IkatProxy IkatControlPlane SSLTerminator SecureFileSync Medusa DynamicRingChanger Pithos InsightsDB Athena Mercury Mantle Stargate InsightsDataTransfer Ergon GoErgon Cerebro Chronos Curator Prism Hera CIM AlertManager Arithmos Catalog Acropolis Uhura NutanixGuestTools MinervaCVM ClusterConfig APLOSEngine APLOS PlacementSolver Lazan Polaris Delphi Security Flow Anduril XTrim ClusterHealth

Waiting on 10.110.10.101 (Up, ZeusLeader) to start:  SysStatCollector IkatProxy IkatControlPlane SSLTerminator SecureFileSync Medusa DynamicRingChanger Pithos InsightsDB Athena Mercury Mantle Stargate InsightsDataTransfer Ergon GoErgon Cerebro Chronos Curator Prism Hera CIM AlertManager Arithmos Catalog Acropolis Uhura NutanixGuestTools MinervaCVM ClusterConfig APLOSEngine APLOS PlacementSolver Lazan Polaris Delphi Security Flow Anduril XTrim ClusterHealth

Waiting on 10.110.10.101 (Up, ZeusLeader) to start:  SysStatCollector IkatProxy IkatControlPlane SSLTerminator SecureFileSync Medusa DynamicRingChanger Pithos InsightsDB Athena Mercury Mantle Stargate InsightsDataTransfer Ergon GoErgon Cerebro Chronos Curator Prism Hera CIM AlertManager Arithmos Catalog Acropolis Uhura NutanixGuestTools MinervaCVM ClusterConfig APLOSEngine APLOS PlacementSolver Lazan Polaris Delphi Security Flow Anduril XTrim ClusterHealth

Waiting on 10.110.10.101 (Up, ZeusLeader) to start:  SysStatCollector IkatProxy IkatControlPlane SSLTerminator SecureFileSync Medusa DynamicRingChanger Pithos InsightsDB Athena Mercury Mantle Stargate InsightsDataTransfer Ergon GoErgon Cerebro Chronos Curator Prism Hera CIM AlertManager Arithmos Catalog Acropolis Uhura NutanixGuestTools MinervaCVM ClusterConfig APLOSEngine APLOS PlacementSolver Lazan Polaris Delphi Security Flow Anduril XTrim ClusterHealth

====================================================

when I do a `watch -d genesis status` I see:

 

Every 2.0s: genesis status                                                                                                           Thu Aug 15 12:48:04 2024

2024-08-15 12:48:05.968698: Services running on this node:
  foundation: o]
  genesis: 2824, 2994, 3018, 3019]
  zookeeper: H4378, 4431, 4432, 4435, 4445, 4462]

=====================================================

 

I do see that the host has the DNS server in /etc/resolv.conf, but the CVM does not, and for the life of me I cant elevate to edit that file, even with the admin user.

Any help is greatly appreciated, thank you!