Prism central server is not reachable

Userlevel 1
Badge +1



I’m trying to figure out why We are unable to login in to Prism central as below message appear when trying to login:


as it show in the dev tools ( Failed to load resource ) 


I have checked the apache and its not working but not sure if the issue has anything to do with httpd

below is the status of httpd on the PCVM

Redirecting to /bin/systemctl status httpd.service
● httpd.service
   Loaded: masked (/dev/null; bad)
   Active: inactive (dead)
nutanix@NTNX-A-PCVM:~$ sudo service httpd start
Redirecting to /bin/systemctl start httpd.service
Failed to start httpd.service: Unit is masked.


after that checked if a service does not start or there is any FATAL logs

for i in `svmips`; do echo "CVM: $i"; ssh $i "ls -ltr /home/nutanix/data/logs/*.FATAL"; done




its look there is many services having problme


below is the output of lazan.FATAL: 

2022-05-10 08:00:27,810Z ERROR 82014 /src/bigtop/infra/infra_server/cluster/service_monitor/service_monitor.c:106 StartServiceMonitor: Child 78634 exited with status:

2022-05-10 08:03:41,698Z ERROR 82014 /src/bigtop/infra/infra_server/cluster/service_monitor/service_monitor.c:106 StartServiceMonitor: Child 92258 exited with status: 1
2022-05-10 08:06:56,303Z ERROR 82014 /src/bigtop/infra/infra_server/cluster/service_monitor/service_monitor.c:106 StartServiceMonitor: Child 106030 exited with status: 1
2022-05-10 08:10:10,281Z ERROR 82014 /src/bigtop/infra/infra_server/cluster/service_monitor/service_monitor.c:106 StartServiceMonitor: Child 119408 exited with status: 1
2022-05-10 08:13:26,794Z ERROR 82014 /src/bigtop/infra/infra_server/cluster/service_monitor/service_monitor.c:106 StartServiceMonitor: Child 2255 exited with status: 1


I See the same ERROR on all FATAL files ( atlas.FATAL catalog.FATAL uhura.FATAL lazan.FATAL)


I Notes neuron_server restarting alot below is the output of the neuron_server.log file:


2022-05-10 08:53:08Z ERROR Error while reading failed plugins file: /appliance/logical/serviceability/neuron_last_failed_plugins
2022-05-10 08:53:08Z INFO Start clean up of smart_alert_metadata entities from IDF
2022-05-10 08:53:08Z ERROR Exception occured during deletion of smart_alert_metadata entities: Failed to send RPC request.
2022-05-10 08:53:08Z INFO is attempting to connect to Zookeeper
2022-05-10 08:53:08Z INFO ZK session establishment complete, sessionId=0x2804ef58f8de8a9, negotiated timeout=20 secs
2022-05-10 08:53:08Z CRITICAL Traceback (most recent call last):
  File "build/bdist.linux-x86_64/egg/util/misc/", line 41, in wrapper
  File "/home/nutanix/neuron/bin/", line 274, in run
  File "/usr/local/nutanix/neuron/lib/py/nutanix_neuron.egg/neuron/utils/", line 58, in initialize_pc_services
    xfit_pc_type = self.__get_xfit_pc_type()
  File "/usr/local/nutanix/neuron/lib/py/nutanix_neuron.egg/neuron/utils/", line 110, in __get_xfit_pc_type
    nucalm_status = prism_central_utils.get_nucalm_enablement_flag()
  File "build/bdist.linux-x86_64/egg/util/prism_central/", line 1191, in get_nucalm_enablement_flag
ImportError: No module named proto.nucalm_enablement_pb2


its look like there is a python script not working ( )


I’m not sure what is the root cause, can anyone help with this issues 


thank you in advance


Best answer by rohan.saksena-55595 13 May 2022, 15:12

View original

This topic has been closed for comments

5 replies

Userlevel 1
Badge +1

Just want to update on this: Based on the command below two PCVM are in Forwarding. Any suggestions on how to solve this problem?

nutanix@NTNX-10-0-22-199-A-PCVM:~$ nodetool -h 0 ring

Address Status State Load Owns Token
X.X.X.199 Up Normal 340.3 GB 33.33%
X.X.X.197 Up Forwarding 38.62 GB 33.33%
X.X.X.198 Up Forwarding 407.12 GB 33.33%


Userlevel 2
Badge +7

Do you mean Prism element (i.e. the nodes themselves) or Prism Central (a separate deployment)?


What output do you get from the cluster status command? It looks like you have two CVMs down.




Userlevel 1
Badge +1

Hello @SteveCoops

No I mean Prism Central ( a separate deployment )

I have three PCVM two of them as shown are in (  Forwarding ) state  

Below is the output:

nutanix@NTNX-198-A-PCVM:~$ cluster status | grep -v UP
2022-05-13 10:24:35,114Z INFO MainThread cluster is attempting to connect to Zookeeper
2022-05-13 10:24:35,117Z INFO Dummy-1 ZK session establishment complete, sessionId=0x1804ee89c359f8f, negotiated timeout=20 secs
2022-05-13 10:24:35,120Z INFO MainThread cluster:2918 Executing action status on SVMs te of the cluster: start
Lockdown mode: Disabled

        CVM: X.X.X.197 Up, ZeusLeader

        CVM: X.X.X.198 Up

        CVM: X.X.X.199 Up
                             Epsilon DOWN       []

Epsilon is only down on the 199 CVM I do not think it has to do anything with the issues here!

Userlevel 2
Badge +3

Hello Abdulrhman,

Thanks for sharing details. This is a Live Troubleshooting Scenario. Please involve Nutanix Support through a Case (created via Support Portal).

Rohan Saksena

Userlevel 1
Badge +1

Hi @rohan

Case Created.

Thank you Guys