Hello All,We are trying to deploy prism central on top of 2 node cluster but that prism central fails to deploy at 100%.Error message that it gives - Encountered Exception in post_deployment step: Failed to enable micro services infrastructure on PC: deploy msp:Error deploying addons: failed to deploy monitoring addon: failed to deploy and verify kube-prometheus: failed to verify kube-prometheus: Operation timed out: failed to verify kube-prometheus: expecting 1 available replica of k8s prometheus in ntnx-system namespace. Currently running: 0Trying to get the pods up but that isnt possible.Checked the resources for the hardware and they are also available.Need help to resolve this issueBelow are the versions - AHV - 11.0AOS- 7.5Prism - 7.5Troubleshooting done so far - Attempted to manually bring the affected pods up and runningDeleted the problematic podEnsured all components are configured in the same time zone (UTC – default for Nutanix)

When the prism is being deployed the deployment fails at 100% where one of he pos is not coming up

+1

selvamani
Trailblazer
Forum|Forum|16 days ago
January 30, 2026

Hi Tushar ,

Prism Central is not supported on a 2-node cluster
Prism Central requires a minimum 3-node cluster to reliably deploy and run its Kubernetes-based microservices. On 2-node clusters, pods like prometheus often fail to schedule or remain pending, which matches what you’re seeing.

I hope thiis is not a raw hardware resource issue
Even if CPU and memory look available, Kubernetes scheduling still fails due to platform constraints on 2-node clusters. That’s why the prometheus replica count stays at 0.

Can you check to confirm following :

kubectl get pods -n ntnx-system
kubectl describe pod <prometheus-pod-name> -n ntnx-system
ncli cluster info

This is the root cause of 2 node cluster

The supported and reliable fix is to expand the cluster to 3 nodes and then redeploy Prism Central. Once the third node is added, the MSP services (including kube-prometheus) should deploy successfully and the PC deployment will complete.

If expanding the cluster is not possible, Prism Central deployment on that environment is not supported, and the failure at 100% is expected behavior.

If you want, share the pod describe output here and we can validate it, but based on the versions and symptoms,I am sure this is almost certainly a cluster sizing limitation rather than a software defect.

Thanks
Selvamani.S

Like

T

+1

Tushar_Tarkas
Author
Trailblazer
Forum|Forum|16 days ago
January 30, 2026

Hi Tushar ,

Prism Central is not supported on a 2-node cluster
Prism Central requires a minimum 3-node cluster to reliably deploy and run its Kubernetes-based microservices. On 2-node clusters, pods like prometheus often fail to schedule or remain pending, which matches what you’re seeing.

I hope thiis is not a raw hardware resource issue
Even if CPU and memory look available, Kubernetes scheduling still fails due to platform constraints on 2-node clusters. That’s why the prometheus replica count stays at 0.

Can you check to confirm following :

kubectl get pods -n ntnx-system
kubectl describe pod <prometheus-pod-name> -n ntnx-system
ncli cluster info

This is the root cause of 2 node cluster

The supported and reliable fix is to expand the cluster to 3 nodes and then redeploy Prism Central. Once the third node is added, the MSP services (including kube-prometheus) should deploy successfully and the PC deployment will complete.

If expanding the cluster is not possible, Prism Central deployment on that environment is not supported, and the failure at 100% is expected behavior.

If you want, share the pod describe output here and we can validate it, but based on the versions and symptoms,I am sure this is almost certainly a cluster sizing limitation rather than a software defect.

Thanks
Selvamani.S

We have one more such setup that is running on a 2 node cluster

Here are the output of the commands -
nutanix@NTNX-172-30-30-15-A-PCVM:~$ kubectl get pods -n ntnx-system
NAME                                                    READY   STATUS      RESTAR                                                                                                                            TS      AGE
alertmanager-main-0                                     0/2     Completed   0                                                                                                                                         11h
csi-snapshot-controller-6f9b6648bc-85hjj                0/1     Error       0                                                                                                                                         11h
csi-snapshot-webhook-594df44d75-9ckq8                   0/1     Error       0                                                                                                                                         11h
fluent-bit-2ptsv                                        0/1     Completed   0                                                                                                                                         11h
kube-state-metrics-7979976844-9lsmd                     0/3     Completed   0                                                                                                                                         11h
mutator-webhook-dep-54887559f5-949b7                    1/1     Running     1 (10h                                                                                                                             ago)   11h
node-exporter-th9j2                                     2/2     Running     2 (10h                                                                                                                             ago)   11h
ntnx-cluster-maintainer-58d44b85bb-drqkz                0/1     Completed   0                                                                                                                                         11h
ntnx-k8s-cluster-maintainer-operator-5c88ff6467-47lpc   0/1     Completed   0                                                                                                                                         11h
nutanix-csi-controller-868bb84888-r8sqp                 7/7     Running     7 (10h                                                                                                                             ago)   11h
nutanix-csi-node-mcjl8                                  3/3     Running     3 (10h                                                                                                                             ago)   11h
prometheus-k8s-0                                        0/2     Pending     0                                                                                                                                         11h
prometheus-operator-699f88d6dc-q2xl4                    0/2     Completed   0                                                                                                                                         11h

Like

T

+1

Tushar_Tarkas
Author
Trailblazer
Forum|Forum|16 days ago
January 30, 2026

We do have a 2 node cluster up and running on the other setup-
Below is the output of the commands
nutanix@NTNX-172-30-30-15-A-PCVM:~$ kubectl describe pod prometheus-k8s-0 -n ntnx-system
Name:                 prometheus-k8s-0
Namespace:            ntnx-system
Priority:             200000000
Priority Class Name: ntnx-system-cluster-critical
Service Account:      prometheus-k8s
Node:                 <none>
Labels:               app.kubernetes.io/component=prometheus
                      app.kubernetes.io/instance=k8s
                      app.kubernetes.io/managed-by=prometheus-operator
                      app.kubernetes.io/name=prometheus
                      app.kubernetes.io/part-of=kube-prometheus
                      app.kubernetes.io/version=2.36.1
                      apps.kubernetes.io/pod-index=0
                      controller-revision-hash=prometheus-k8s-7669bbdcf
                      operator.prometheus.io/name=k8s
                      operator.prometheus.io/shard=0
                      prometheus=k8s
                      statefulset.kubernetes.io/pod-name=prometheus-k8s-0
Annotations:          kubectl.kubernetes.io/default-container: prometheus
Status:               Pending
IP:
IPs:                  <none>
Controlled By:        StatefulSet/prometheus-k8s
Init Containers:
  init-config-reloader:
    Image:      nutanix/prometheus-config-reloader:v0.57.0
    Port:       8080/TCP
    Host Port: 0/TCP
    Command:
      /bin/prometheus-config-reloader
    Args:
      --watch-interval=0
      --listen-address=:8080
      --config-file=/etc/prometheus/config/prometheus.yaml.gz
      --config-envsubst-file=/etc/prometheus/config_out/prometheus.env.yaml
      --watched-dir=/etc/prometheus/rules/prometheus-k8s-rulefiles-0
    Limits:
      cpu:     100m
      memory: 50Mi
    Requests:
      cpu:     100m
      memory: 50Mi
    Environment:
      POD_NAME: prometheus-k8s-0 (v1:metadata.name)
      SHARD:     0
    Mounts:
      /etc/prometheus/config from config (rw)
      /etc/prometheus/config_out from config-out (rw)
      /etc/prometheus/rules/prometheus-k8s-rulefiles-0 from prometheus-k8s-rulefiles-0 (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-zwdcp (ro)
Containers:
  prometheus:
    Image:      nutanix/prometheus:v2.36.1
    Port:       9090/TCP
    Host Port: 0/TCP
    Args:
      --web.console.templates=/etc/prometheus/consoles
      --web.console.libraries=/etc/prometheus/console_libraries
      --storage.tsdb.retention.time=10d
      --storage.tsdb.retention.size=15GB
      --config.file=/etc/prometheus/config_out/prometheus.env.yaml
      --storage.tsdb.path=/prometheus
      --web.enable-lifecycle
      --web.route-prefix=/
      --web.config.file=/etc/prometheus/web_config/web-config.yaml
    Limits:
      cpu:     500m
      memory: 1536Mi
    Requests:
      cpu:        200m
      memory:     750Mi
    Liveness:     http-get http://:web/-/healthy delay=0s timeout=3s period=5s #success=1 #failure=6
    Readiness:    http-get http://:web/-/ready delay=0s timeout=3s period=5s #success=1 #failure=3
    Startup:      http-get http://:web/-/ready delay=0s timeout=3s period=15s #success=1 #failure=60
    Environment: <none>
    Mounts:
      /etc/prometheus/certs from tls-assets (ro)
      /etc/prometheus/config_out from config-out (ro)
      /etc/prometheus/rules/prometheus-k8s-rulefiles-0 from prometheus-k8s-rulefiles-0 (rw)
      /etc/prometheus/secrets/kube-etcd-client-certs from secret-kube-etcd-client-certs (ro)
      /etc/prometheus/secrets/kubelet-client-certs from secret-kubelet-client-certs (ro)
      /etc/prometheus/secrets/prometheus-certs from secret-prometheus-certs (ro)
      /etc/prometheus/web_config/web-config.yaml from web-config (ro,path="web-config.yaml")
      /prometheus from prometheus-k8s-db (rw,path="prometheus-db")
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-zwdcp (ro)
  config-reloader:
    Image:      nutanix/prometheus-config-reloader:v0.57.0
    Port:       8080/TCP
    Host Port: 0/TCP
    Command:
      /bin/prometheus-config-reloader
    Args:
      --listen-address=:8080
      --reload-url=http://127.0.0.1:9090/-/reload
      --config-file=/etc/prometheus/config/prometheus.yaml.gz
      --config-envsubst-file=/etc/prometheus/config_out/prometheus.env.yaml
      --watched-dir=/etc/prometheus/rules/prometheus-k8s-rulefiles-0
    Limits:
      cpu:     100m
      memory: 50Mi
    Requests:
      cpu:     100m
      memory: 50Mi
    Environment:
      POD_NAME: prometheus-k8s-0 (v1:metadata.name)
      SHARD:     0
    Mounts:
      /etc/prometheus/config from config (rw)
      /etc/prometheus/config_out from config-out (rw)
      /etc/prometheus/rules/prometheus-k8s-rulefiles-0 from prometheus-k8s-rulefiles-0 (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-zwdcp (ro)
Conditions:
  Type           Status
  PodScheduled   False
Volumes:
  prometheus-k8s-db:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName: prometheus-k8s-db-prometheus-k8s-0
    ReadOnly:   false
  config:
    Type:        Secret (a volume populated by a Secret)
    SecretName: prometheus-k8s
    Optional:    false
  tls-assets:
    Type:                Projected (a volume that contains injected data from multiple sources)
    SecretName:          prometheus-k8s-tls-assets-0
    SecretOptionalName: <nil>
  config-out:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit: <unset>
  prometheus-k8s-rulefiles-0:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      prometheus-k8s-rulefiles-0
    Optional: false
  web-config:
    Type:        Secret (a volume populated by a Secret)
    SecretName: prometheus-k8s-web-config
    Optional:    false
  secret-kube-etcd-client-certs:
    Type:        Secret (a volume populated by a Secret)
    SecretName: kube-etcd-client-certs
    Optional:    false
  secret-kubelet-client-certs:
    Type:        Secret (a volume populated by a Secret)
    SecretName: kubelet-client-certs
    Optional:    false
  secret-prometheus-certs:
    Type:        Secret (a volume populated by a Secret)
    SecretName: prometheus-certs
    Optional:    false
  kube-api-access-zwdcp:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds: 3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              kubernetes.io/os=linux
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 30s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 30s
Events:
  Type     Reason            Age                 From               Message
  ----     ------            ----                ----               -------
  Warning FailedScheduling 10h (x17 over 11h) default-scheduler 0/1 nodes are available: pod has unbound immediate PersistentVolumeClaims.                                                                 preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling..
  Warning FailedScheduling 34s (x2 over 85s)   default-scheduler 0/1 nodes are available: pod has unbound immediate PersistentVolumeClaims.                                                                 preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling..
nutanix@NTNX-172-30-30-15-A-PCVM:~$ ncli cluster info

Error: Cannot connect to Prism Gateway

Like

R

Rishi99
Adventurer
Forum|Forum|16 days ago
January 30, 2026

Thanks, sharing output here

Like

T

+1

Tushar_Tarkas
Author
Trailblazer
Forum|Forum|16 days ago
January 30, 2026

nutanix@NTNX-172-30-30-15-A-PCVM:~$ kubectl describe pod prometheus-k8s-0 -n ntnx-system
Name:                 prometheus-k8s-0
Namespace:            ntnx-system
Priority:             200000000
Priority Class Name: ntnx-system-cluster-critical
Service Account:      prometheus-k8s
Node:                 <none>
Labels:               app.kubernetes.io/component=prometheus
                      app.kubernetes.io/instance=k8s
                      app.kubernetes.io/managed-by=prometheus-operator
                      app.kubernetes.io/name=prometheus
                      app.kubernetes.io/part-of=kube-prometheus
                      app.kubernetes.io/version=2.36.1
                      apps.kubernetes.io/pod-index=0
                      controller-revision-hash=prometheus-k8s-7669bbdcf
                      operator.prometheus.io/name=k8s
                      operator.prometheus.io/shard=0
                      prometheus=k8s
                      statefulset.kubernetes.io/pod-name=prometheus-k8s-0
Annotations:          kubectl.kubernetes.io/default-container: prometheus
Status:               Pending
IP:
IPs:                  <none>
Controlled By:        StatefulSet/prometheus-k8s
Init Containers:
init-config-reloader:
    Image:      nutanix/prometheus-config-reloader:v0.57.0
    Port:       8080/TCP
    Host Port: 0/TCP
    Command:
      /bin/prometheus-config-reloader
    Args:
      --watch-interval=0
      --listen-address=:8080
      --config-file=/etc/prometheus/config/prometheus.yaml.gz
      --config-envsubst-file=/etc/prometheus/config_out/prometheus.env.yaml
      --watched-dir=/etc/prometheus/rules/prometheus-k8s-rulefiles-0
    Limits:
      cpu:     100m
      memory: 50Mi
    Requests:
      cpu:     100m
      memory: 50Mi
    Environment:
      POD_NAME: prometheus-k8s-0 (v1:metadata.name)
      SHARD:     0
    Mounts:
      /etc/prometheus/config from config (rw)
      /etc/prometheus/config_out from config-out (rw)
      /etc/prometheus/rules/prometheus-k8s-rulefiles-0 from prometheus-k8s-rulefiles-0 (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-zwdcp (ro)
Containers:
prometheus:
    Image:      nutanix/prometheus:v2.36.1
    Port:       9090/TCP
    Host Port: 0/TCP
    Args:
      --web.console.templates=/etc/prometheus/consoles
      --web.console.libraries=/etc/prometheus/console_libraries
      --storage.tsdb.retention.time=10d
      --storage.tsdb.retention.size=15GB
      --config.file=/etc/prometheus/config_out/prometheus.env.yaml
      --storage.tsdb.path=/prometheus
      --web.enable-lifecycle
      --web.route-prefix=/
      --web.config.file=/etc/prometheus/web_config/web-config.yaml
    Limits:
      cpu:     500m
      memory: 1536Mi
    Requests:
      cpu:        200m
      memory:     750Mi
    Liveness:     http-get http://:web/-/healthy delay=0s timeout=3s period=5s #success=1 #failure=6
    Readiness:    http-get http://:web/-/ready delay=0s timeout=3s period=5s #success=1 #failure=3
    Startup:      http-get http://:web/-/ready delay=0s timeout=3s period=15s #success=1 #failure=60
    Environment: <none>
    Mounts:
      /etc/prometheus/certs from tls-assets (ro)
      /etc/prometheus/config_out from config-out (ro)
      /etc/prometheus/rules/prometheus-k8s-rulefiles-0 from prometheus-k8s-rulefiles-0 (rw)
      /etc/prometheus/secrets/kube-etcd-client-certs from secret-kube-etcd-client-certs (ro)
      /etc/prometheus/secrets/kubelet-client-certs from secret-kubelet-client-certs (ro)
      /etc/prometheus/secrets/prometheus-certs from secret-prometheus-certs (ro)
      /etc/prometheus/web_config/web-config.yaml from web-config (ro,path="web-config.yaml")
      /prometheus from prometheus-k8s-db (rw,path="prometheus-db")
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-zwdcp (ro)
config-reloader:
    Image:      nutanix/prometheus-config-reloader:v0.57.0
    Port:       8080/TCP
    Host Port: 0/TCP
    Command:
      /bin/prometheus-config-reloader
    Args:
      --listen-address=:8080
      --reload-url=http://127.0.0.1:9090/-/reload
      --config-file=/etc/prometheus/config/prometheus.yaml.gz
      --config-envsubst-file=/etc/prometheus/config_out/prometheus.env.yaml
      --watched-dir=/etc/prometheus/rules/prometheus-k8s-rulefiles-0
    Limits:
      cpu:     100m
      memory: 50Mi
    Requests:
      cpu:     100m
      memory: 50Mi
    Environment:
      POD_NAME: prometheus-k8s-0 (v1:metadata.name)
      SHARD:     0
    Mounts:
      /etc/prometheus/config from config (rw)
      /etc/prometheus/config_out from config-out (rw)
      /etc/prometheus/rules/prometheus-k8s-rulefiles-0 from prometheus-k8s-rulefiles-0 (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-zwdcp (ro)
Conditions:
Type           Status
PodScheduled   False
Volumes:
prometheus-k8s-db:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName: prometheus-k8s-db-prometheus-k8s-0
    ReadOnly:   false
config:
    Type:        Secret (a volume populated by a Secret)
    SecretName: prometheus-k8s
    Optional:    false
tls-assets:
    Type:                Projected (a volume that contains injected data from multiple sources)
    SecretName:          prometheus-k8s-tls-assets-0
    SecretOptionalName: <nil>
config-out:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit: <unset>
prometheus-k8s-rulefiles-0:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      prometheus-k8s-rulefiles-0
    Optional: false
web-config:
    Type:        Secret (a volume populated by a Secret)
    SecretName: prometheus-k8s-web-config
    Optional:    false
secret-kube-etcd-client-certs:
    Type:        Secret (a volume populated by a Secret)
    SecretName: kube-etcd-client-certs
    Optional:    false
secret-kubelet-client-certs:
    Type:        Secret (a volume populated by a Secret)
    SecretName: kubelet-client-certs
    Optional:    false
secret-prometheus-certs:
    Type:        Secret (a volume populated by a Secret)
    SecretName: prometheus-certs
    Optional:    false
kube-api-access-zwdcp:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds: 3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              kubernetes.io/os=linux
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 30s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 30s
Events:
Type     Reason            Age                   From               Message
----     ------            ----                  ----               -------
Warning FailedScheduling 17m (x55 over 4h43m) default-scheduler 0/1 nodes are available: pod has unbound immediate PersistentVolumeClaims. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling..

    Cluster Id                : 6f1f026e-16cb-47ed-ad8a-da17319a9b01::3281675071644146433
    Cluster Uuid              : 6f1f026e-16cb-47ed-ad8a-da17319a9b01
    Cluster Name              : prism
    Cluster Version           : pc.7.5.0.5
    Cluster Full Version      : el8.5-release-ganges-7.5.0.5-stable-33e266ab5c96cdbe92aa9ea64584d38bbea13cc8
    Is LTS                    : false
    External Data Services... :
    Support Verbosity Level   : BASIC_COREDUMP
    Lock Down Status          : Disabled
    Password Remote Login ... : Enabled
    Timezone                  : Atlantic/Reykjavik
    NCC Version               : ncc-5.3.0
    Degraded Node Monitoring : Enabled

Like

R

Rishi99
Adventurer
Forum|Forum|16 days ago
January 30, 2026

nutanix@NTNX-172-30-30-15-A-PCVM:~$ kubectl describe pod prometheus-k8s-0 -n ntnx-system
Name:                 prometheus-k8s-0
Namespace:            ntnx-system
Priority:             200000000
Priority Class Name: ntnx-system-cluster-critical
Service Account:      prometheus-k8s
Node:                 <none>
Labels:               app.kubernetes.io/component=prometheus
                      app.kubernetes.io/instance=k8s
                      app.kubernetes.io/managed-by=prometheus-operator
                      app.kubernetes.io/name=prometheus
                      app.kubernetes.io/part-of=kube-prometheus
                      app.kubernetes.io/version=2.36.1
                      apps.kubernetes.io/pod-index=0
                      controller-revision-hash=prometheus-k8s-7669bbdcf
                      operator.prometheus.io/name=k8s
                      operator.prometheus.io/shard=0
                      prometheus=k8s
                      statefulset.kubernetes.io/pod-name=prometheus-k8s-0
Annotations:          kubectl.kubernetes.io/default-container: prometheus
Status:               Pending
IP:
IPs:                  <none>
Controlled By:        StatefulSet/prometheus-k8s
Init Containers:
init-config-reloader:
    Image:      nutanix/prometheus-config-reloader:v0.57.0
    Port:       8080/TCP
    Host Port: 0/TCP
    Command:
      /bin/prometheus-config-reloader
    Args:
      --watch-interval=0
      --listen-address=:8080
      --config-file=/etc/prometheus/config/prometheus.yaml.gz
      --config-envsubst-file=/etc/prometheus/config_out/prometheus.env.yaml
      --watched-dir=/etc/prometheus/rules/prometheus-k8s-rulefiles-0
    Limits:
      cpu:     100m
      memory: 50Mi
    Requests:
      cpu:     100m
      memory: 50Mi
    Environment:
      POD_NAME: prometheus-k8s-0 (v1:metadata.name)
      SHARD:     0
    Mounts:
      /etc/prometheus/config from config (rw)
      /etc/prometheus/config_out from config-out (rw)
      /etc/prometheus/rules/prometheus-k8s-rulefiles-0 from prometheus-k8s-rulefiles-0 (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-zwdcp (ro)
Containers:
prometheus:
    Image:      nutanix/prometheus:v2.36.1
    Port:       9090/TCP
    Host Port: 0/TCP
    Args:
      --web.console.templates=/etc/prometheus/consoles
      --web.console.libraries=/etc/prometheus/console_libraries
      --storage.tsdb.retention.time=10d
      --storage.tsdb.retention.size=15GB
      --config.file=/etc/prometheus/config_out/prometheus.env.yaml
      --storage.tsdb.path=/prometheus
      --web.enable-lifecycle
      --web.route-prefix=/
      --web.config.file=/etc/prometheus/web_config/web-config.yaml
    Limits:
      cpu:     500m
      memory: 1536Mi
    Requests:
      cpu:        200m
      memory:     750Mi
    Liveness:     http-get http://:web/-/healthy delay=0s timeout=3s period=5s #success=1 #failure=6
    Readiness:    http-get http://:web/-/ready delay=0s timeout=3s period=5s #success=1 #failure=3
    Startup:      http-get http://:web/-/ready delay=0s timeout=3s period=15s #success=1 #failure=60
    Environment: <none>
    Mounts:
      /etc/prometheus/certs from tls-assets (ro)
      /etc/prometheus/config_out from config-out (ro)
      /etc/prometheus/rules/prometheus-k8s-rulefiles-0 from prometheus-k8s-rulefiles-0 (rw)
      /etc/prometheus/secrets/kube-etcd-client-certs from secret-kube-etcd-client-certs (ro)
      /etc/prometheus/secrets/kubelet-client-certs from secret-kubelet-client-certs (ro)
      /etc/prometheus/secrets/prometheus-certs from secret-prometheus-certs (ro)
      /etc/prometheus/web_config/web-config.yaml from web-config (ro,path="web-config.yaml")
      /prometheus from prometheus-k8s-db (rw,path="prometheus-db")
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-zwdcp (ro)
config-reloader:
    Image:      nutanix/prometheus-config-reloader:v0.57.0
    Port:       8080/TCP
    Host Port: 0/TCP
    Command:
      /bin/prometheus-config-reloader
    Args:
      --listen-address=:8080
      --reload-url=http://127.0.0.1:9090/-/reload
      --config-file=/etc/prometheus/config/prometheus.yaml.gz
      --config-envsubst-file=/etc/prometheus/config_out/prometheus.env.yaml
      --watched-dir=/etc/prometheus/rules/prometheus-k8s-rulefiles-0
    Limits:
      cpu:     100m
      memory: 50Mi
    Requests:
      cpu:     100m
      memory: 50Mi
    Environment:
      POD_NAME: prometheus-k8s-0 (v1:metadata.name)
      SHARD:     0
    Mounts:
      /etc/prometheus/config from config (rw)
      /etc/prometheus/config_out from config-out (rw)
      /etc/prometheus/rules/prometheus-k8s-rulefiles-0 from prometheus-k8s-rulefiles-0 (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-zwdcp (ro)
Conditions:
Type           Status
PodScheduled   False
Volumes:
prometheus-k8s-db:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName: prometheus-k8s-db-prometheus-k8s-0
    ReadOnly:   false
config:
    Type:        Secret (a volume populated by a Secret)
    SecretName: prometheus-k8s
    Optional:    false
tls-assets:
    Type:                Projected (a volume that contains injected data from multiple sources)
    SecretName:          prometheus-k8s-tls-assets-0
    SecretOptionalName: <nil>
config-out:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit: <unset>
prometheus-k8s-rulefiles-0:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      prometheus-k8s-rulefiles-0
    Optional: false
web-config:
    Type:        Secret (a volume populated by a Secret)
    SecretName: prometheus-k8s-web-config
    Optional:    false
secret-kube-etcd-client-certs:
    Type:        Secret (a volume populated by a Secret)
    SecretName: kube-etcd-client-certs
    Optional:    false
secret-kubelet-client-certs:
    Type:        Secret (a volume populated by a Secret)
    SecretName: kubelet-client-certs
    Optional:    false
secret-prometheus-certs:
    Type:        Secret (a volume populated by a Secret)
    SecretName: prometheus-certs
    Optional:    false
kube-api-access-zwdcp:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds: 3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              kubernetes.io/os=linux
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 30s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 30s
Events:
Type     Reason            Age                   From               Message
----     ------            ----                  ----               -------
Warning FailedScheduling 17m (x55 over 4h43m) default-scheduler 0/1 nodes are available: pod has unbound immediate PersistentVolumeClaims. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling..

    Cluster Id                : 6f1f026e-16cb-47ed-ad8a-da17319a9b01::3281675071644146433
    Cluster Uuid              : 6f1f026e-16cb-47ed-ad8a-da17319a9b01
    Cluster Name              : prism
    Cluster Version           : pc.7.5.0.5
    Cluster Full Version      : el8.5-release-ganges-7.5.0.5-stable-33e266ab5c96cdbe92aa9ea64584d38bbea13cc8
    Is LTS                    : false
    External Data Services... :
    Support Verbosity Level   : BASIC_COREDUMP
    Lock Down Status          : Disabled
    Password Remote Login ... : Enabled
    Timezone                  : Atlantic/Reykjavik
    NCC Version               : ncc-5.3.0
    Degraded Node Monitoring : Enabled

Like

J

+3

jamali.ahmad
Outrider
Forum|Forum|13 days ago
February 2, 2026

Hi @selvamani ,

when you are saying PC is not supported on 2 node cluster, is there any official document for this ?is it same for single node PC and scale-out PC?

Hi @Rishi99

are you trying to install a scaleout PC or single node PC on your cluster?
have you tried installing a single node PC and see if it works?

Like

+1

selvamani
Trailblazer
Forum|Forum|13 days ago
February 2, 2026

Hi @jamali.ahmad ,

You are right, and I want to clarify my earlier message. Prism Central can run on a 2-node cluster, but there is an important limitation with Microservices (MSP).

What’s happening is that Microservices require 3 nodes. The Microservices Infrastructure runs Kubernetes inside Prism Central and needs at least 3 nodes to work reliably.

On a 2-node cluster, the system often cannot start the kube-prometheus pods during post-deployment. This causes the installation to get stuck and eventually fail.

The Prism Central VM itself is supported on 2 nodes, but enabling Microservices on a 2-node cluster is unstable and commonly fails.

If you cannot add a third node, redeploy Prism Central and do not enable Microservices Infrastructure. This will allow the deployment to complete successfully and still provide standard Prism Central management features.

Sorry for the earlier confusion.

Thanks
Selvamani.S

Like

T

+1

Tushar_Tarkas
Author
Trailblazer
Forum|Forum|13 days ago
February 2, 2026

nutanix@NTNX-172-30-30-15-A-PCVM:~$ kubectl get pods -n ntnx-system
NAME READY STATUS RESTARTS AGE
alertmanager-main-0 2/2 Running 2 (15h ago) 16h
csi-snapshot-controller-6f9b6648bc-85hjj 1/1 Running 1 (15h ago) 16h
csi-snapshot-webhook-594df44d75-9ckq8 1/1 Running 1 (15h ago) 16h
fluent-bit-2ptsv 1/1 Running 1 (15h ago) 16h
kube-state-metrics-7979976844-9lsmd 3/3 Running 3 (15h ago) 16h
mutator-webhook-dep-54887559f5-949b7 1/1 Running 1 (15h ago) 16h
node-exporter-th9j2 2/2 Running 2 (15h ago) 16h
ntnx-cluster-maintainer-6b588f4db4-jb4ht 1/1 Running 0 4h41m
ntnx-k8s-cluster-maintainer-operator-5c88ff6467-47lpc 1/1 Running 1 (15h ago) 16h
nutanix-csi-controller-868bb84888-r8sqp 7/7 Running 14 (4h42m ago) 16h
nutanix-csi-node-mcjl8 3/3 Running 6 (4h42m ago) 16h
prometheus-k8s-0 0/2 Pending 0 16h
prometheus-operator-699f88d6dc-q2xl4 2/2 Running 2 (15h ago) 16h

Like

T

+1

Tushar_Tarkas
Author
Trailblazer
Forum|Forum|13 days ago
February 2, 2026

Cluster Id                : 6f1f026e-16cb-47ed-ad8a-da17319a9b01::3281675071644146433
    Cluster Uuid              : 6f1f026e-16cb-47ed-ad8a-da17319a9b01
    Cluster Name              : prism
    Cluster Version           : pc.7.5.0.5
    Cluster Full Version      : el8.5-release-ganges-7.5.0.5-stable-33e266ab5c96cdbe92aa9ea64584d38bbea13cc8
    Is LTS                    : false
    External Data Services... :
    Support Verbosity Level   : BASIC_COREDUMP
    Lock Down Status          : Disabled
    Password Remote Login ... : Enabled
    Timezone                  : Atlantic/Reykjavik
    NCC Version               : ncc-5.3.0
    Degraded Node Monitoring : Enabled

Like

T

+1

Tushar_Tarkas
Author
Trailblazer
Forum|Forum|13 days ago
February 2, 2026

nutanix@NTNX-172-30-30-15-A-PCVM:~$ kubectl describe pod prometheus-k8s-0 -n ntnx-system
Name:                 prometheus-k8s-0
Namespace:            ntnx-system
Priority:             200000000
Priority Class Name: ntnx-system-cluster-critical
Service Account:      prometheus-k8s
Node:                 <none>
Labels:               app.kubernetes.io/component=prometheus
                      app.kubernetes.io/instance=k8s
                      app.kubernetes.io/managed-by=prometheus-operator
                      app.kubernetes.io/name=prometheus
                      app.kubernetes.io/part-of=kube-prometheus
                      app.kubernetes.io/version=2.36.1
                      apps.kubernetes.io/pod-index=0
                      controller-revision-hash=prometheus-k8s-7669bbdcf
                      operator.prometheus.io/name=k8s
                      operator.prometheus.io/shard=0
                      prometheus=k8s
                      statefulset.kubernetes.io/pod-name=prometheus-k8s-0
Annotations:          kubectl.kubernetes.io/default-container: prometheus
Status:               Pending
IP:
IPs:                  <none>
Controlled By:        StatefulSet/prometheus-k8s
Init Containers:
init-config-reloader:
    Image:      nutanix/prometheus-config-reloader:v0.57.0
    Port:       8080/TCP
    Host Port: 0/TCP
    Command:
      /bin/prometheus-config-reloader
    Args:
      --watch-interval=0
      --listen-address=:8080
      --config-file=/etc/prometheus/config/prometheus.yaml.gz
      --config-envsubst-file=/etc/prometheus/config_out/prometheus.env.yaml
      --watched-dir=/etc/prometheus/rules/prometheus-k8s-rulefiles-0
    Limits:
      cpu:     100m
      memory: 50Mi
    Requests:
      cpu:     100m
      memory: 50Mi
    Environment:
      POD_NAME: prometheus-k8s-0 (v1:metadata.name)
      SHARD:     0
    Mounts:
      /etc/prometheus/config from config (rw)
      /etc/prometheus/config_out from config-out (rw)
      /etc/prometheus/rules/prometheus-k8s-rulefiles-0 from prometheus-k8s-rulefiles-0 (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-zwdcp (ro)
Containers:
prometheus:
    Image:      nutanix/prometheus:v2.36.1
    Port:       9090/TCP
    Host Port: 0/TCP
    Args:
      --web.console.templates=/etc/prometheus/consoles
      --web.console.libraries=/etc/prometheus/console_libraries
      --storage.tsdb.retention.time=10d
      --storage.tsdb.retention.size=15GB
      --config.file=/etc/prometheus/config_out/prometheus.env.yaml
      --storage.tsdb.path=/prometheus
      --web.enable-lifecycle
      --web.route-prefix=/
      --web.config.file=/etc/prometheus/web_config/web-config.yaml
    Limits:
      cpu:     500m
      memory: 1536Mi
    Requests:
      cpu:        200m
      memory:     750Mi
    Liveness:     http-get http://:web/-/healthy delay=0s timeout=3s period=5s #success=1 #failure=6
    Readiness:    http-get http://:web/-/ready delay=0s timeout=3s period=5s #success=1 #failure=3
    Startup:      http-get http://:web/-/ready delay=0s timeout=3s period=15s #success=1 #failure=60
    Environment: <none>
    Mounts:
      /etc/prometheus/certs from tls-assets (ro)
      /etc/prometheus/config_out from config-out (ro)
      /etc/prometheus/rules/prometheus-k8s-rulefiles-0 from prometheus-k8s-rulefiles-0 (rw)
      /etc/prometheus/secrets/kube-etcd-client-certs from secret-kube-etcd-client-certs (ro)
      /etc/prometheus/secrets/kubelet-client-certs from secret-kubelet-client-certs (ro)
      /etc/prometheus/secrets/prometheus-certs from secret-prometheus-certs (ro)
      /etc/prometheus/web_config/web-config.yaml from web-config (ro,path="web-config.yaml")
      /prometheus from prometheus-k8s-db (rw,path="prometheus-db")
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-zwdcp (ro)
config-reloader:
    Image:      nutanix/prometheus-config-reloader:v0.57.0
    Port:       8080/TCP
    Host Port: 0/TCP
    Command:
      /bin/prometheus-config-reloader
    Args:
      --listen-address=:8080
      --reload-url=http://127.0.0.1:9090/-/reload
      --config-file=/etc/prometheus/config/prometheus.yaml.gz
      --config-envsubst-file=/etc/prometheus/config_out/prometheus.env.yaml
      --watched-dir=/etc/prometheus/rules/prometheus-k8s-rulefiles-0
    Limits:
      cpu:     100m
      memory: 50Mi
    Requests:
      cpu:     100m
      memory: 50Mi
    Environment:
      POD_NAME: prometheus-k8s-0 (v1:metadata.name)
      SHARD:     0
    Mounts:
      /etc/prometheus/config from config (rw)
      /etc/prometheus/config_out from config-out (rw)
      /etc/prometheus/rules/prometheus-k8s-rulefiles-0 from prometheus-k8s-rulefiles-0 (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-zwdcp (ro)
Conditions:
Type           Status
PodScheduled   False
Volumes:
prometheus-k8s-db:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName: prometheus-k8s-db-prometheus-k8s-0
    ReadOnly:   false
config:
    Type:        Secret (a volume populated by a Secret)
    SecretName: prometheus-k8s
    Optional:    false
tls-assets:
    Type:                Projected (a volume that contains injected data from multiple sources)
    SecretName:          prometheus-k8s-tls-assets-0
    SecretOptionalName: <nil>
config-out:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit: <unset>
prometheus-k8s-rulefiles-0:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      prometheus-k8s-rulefiles-0
    Optional: false
web-config:
    Type:        Secret (a volume populated by a Secret)
    SecretName: prometheus-k8s-web-config
    Optional:    false
secret-kube-etcd-client-certs:
    Type:        Secret (a volume populated by a Secret)
    SecretName: kube-etcd-client-certs
    Optional:    false
secret-kubelet-client-certs:
    Type:        Secret (a volume populated by a Secret)
    SecretName: kubelet-client-certs
    Optional:    false
secret-prometheus-certs:
    Type:        Secret (a volume populated by a Secret)
    SecretName: prometheus-certs
    Optional:    false
kube-api-access-zwdcp:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds: 3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              kubernetes.io/os=linux
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 30s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 30s
Events:
Type     Reason            Age                   From               Message
----     ------            ----                  ----               -------
Warning FailedScheduling 17m (x55 over 4h43m) default-scheduler 0/1 nodes are available: pod has unbound immediate PersistentVolumeClaims. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling..

Like

T

+1

Tushar_Tarkas
Author
Trailblazer
Forum|Forum|13 days ago
February 2, 2026

nutanix@NTNX-172-30-30-15-A-PCVM:~$  kubectl describe pod prometheus-k8s-0  -n ntnx-system
Name:                 prometheus-k8s-0
Namespace:            ntnx-system
Priority:             200000000
Priority Class Name:  ntnx-system-cluster-critical
Service Account:      prometheus-k8s
Node:                 <none>
Labels:               app.kubernetes.io/component=prometheus
                      app.kubernetes.io/instance=k8s
                      app.kubernetes.io/managed-by=prometheus-operator
                      app.kubernetes.io/name=prometheus
                      app.kubernetes.io/part-of=kube-prometheus
                      app.kubernetes.io/version=2.36.1
                      apps.kubernetes.io/pod-index=0
                      controller-revision-hash=prometheus-k8s-7669bbdcf
                      operator.prometheus.io/name=k8s
                      operator.prometheus.io/shard=0
                      prometheus=k8s
                      statefulset.kubernetes.io/pod-name=prometheus-k8s-0
Annotations:          kubectl.kubernetes.io/default-container: prometheus
Status:               Pending
IP:
IPs:                  <none>
Controlled By:        StatefulSet/prometheus-k8s
Init Containers:
  init-config-reloader:
    Image:      nutanix/prometheus-config-reloader:v0.57.0
    Port:       8080/TCP
    Host Port:  0/TCP
    Command:
      /bin/prometheus-config-reloader
    Args:
      --watch-interval=0
      --listen-address=:8080
      --config-file=/etc/prometheus/config/prometheus.yaml.gz
      --config-envsubst-file=/etc/prometheus/config_out/prometheus.env.yaml
      --watched-dir=/etc/prometheus/rules/prometheus-k8s-rulefiles-0
    Limits:
      cpu:     100m
      memory:  50Mi
    Requests:
      cpu:     100m
      memory:  50Mi
    Environment:
      POD_NAME:  prometheus-k8s-0 (v1:metadata.name)
      SHARD:     0
    Mounts:
      /etc/prometheus/config from config (rw)
      /etc/prometheus/config_out from config-out (rw)
      /etc/prometheus/rules/prometheus-k8s-rulefiles-0 from prometheus-k8s-rulefiles-0 (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-zwdcp (ro)
Containers:
  prometheus:
    Image:      nutanix/prometheus:v2.36.1
    Port:       9090/TCP
    Host Port:  0/TCP
    Args:
      --web.console.templates=/etc/prometheus/consoles
      --web.console.libraries=/etc/prometheus/console_libraries
      --storage.tsdb.retention.time=10d
      --storage.tsdb.retention.size=15GB
      --config.file=/etc/prometheus/config_out/prometheus.env.yaml
      --storage.tsdb.path=/prometheus
      --web.enable-lifecycle
      --web.route-prefix=/
      --web.config.file=/etc/prometheus/web_config/web-config.yaml
    Limits:
      cpu:     500m
      memory:  1536Mi
    Requests:
      cpu:        200m
      memory:     750Mi
    Liveness:     http-get http://:web/-/healthy delay=0s timeout=3s period=5s #success=1 #failure=6
    Readiness:    http-get http://:web/-/ready delay=0s timeout=3s period=5s #success=1 #failure=3
    Startup:      http-get http://:web/-/ready delay=0s timeout=3s period=15s #success=1 #failure=60
    Environment:  <none>
    Mounts:
      /etc/prometheus/certs from tls-assets (ro)
      /etc/prometheus/config_out from config-out (ro)
      /etc/prometheus/rules/prometheus-k8s-rulefiles-0 from prometheus-k8s-rulefiles-0 (rw)
      /etc/prometheus/secrets/kube-etcd-client-certs from secret-kube-etcd-client-certs (ro)
      /etc/prometheus/secrets/kubelet-client-certs from secret-kubelet-client-certs (ro)
      /etc/prometheus/secrets/prometheus-certs from secret-prometheus-certs (ro)
      /etc/prometheus/web_config/web-config.yaml from web-config (ro,path="web-config.yaml")
      /prometheus from prometheus-k8s-db (rw,path="prometheus-db")
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-zwdcp (ro)
  config-reloader:
    Image:      nutanix/prometheus-config-reloader:v0.57.0
    Port:       8080/TCP
    Host Port:  0/TCP
    Command:
      /bin/prometheus-config-reloader
    Args:
      --listen-address=:8080
      --reload-url=http://127.0.0.1:9090/-/reload
      --config-file=/etc/prometheus/config/prometheus.yaml.gz
      --config-envsubst-file=/etc/prometheus/config_out/prometheus.env.yaml
      --watched-dir=/etc/prometheus/rules/prometheus-k8s-rulefiles-0
    Limits:
      cpu:     100m
      memory:  50Mi
    Requests:
      cpu:     100m
      memory:  50Mi
    Environment:
      POD_NAME:  prometheus-k8s-0 (v1:metadata.name)
      SHARD:     0
    Mounts:
      /etc/prometheus/config from config (rw)
      /etc/prometheus/config_out from config-out (rw)
      /etc/prometheus/rules/prometheus-k8s-rulefiles-0 from prometheus-k8s-rulefiles-0 (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-zwdcp (ro)
Conditions:
  Type           Status
  PodScheduled   False
Volumes:
  prometheus-k8s-db:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  prometheus-k8s-db-prometheus-k8s-0
    ReadOnly:   false
  config:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  prometheus-k8s
    Optional:    false
  tls-assets:
    Type:                Projected (a volume that contains injected data from multiple sources)
    SecretName:          prometheus-k8s-tls-assets-0
    SecretOptionalName:  <nil>
  config-out:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  prometheus-k8s-rulefiles-0:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      prometheus-k8s-rulefiles-0
    Optional:  false
  web-config:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  prometheus-k8s-web-config
    Optional:    false
  secret-kube-etcd-client-certs:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  kube-etcd-client-certs
    Optional:    false
  secret-kubelet-client-certs:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  kubelet-client-certs
    Optional:    false
  secret-prometheus-certs:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  prometheus-certs
    Optional:    false
  kube-api-access-zwdcp:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              kubernetes.io/os=linux
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 30s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 30s
Events:
  Type     Reason            Age                   From               Message
  ----     ------            ----                  ----               -------
  Warning  FailedScheduling  17m (x55 over 4h43m)  default-scheduler  0/1 nodes are available: pod has unbound immediate PersistentVolumeClaims. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling..

Like

T

+1

Tushar_Tarkas
Author
Trailblazer
Forum|Forum|13 days ago
February 2, 2026

nutanix@NTNX-172-30-30-15-A-PCVM:~$  kubectl describe pod prometheus-k8s-0  -n ntnx-system
Name:                 prometheus-k8s-0
Namespace:            ntnx-system
Priority:             200000000
Priority Class Name:  ntnx-system-cluster-critical
Service Account:      prometheus-k8s
Node:                 <none>
Labels:               app.kubernetes.io/component=prometheus
                      app.kubernetes.io/instance=k8s
                      app.kubernetes.io/managed-by=prometheus-operator
                      app.kubernetes.io/name=prometheus
                      app.kubernetes.io/part-of=kube-prometheus
                      app.kubernetes.io/version=2.36.1
                      apps.kubernetes.io/pod-index=0
                      controller-revision-hash=prometheus-k8s-7669bbdcf
                      operator.prometheus.io/name=k8s
                      operator.prometheus.io/shard=0
                      prometheus=k8s
                      statefulset.kubernetes.io/pod-name=prometheus-k8s-0
Annotations:          kubectl.kubernetes.io/default-container: prometheus
Status:               Pending
IP:
IPs:                  <none>
Controlled By:        StatefulSet/prometheus-k8s
Init Containers:
  init-config-reloader:
    Image:      nutanix/prometheus-config-reloader:v0.57.0
    Port:       8080/TCP
    Host Port:  0/TCP
    Command:
      /bin/prometheus-config-reloader
    Args:
      --watch-interval=0
      --listen-address=:8080
      --config-file=/etc/prometheus/config/prometheus.yaml.gz
      --config-envsubst-file=/etc/prometheus/config_out/prometheus.env.yaml
      --watched-dir=/etc/prometheus/rules/prometheus-k8s-rulefiles-0
    Limits:
      cpu:     100m
      memory:  50Mi
    Requests:
      cpu:     100m
      memory:  50Mi
    Environment:
      POD_NAME:  prometheus-k8s-0 (v1:metadata.name)
      SHARD:     0
    Mounts:
      /etc/prometheus/config from config (rw)
      /etc/prometheus/config_out from config-out (rw)
      /etc/prometheus/rules/prometheus-k8s-rulefiles-0 from prometheus-k8s-rulefiles-0 (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-zwdcp (ro)
Containers:
  prometheus:
    Image:      nutanix/prometheus:v2.36.1
    Port:       9090/TCP
    Host Port:  0/TCP
    Args:
      --web.console.templates=/etc/prometheus/consoles
      --web.console.libraries=/etc/prometheus/console_libraries
      --storage.tsdb.retention.time=10d
      --storage.tsdb.retention.size=15GB
      --config.file=/etc/prometheus/config_out/prometheus.env.yaml
      --storage.tsdb.path=/prometheus
      --web.enable-lifecycle
      --web.route-prefix=/
      --web.config.file=/etc/prometheus/web_config/web-config.yaml
    Limits:
      cpu:     500m
      memory:  1536Mi
    Requests:
      cpu:        200m
      memory:     750Mi
    Liveness:     http-get http://:web/-/healthy delay=0s timeout=3s period=5s #success=1 #failure=6
    Readiness:    http-get http://:web/-/ready delay=0s timeout=3s period=5s #success=1 #failure=3
    Startup:      http-get http://:web/-/ready delay=0s timeout=3s period=15s #success=1 #failure=60
    Environment:  <none>
    Mounts:
      /etc/prometheus/certs from tls-assets (ro)
      /etc/prometheus/config_out from config-out (ro)
      /etc/prometheus/rules/prometheus-k8s-rulefiles-0 from prometheus-k8s-rulefiles-0 (rw)
      /etc/prometheus/secrets/kube-etcd-client-certs from secret-kube-etcd-client-certs (ro)
      /etc/prometheus/secrets/kubelet-client-certs from secret-kubelet-client-certs (ro)
      /etc/prometheus/secrets/prometheus-certs from secret-prometheus-certs (ro)
      /etc/prometheus/web_config/web-config.yaml from web-config (ro,path="web-config.yaml")
      /prometheus from prometheus-k8s-db (rw,path="prometheus-db")
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-zwdcp (ro)
  config-reloader:
    Image:      nutanix/prometheus-config-reloader:v0.57.0
    Port:       8080/TCP
    Host Port:  0/TCP
    Command:
      /bin/prometheus-config-reloader
    Args:
      --listen-address=:8080
      --reload-url=http://127.0.0.1:9090/-/reload
      --config-file=/etc/prometheus/config/prometheus.yaml.gz
      --config-envsubst-file=/etc/prometheus/config_out/prometheus.env.yaml
      --watched-dir=/etc/prometheus/rules/prometheus-k8s-rulefiles-0
    Limits:
      cpu:     100m
      memory:  50Mi
    Requests:
      cpu:     100m
      memory:  50Mi
    Environment:
      POD_NAME:  prometheus-k8s-0 (v1:metadata.name)
      SHARD:     0
    Mounts:
      /etc/prometheus/config from config (rw)
      /etc/prometheus/config_out from config-out (rw)
      /etc/prometheus/rules/prometheus-k8s-rulefiles-0 from prometheus-k8s-rulefiles-0 (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-zwdcp (ro)
Conditions:
  Type           Status
  PodScheduled   False
Volumes:
  prometheus-k8s-db:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  prometheus-k8s-db-prometheus-k8s-0
    ReadOnly:   false
  config:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  prometheus-k8s
    Optional:    false
  tls-assets:
    Type:                Projected (a volume that contains injected data from multiple sources)
    SecretName:          prometheus-k8s-tls-assets-0
    SecretOptionalName:  <nil>
  config-out:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  prometheus-k8s-rulefiles-0:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      prometheus-k8s-rulefiles-0
    Optional:  false
  web-config:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  prometheus-k8s-web-config
    Optional:    false
  secret-kube-etcd-client-certs:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  kube-etcd-client-certs
    Optional:    false
  secret-kubelet-client-certs:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  kubelet-client-certs
    Optional:    false
  secret-prometheus-certs:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  prometheus-certs
    Optional:    false
  kube-api-access-zwdcp:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              kubernetes.io/os=linux
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 30s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 30s
Events:
  Type     Reason            Age                   From               Message
  ----     ------            ----                  ----               -------
  Warning  FailedScheduling  17m (x55 over 4h43m)  default-scheduler  0/1 nodes are available: pod has unbound immediate PersistentVolumeClaims. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling..

Final output

Like

T

+1

Tushar_Tarkas
Author
Trailblazer
Forum|Forum|13 days ago
February 2, 2026

nutanix@NTNX-172-30-30-15-A-PCVM:~$  kubectl describe pod prometheus-k8s-0  -n ntnx-system
Name:                 prometheus-k8s-0
Namespace:            ntnx-system
Priority:             200000000
Priority Class Name:  ntnx-system-cluster-critical
Service Account:      prometheus-k8s
Node:                 <none>
Labels:               app.kubernetes.io/component=prometheus
                      app.kubernetes.io/instance=k8s
                      app.kubernetes.io/managed-by=prometheus-operator
                      app.kubernetes.io/name=prometheus
                      app.kubernetes.io/part-of=kube-prometheus
                      app.kubernetes.io/version=2.36.1
                      apps.kubernetes.io/pod-index=0
                      controller-revision-hash=prometheus-k8s-7669bbdcf
                      operator.prometheus.io/name=k8s
                      operator.prometheus.io/shard=0
                      prometheus=k8s
                      statefulset.kubernetes.io/pod-name=prometheus-k8s-0
Annotations:          kubectl.kubernetes.io/default-container: prometheus
Status:               Pending
IP:
IPs:                  <none>
Controlled By:        StatefulSet/prometheus-k8s
Init Containers:
  init-config-reloader:
    Image:      nutanix/prometheus-config-reloader:v0.57.0
    Port:       8080/TCP
    Host Port:  0/TCP
    Command:
      /bin/prometheus-config-reloader
    Args:
      --watch-interval=0
      --listen-address=:8080
      --config-file=/etc/prometheus/config/prometheus.yaml.gz
      --config-envsubst-file=/etc/prometheus/config_out/prometheus.env.yaml
      --watched-dir=/etc/prometheus/rules/prometheus-k8s-rulefiles-0
    Limits:
      cpu:     100m
      memory:  50Mi
    Requests:
      cpu:     100m
      memory:  50Mi
    Environment:
      POD_NAME:  prometheus-k8s-0 (v1:metadata.name)
      SHARD:     0
    Mounts:
      /etc/prometheus/config from config (rw)
      /etc/prometheus/config_out from config-out (rw)
      /etc/prometheus/rules/prometheus-k8s-rulefiles-0 from prometheus-k8s-rulefiles-0 (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-zwdcp (ro)
Containers:
  prometheus:
    Image:      nutanix/prometheus:v2.36.1
    Port:       9090/TCP
    Host Port:  0/TCP
    Args:
      --web.console.templates=/etc/prometheus/consoles
      --web.console.libraries=/etc/prometheus/console_libraries
      --storage.tsdb.retention.time=10d
      --storage.tsdb.retention.size=15GB
      --config.file=/etc/prometheus/config_out/prometheus.env.yaml
      --storage.tsdb.path=/prometheus
      --web.enable-lifecycle
      --web.route-prefix=/
      --web.config.file=/etc/prometheus/web_config/web-config.yaml
    Limits:
      cpu:     500m
      memory:  1536Mi
    Requests:
      cpu:        200m
      memory:     750Mi
    Liveness:     http-get http://:web/-/healthy delay=0s timeout=3s period=5s #success=1 #failure=6
    Readiness:    http-get http://:web/-/ready delay=0s timeout=3s period=5s #success=1 #failure=3
    Startup:      http-get http://:web/-/ready delay=0s timeout=3s period=15s #success=1 #failure=60
    Environment:  <none>
    Mounts:
      /etc/prometheus/certs from tls-assets (ro)
      /etc/prometheus/config_out from config-out (ro)
      /etc/prometheus/rules/prometheus-k8s-rulefiles-0 from prometheus-k8s-rulefiles-0 (rw)
      /etc/prometheus/secrets/kube-etcd-client-certs from secret-kube-etcd-client-certs (ro)
      /etc/prometheus/secrets/kubelet-client-certs from secret-kubelet-client-certs (ro)
      /etc/prometheus/secrets/prometheus-certs from secret-prometheus-certs (ro)
      /etc/prometheus/web_config/web-config.yaml from web-config (ro,path="web-config.yaml")
      /prometheus from prometheus-k8s-db (rw,path="prometheus-db")
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-zwdcp (ro)
  config-reloader:
    Image:      nutanix/prometheus-config-reloader:v0.57.0
    Port:       8080/TCP
    Host Port:  0/TCP
    Command:
      /bin/prometheus-config-reloader
    Args:
      --listen-address=:8080
      --reload-url=http://127.0.0.1:9090/-/reload
      --config-file=/etc/prometheus/config/prometheus.yaml.gz
      --config-envsubst-file=/etc/prometheus/config_out/prometheus.env.yaml
      --watched-dir=/etc/prometheus/rules/prometheus-k8s-rulefiles-0
    Limits:
      cpu:     100m
      memory:  50Mi
    Requests:
      cpu:     100m
      memory:  50Mi
    Environment:
      POD_NAME:  prometheus-k8s-0 (v1:metadata.name)
      SHARD:     0
    Mounts:
      /etc/prometheus/config from config (rw)
      /etc/prometheus/config_out from config-out (rw)
      /etc/prometheus/rules/prometheus-k8s-rulefiles-0 from prometheus-k8s-rulefiles-0 (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-zwdcp (ro)
Conditions:
  Type           Status
  PodScheduled   False
Volumes:
  prometheus-k8s-db:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  prometheus-k8s-db-prometheus-k8s-0
    ReadOnly:   false
  config:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  prometheus-k8s
    Optional:    false
  tls-assets:
    Type:                Projected (a volume that contains injected data from multiple sources)
    SecretName:          prometheus-k8s-tls-assets-0
    SecretOptionalName:  <nil>
  config-out:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  prometheus-k8s-rulefiles-0:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      prometheus-k8s-rulefiles-0
    Optional:  false
  web-config:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  prometheus-k8s-web-config
    Optional:    false
  secret-kube-etcd-client-certs:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  kube-etcd-client-certs
    Optional:    false
  secret-kubelet-client-certs:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  kubelet-client-certs
    Optional:    false
  secret-prometheus-certs:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  prometheus-certs
    Optional:    false
  kube-api-access-zwdcp:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              kubernetes.io/os=linux
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 30s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 30s
Events:
  Type     Reason            Age                   From               Message
  ----     ------            ----                  ----               -------
  Warning  FailedScheduling  17m (x55 over 4h43m)  default-scheduler  0/1 nodes are available: pod has unbound immediate PersistentVolumeClaims. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling..

Like

T

+1

Tushar_Tarkas
Author
Trailblazer
Forum|Forum|13 days ago
February 2, 2026

nutanix@NTNX-172-30-30-15-A-PCVM:~$  kubectl describe pod prometheus-k8s-0  -n ntnx-system
Name:                 prometheus-k8s-0
Namespace:            ntnx-system
Priority:             200000000
Priority Class Name:  ntnx-system-cluster-critical
Service Account:      prometheus-k8s
Node:                 <none>
Labels:               app.kubernetes.io/component=prometheus
                      app.kubernetes.io/instance=k8s
                      app.kubernetes.io/managed-by=prometheus-operator
                      app.kubernetes.io/name=prometheus
                      app.kubernetes.io/part-of=kube-prometheus
                      app.kubernetes.io/version=2.36.1
                      apps.kubernetes.io/pod-index=0
                      controller-revision-hash=prometheus-k8s-7669bbdcf
                      operator.prometheus.io/name=k8s
                      operator.prometheus.io/shard=0
                      prometheus=k8s
                      statefulset.kubernetes.io/pod-name=prometheus-k8s-0
Annotations:          kubectl.kubernetes.io/default-container: prometheus
Status:               Pending
IP:
IPs:                  <none>
Controlled By:        StatefulSet/prometheus-k8s
Init Containers:
  init-config-reloader:
    Image:      nutanix/prometheus-config-reloader:v0.57.0
    Port:       8080/TCP
    Host Port:  0/TCP
    Command:
      /bin/prometheus-config-reloader
    Args:
      --watch-interval=0
      --listen-address=:8080
      --config-file=/etc/prometheus/config/prometheus.yaml.gz
      --config-envsubst-file=/etc/prometheus/config_out/prometheus.env.yaml
      --watched-dir=/etc/prometheus/rules/prometheus-k8s-rulefiles-0
    Limits:
      cpu:     100m
      memory:  50Mi
    Requests:
      cpu:     100m
      memory:  50Mi
    Environment:
      POD_NAME:  prometheus-k8s-0 (v1:metadata.name)
      SHARD:     0
    Mounts:
      /etc/prometheus/config from config (rw)
      /etc/prometheus/config_out from config-out (rw)
      /etc/prometheus/rules/prometheus-k8s-rulefiles-0 from prometheus-k8s-rulefiles-0 (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-zwdcp (ro)
Containers:
  prometheus:
    Image:      nutanix/prometheus:v2.36.1
    Port:       9090/TCP
    Host Port:  0/TCP
    Args:
      --web.console.templates=/etc/prometheus/consoles
      --web.console.libraries=/etc/prometheus/console_libraries
      --storage.tsdb.retention.time=10d
      --storage.tsdb.retention.size=15GB
      --config.file=/etc/prometheus/config_out/prometheus.env.yaml
      --storage.tsdb.path=/prometheus
      --web.enable-lifecycle
      --web.route-prefix=/
      --web.config.file=/etc/prometheus/web_config/web-config.yaml
    Limits:
      cpu:     500m
      memory:  1536Mi
    Requests:
      cpu:        200m
      memory:     750Mi
    Liveness:     http-get http://:web/-/healthy delay=0s timeout=3s period=5s #success=1 #failure=6
    Readiness:    http-get http://:web/-/ready delay=0s timeout=3s period=5s #success=1 #failure=3
    Startup:      http-get http://:web/-/ready delay=0s timeout=3s period=15s #success=1 #failure=60
    Environment:  <none>
    Mounts:
      /etc/prometheus/certs from tls-assets (ro)
      /etc/prometheus/config_out from config-out (ro)
      /etc/prometheus/rules/prometheus-k8s-rulefiles-0 from prometheus-k8s-rulefiles-0 (rw)
      /etc/prometheus/secrets/kube-etcd-client-certs from secret-kube-etcd-client-certs (ro)
      /etc/prometheus/secrets/kubelet-client-certs from secret-kubelet-client-certs (ro)
      /etc/prometheus/secrets/prometheus-certs from secret-prometheus-certs (ro)
      /etc/prometheus/web_config/web-config.yaml from web-config (ro,path="web-config.yaml")
      /prometheus from prometheus-k8s-db (rw,path="prometheus-db")
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-zwdcp (ro)
  config-reloader:
    Image:      nutanix/prometheus-config-reloader:v0.57.0
    Port:       8080/TCP
    Host Port:  0/TCP
    Command:
      /bin/prometheus-config-reloader
    Args:
      --listen-address=:8080
      --reload-url=http://127.0.0.1:9090/-/reload
      --config-file=/etc/prometheus/config/prometheus.yaml.gz
      --config-envsubst-file=/etc/prometheus/config_out/prometheus.env.yaml
      --watched-dir=/etc/prometheus/rules/prometheus-k8s-rulefiles-0
    Limits:
      cpu:     100m
      memory:  50Mi
    Requests:
      cpu:     100m
      memory:  50Mi
    Environment:
      POD_NAME:  prometheus-k8s-0 (v1:metadata.name)
      SHARD:     0
    Mounts:
      /etc/prometheus/config from config (rw)
      /etc/prometheus/config_out from config-out (rw)
      /etc/prometheus/rules/prometheus-k8s-rulefiles-0 from prometheus-k8s-rulefiles-0 (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-zwdcp (ro)
Conditions:
  Type           Status
  PodScheduled   False
Volumes:
  prometheus-k8s-db:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  prometheus-k8s-db-prometheus-k8s-0
    ReadOnly:   false
  config:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  prometheus-k8s
    Optional:    false
  tls-assets:
    Type:                Projected (a volume that contains injected data from multiple sources)
    SecretName:          prometheus-k8s-tls-assets-0
    SecretOptionalName:  <nil>
  config-out:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  prometheus-k8s-rulefiles-0:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      prometheus-k8s-rulefiles-0
    Optional:  false
  web-config:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  prometheus-k8s-web-config
    Optional:    false
  secret-kube-etcd-client-certs:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  kube-etcd-client-certs
    Optional:    false
  secret-kubelet-client-certs:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  kubelet-client-certs
    Optional:    false
  secret-prometheus-certs:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  prometheus-certs
    Optional:    false
  kube-api-access-zwdcp:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              kubernetes.io/os=linux
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 30s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 30s
Events:
  Type     Reason            Age                   From               Message
  ----     ------            ----                  ----               -------
  Warning  FailedScheduling  17m (x55 over 4h43m)  default-scheduler  0/1 nodes are available: pod has unbound immediate PersistentVolumeClaims. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling.

Like

Sign up

Login to the community

Scanning file for viruses.

This file cannot be downloaded