Kubernetes cluster not listed in Karbon management GUI

Sometimes Kubernetes clusters are not listed when we try to open Karbon console from Karbon management GUI on Prism Central. This is due to Karbon (karbon_core & karbon_ui) containers being in an unhealthy state. This can be due to various reasons, one of them being docker containers are unable to get the proper resources or docker is unable to establish a connection with Karbon containers. Below is a screenshot of what one would notice when one would go to the Karbon console and no Kubernetes clusters are listed.

Symptoms are container is unhealthy and here is the simple fix to try as a first step:

 The Karbon container karbon_core is unhealthy.

 The Karbon container karbon_core is unhealthy.
nutanix@NTNX-10-X-X-X-A-PCVM:~$ docker ps
	CONTAINER ID        IMAGE                COMMAND             CREATED             STATUS                    PORTS               NAMES
	3cf0cb9a9dee        karbon-core:v1.0.1   "/start.sh"         5 months ago        Up 5 months (unhealthy)                       karbon-core
	912c005d1585        karbon-ui:v1.0.1     "/start.sh"         7 months ago        Up 7 months                                  karbon-ui
	nutanix@NTNX-10-10-100-225-A-PCVM:~$

Below are some of the things you need to keep in mind while executing the action plan:

1. Calm - Ensure no activity is happening in Calm at this stage.
2. Karbon - During the docker restart of Karbon, you would not be able to authenticate to the API server. Hence kubectl will fail till the karbon_core service comes up.
3. Nutanix Objects - Ensure no upgrades or object store deployments are in progress.
4. Nutanix Flow - It will not be possible to create, update, or delete security policies and categories. It will not be possible to assign or remove VM to/from Categories.

Solution:

1. Stop Karbon services, karbon_core and karbon_ui:

nutanix@PCVM:~ genesis stop karbon_core
nutanix@PCVM:~ genesis stop karbon_ui

2. Restart the docker daemon:

nutanix@PCVM$ sudo systemctl stop docker-latest
nutanix@PCVM$ sudo systemctl start docker-latest

3. Once docker daemon is restarted, start the Karbon services, karbon_core and karbon_ui:

nutanix@PCVM$ cluster start 

4. Check the status of Karbon containers. They should be in a healthy state now: 

nutanix@PCVM$ docker ps

CONTAINER ID        IMAGE                COMMAND             CREATED              STATUS                        PORTS               NAMES
319fad4e6a0e        karbon-core:v1.0.1   "/start.sh"         About a minute ago   Up About a minute (healthy)                       karbon-core
8265a6289f0c        karbon-ui:v1.0.1     "/start.sh"         About a minute ago   Up About a minute                                 karbon-ui