Solved

Karbon CSR Request Stuck in Approved State

  • 8 October 2020
  • 9 replies
  • 830 views

Badge

Hello, After a new deployment, I downloaded the kubeconfig and upon trying to approve a Certificate Signing Request (CSR), the csr just sits in an approved state, but never becomes Issued. Is there a restriction or something on the default-kubernetes-<clustername> user that I might not be familiar with? That or maybe my process is wrong, its odd that it just sits in an Approved state. Any help is appreciated!

icon

Best answer by vshuguet 12 October 2020, 14:25

View original

This topic has been closed for comments

9 replies

Userlevel 4
Badge +3

Faced same issue when planned to use certificates as authentication method but didn’t solwed it. 

Now I’m using Prism Central users to authenticate, and k8s RBAC. This is the most easy and flexible way for me. 

Badge

Thanks for the quick reply @mikkisse 

I don’t believe I am familiar with the integration of PC Users and k8s RBAC. I think I saw a blog or post on that while I was searching before I wrote this so let me see if I can go back and find it. Would be nice to understand why we both hit the same issue. Admittedly, I am not an expert with certificates so I don’t know where the issue might lie. 

Userlevel 4
Badge +3

I wrote about t in my blog. It’s in russian, but you can translate it: Connecting to a Kubernetes cluster deployed with Nutanix Karbon. Automating the kubeconfig file request process.

Badge

Thanks, 

I just went through that. While I can hit the Karbon URL and login with my AD username that I mapped in prism central as a User Admin, the kubeconfig file it downloads for the cluster still just says default-user-<clustername> - should that be my username instead? I found a post online where the person had the same issues as we did around certs and they added this:

I got similar problem. When I check with the following command:

kubectl get svc

It seems that the status of the csr is approved, but not issued. Any idea how to fix it?

[Updated] I found the problem. It is because the kube-controller-manager missed these options:

--cluster-signing-cert-file and --cluster-signing-key-file

I could find the kube-apiserver.yaml on the master node which seemed to have the config in there for the controller and those options weren’t in there. However, I am not sure what to point them to as a test since I’m over my head at that point with certificates. Any thoughts to why my username might not be showing in the downloaded kubeconfig file? The username I mapped in PC matches the clusterrolebinding I created in K8S. They are AD users, but the domain has been left off of the username in each configuration so they match.

Thanks! Nicely written blog by the way!

Userlevel 1
Badge +4

Hey @Keith33 ,

Your username is actually encoded in your token, so don’t worry about that part, Karbon/PC will take care of that in the background for you.

As long as you are logged into PC with your AD user, the Kubeconfig you download from there is indeed “your own” and represent your own user. You can check that using (for example) the “whoami” plugin from Krew

√ [shuguet @ mac:~] % ssh pcvm karbonctl cluster kubeconfig --cluster-name shu-prod-calico > ~/.kube/config
√ [shuguet @ mac:~] % kubectl get nodes
NAME STATUS ROLES AGE VERSION
karbon-shu-prod-calico-a35db6-k8s-master-0 Ready master 35d v1.16.13
karbon-shu-prod-calico-a35db6-k8s-master-1 Ready master 35d v1.16.13
karbon-shu-prod-calico-a35db6-k8s-worker-0 Ready node 35d v1.16.13
karbon-shu-prod-calico-a35db6-k8s-worker-1 Ready node 35d v1.16.13
karbon-shu-prod-calico-a35db6-k8s-worker-2 Ready node 35d v1.16.13
√ [shuguet @ mac:~] % kubectl whoami
shu@karbondemo.local

 

Userlevel 4
Badge +3

Hey @vshuguet so certificates in approved and never issued state it’s ok in Karbon deployments? 

Badge

Thanks @vshuguet. That works. 

My initial need was to figure out how we can get CSR’s approved - does Karbon do that?

What I would like to do is setup a certificate-based user access like this: Kubernetes Tips: Give Access To Your Cluster With A Client Certificate

However, I am unable to get past the CSR part since it never issues and thus I can never see a client certificate. My ultimate goal here (and I might just not be that savvy enough with certs) is to allow remote authentication to a Karbon cluster via a certificate.

Userlevel 1
Badge +4

Karbon Clusters (currently, working on it, depends on other parts of the platform too) doesn’t support certificate based authentication. We do support user-based token authentication as explained above (even with directory users, thanks to our integrated auth in Prism Central).

We also support ServiceAccounts if you need longer/persistant token over 24h (for example to integrate with your CI/CD pipeline). In fact we haven’t changed anything related to token based authentication, we just added a “hook” to also enable us to auth users via Prism Central. That way, the “browsing account” credentials (the account used to check on the Directory if the credentials supplied by the user are correct) aren’t anywhere on the Kubernetes clusters, and are instead centrally managed inside Prism Central.

We simply have not enabled Certificate Based authentication (or any kind of certificate issuance) in the Kubernetes clusters, until the platform is able to provide us with a secure source of certificates (as I said, working on it).

Badge

Thanks @vshuguet 

This is the info I was looking to confirm. I can get things working with the service accounts and bearer tokens. It was the certificate-based authentication I was looking to test out, but since it isn’t supported, this would indicate why I am able to approve a request but not see anything issued. 

Thank you both for all the replies. Understanding the PC hook was very valuable here. 

Thanks!