+2Hi,
It’s possible to manually adjust the file /var/nutanix/etc/kubernetes/manifests/kube-apiserver.yaml and apply this update into the kubernetes cluster?
I tried to adjust and after run:
sudo systemctl daemon-reload && sudo systemctl restart kubelet-master
But when I describe the kube-api pod I see that the adjusts are not applied.
Anibal
Best answer by Anibal Ulisses
Hello
Basically are needed to adjust 2 files.
First file located at /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/kubernetes/helm.rb
Original file:
# frozen_string_literal: true
module Gitlab
module Kubernetes
module Helm
HELM_VERSION = '2.16.6'
KUBECTL_VERSION = '1.13.12'
NAMESPACE = 'gitlab-managed-apps'
NAMESPACE_LABELS = { 'app.gitlab.com/managed_by' => :gitlab }.freeze
SERVICE_ACCOUNT = 'tiller'
CLUSTER_ROLE_BINDING = 'tiller-admin'
CLUSTER_ROLE = 'cluster-admin'
end
end
end
Adjusted file:
# frozen_string_literal: true
module Gitlab
module Kubernetes
module Helm
HELM_VERSION = '2.16.6'
KUBECTL_VERSION = '1.13.12'
HTTP_PROXY= 'http://<proxy-ip>:3128'
HTTPS_PROXY= 'http://<proxy-ip>:3128'
NO_PROXY= "172.19.0.0/16, 172.20.0.0/16, <my-network-ip-range>/16, localhost, 127.0.0.1"
NAMESPACE = 'gitlab-managed-apps'
NAMESPACE_LABELS = { 'app.gitlab.com/managed_by' => :gitlab }.freeze
SERVICE_ACCOUNT = 'tiller'
CLUSTER_ROLE_BINDING = 'tiller-admin'
CLUSTER_ROLE = 'cluster-admin'
end
end
end
for the NO_PROXY, the 1st and 2nd ip-range are the internal k8s network range, it’s really important to keep the space after the comma, without this space it’s doesn’t work. I worked hard to identify this space.
Second file located at /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/kubernetes/helm/pod.rd
Original part of the file that need to be adjusted:
def generate_pod_env(command)
{
HELM_VERSION: Gitlab::Kubernetes::Helm::HELM_VERSION,
TILLER_NAMESPACE: namespace_name,
COMMAND_SCRIPT: command.generate_script
}.map { |key, value| { name: key, value: value } }
end
Adjusted part of the file:
def generate_pod_env(command)
{
HELM_VERSION: Gitlab::Kubernetes::Helm::HELM_VERSION,
HTTP_PROXY: Gitlab::Kubernetes::Helm::HTTP_PROXY,
HTTPS_PROXY: Gitlab::Kubernetes::Helm::HTTPS_PROXY,
NO_PROXY: Gitlab::Kubernetes::Helm::NO_PROXY,
TILLER_NAMESPACE: namespace_name,
COMMAND_SCRIPT: command.generate_script
}.map { |key, value| { name: key, value: value } }
end
With this adjustments I can deploy the helm tiller and all other applications with success.
Now I’m testing the deploy.
Anibal
+4Hello
Karbon deploys “Managed” Kubernetes environments. What we mean by that is those environments are setup by Nutanix using our best practices and supported by us. The tradeoff is that no manual customization is possible unless instructed by Nutanix Support.
If you are looking for a customizable Kubernets solution that you intend to maintain on your own, then I would suggest getting in touch with your Nutanix account team and looking into Nutanix Calm, which by default has a Kubernetes blueprint that you can customize.
You’ll be responsible for supporting your Kubernetes cluster after that, but it gives you a fast start that is easily customizable to your needs.
Best regards,
+1Hi
Modifying /var/nutanix/etc/kubernetes/manifests/kube-apiserver.yaml file is not supported. These files are normally overwritten during k8s upgrade or host upgrade.
+2+4Where are you pulling your pod/containers images from? Internet or an internal repo?
If you’re pulling them from the Internet, then I don’t see the need for a proxy, so I’m assuming it is from an internal repo?
If so, then you could “build” a container image internally that would “bake in” those settings (assuming env variables like “http_proxy” and “https_proxy”?) then use those to deploy, so that you do not need a PodPreset?
We can look at enabling PodPreset for a future version of Karbon. Could you open a Request For Enhancement (RFE) ticket on your support portal so that we can track this appropriately?
Best regards,
+2If you take a look I already post some questions about it.
Ok, I can open a RFE, but I need to fix the integration between GitLab and Karbon, over proxy, in 2 weeks, I’m not sure if a RFE will be solved in this time.. Without any way I really need to think about to move out from Karbon, and I really didn’t want to do it...
+42 weeks isn’t enough time to implement a new feature, test it properly and release it, while we have other development in flight, for sure.
We can however look at it for futur releases.
I haven’t deployed GitLab intergation with Kubernetes in a while, but I’m pretty sure some of our other customers have, so there ought to be a way to do that without a PodPreset, I’m just not exactly sure how.
Are you using the OpenSource version of GitLab or do you have support from them?
If you have support from them you could reach out and explore other options with them.
+2I really didn’t understand why on this case https://next.nutanix.com/karbon-kubernetes-service-30/istio-service-mesh-on-nutanix-karbon-32639 are suggested to adjust the kube-api, restart it and works. and If I try didn’t work for me.
unfortunately I think that I need to move away from Karbon and If I start with another solution will be hard to move back.
+5Don’t know if this would work, but you could use in your GitLab-Gitaly chart the parameter extraEnv to pass OS proxy configuration like you would do in a typical Linux VM. Usually this is done in the Dockerfile, but maybe in this case could work too.
extraEnv:
HTTP_PROXY: "http://1.1.1.1:3128"
HTTPS_PROXY: "http://1.1.1.1:3128"
NO_PROXY: "localhost,127.0.0.1,<lan_IPs>"https://docs.gitlab.com/charts/charts/gitlab/gitaly/index.html
+5For additional reference just found this in Gitlab website.
+2Hello
Best Regards
+2hi
nothing worked for me. basically helm tiller installation, from gitlab CI/CD integration, are not getting the proxy information, due this I can’t have a success installation.
+2how did you implement mutatingwebhook ? because it works with any settings you want to implement
you can look an exemple here https://github.com/tuxtof/nodeaffinity-webhook
where i inject nodeAffinity in certain condition
you can mimic the same idea to add additonal proxy env variable to your pod, it is a little bit complex, but if gitlab don’t propose by himself a proper solution or offline approach i dont have any other idea
+2+2I solved the issue with code adjusting at gitlab, by my self, that worked fine.
+2Hello
that’s a very good news, can you say a little bit more how you solve this gitlab problem, always interesting to know
Best regards
+2Hello
Basically are needed to adjust 2 files.
First file located at /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/kubernetes/helm.rb
Original file:
# frozen_string_literal: true
module Gitlab
module Kubernetes
module Helm
HELM_VERSION = '2.16.6'
KUBECTL_VERSION = '1.13.12'
NAMESPACE = 'gitlab-managed-apps'
NAMESPACE_LABELS = { 'app.gitlab.com/managed_by' => :gitlab }.freeze
SERVICE_ACCOUNT = 'tiller'
CLUSTER_ROLE_BINDING = 'tiller-admin'
CLUSTER_ROLE = 'cluster-admin'
end
end
end
Adjusted file:
# frozen_string_literal: true
module Gitlab
module Kubernetes
module Helm
HELM_VERSION = '2.16.6'
KUBECTL_VERSION = '1.13.12'
HTTP_PROXY= 'http://<proxy-ip>:3128'
HTTPS_PROXY= 'http://<proxy-ip>:3128'
NO_PROXY= "172.19.0.0/16, 172.20.0.0/16, <my-network-ip-range>/16, localhost, 127.0.0.1"
NAMESPACE = 'gitlab-managed-apps'
NAMESPACE_LABELS = { 'app.gitlab.com/managed_by' => :gitlab }.freeze
SERVICE_ACCOUNT = 'tiller'
CLUSTER_ROLE_BINDING = 'tiller-admin'
CLUSTER_ROLE = 'cluster-admin'
end
end
end
for the NO_PROXY, the 1st and 2nd ip-range are the internal k8s network range, it’s really important to keep the space after the comma, without this space it’s doesn’t work. I worked hard to identify this space.
Second file located at /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/kubernetes/helm/pod.rd
Original part of the file that need to be adjusted:
def generate_pod_env(command)
{
HELM_VERSION: Gitlab::Kubernetes::Helm::HELM_VERSION,
TILLER_NAMESPACE: namespace_name,
COMMAND_SCRIPT: command.generate_script
}.map { |key, value| { name: key, value: value } }
end
Adjusted part of the file:
def generate_pod_env(command)
{
HELM_VERSION: Gitlab::Kubernetes::Helm::HELM_VERSION,
HTTP_PROXY: Gitlab::Kubernetes::Helm::HTTP_PROXY,
HTTPS_PROXY: Gitlab::Kubernetes::Helm::HTTPS_PROXY,
NO_PROXY: Gitlab::Kubernetes::Helm::NO_PROXY,
TILLER_NAMESPACE: namespace_name,
COMMAND_SCRIPT: command.generate_script
}.map { |key, value| { name: key, value: value } }
end
With this adjustments I can deploy the helm tiller and all other applications with success.
Now I’m testing the deploy.
Anibal
+2ok pure gitlab hacking, pushing env var in each function used by gitlab to deploy things
good job
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.
