You’re likely aware that Kubernetes Pods (and containers in general) are ephemeral, meaning it’s expected that they can die at any time. If you’re new to Kubernetes, this can be quite alarming, however Kubernetes objects like ReplicaSets or Deployments will automatically replace these failed Pods.
This replacement of Pods does present a problem around network connectivity. While each Pod has a unique IP that’s internal to the Kubernetes cluster, a newly generated Pod is not guaranteed to have the same IP. This means that you cannot rely on Pod IPs to access your Kubernetes based application. Instead, Kubernetes has three main ways of providing external access to your application:
- NodePort: A Kubernetes Service which exposes your Pods via a port (generally 30000–32767), which when used in conjunction with a Kubernetes node’s IP (http://nodeIP:port) provides external access. This is the easiest way to provide access to your application, but is not recommended for production workloads.
- LoadBalancer: Another Kubernetes Service that exposes your Pods, which by default is only available via public cloud providers. The exact implementation varies by cloud provider, but this type of Service provides a unique IP address or domain name for your application. This is the simplest production-viable method for exposing your application.
- Ingress: A separate Kubernetes object which provides access to a Kubernetes Service (note that NodePort and LoadBalancer themselves are Services), rather than direct Pod access. Ingress provides advanced features like load balancing, SSL termination and name-based virtual hosting, but relies on a 3rd party implementation of an IngressController in order to work. This is the most feature rich, yet complicated method of exposing your Kubernetes applications.
As mentioned, by default,a LoadBalancer Service is only available on cloud providers, not privately hosted Kubernetes clusters like Nutanix Karbon. However, as we’ll see in this blog post, it’s a very straightforward process to utilize MetalLB to provide LoadBalancer Services for Nutanix Karbon.
Note: while Ingress and IngressController are great methods of exposing your Kubernetes application, they’re outside the scope of this blog.
PrerequisitesThis blog post will be building off my colleague Jose Gomez’s Tillerless Helm blog post, so it’s recommended to run through that prior to this post. In that post, Jose goes through configuring Helm, and installing Consul through a Helm chart. By default, the Consul Helm chart utilizes a NodePort type of Service, which is evident at the end of Jose’s post where he finds a Node IP (10.10.56.162), and utilizes port 31820 to access the Consul application:
In this blog post, we’ll deploy the same Consul application, but instead use a LoadBalancer Service rather than NodePort.
MetalLB InstallationMetalLB can be installed cluster-wide via a single kubectl command (depending on the age of this post, it is recommended to navigate to the MetalLB installation page to ensure there hasn’t been an updated release).
$ kubectl apply -f https://raw.githubusercontent.com/google/metallb/v0.7.3/manifests/metallb.yaml
After running the above command, you should see various components of MetalLB get created. For instance, to view the pods that were created:
$ kubectl -n metallb-system get pods
NAME READY STATUS RESTARTS AGE
controller-7cc9c87cfb-f7rxt 1/1 Running 0 3m28s
speaker-q5x7w 1/1 Running 0 3m28s
speaker-qnbqg 1/1 Running 0 3m28s
speaker-qwtjl 1/1 Running 0 3m28s
Now that installation is complete, it’s time to move on to the configuration step.
MetalLB ConfigurationGenerally speaking, MetalLB can be configured in two ways:
- Layer 2 Configuration - this method relies on the IPs that MetalLB hands out to be in the same L2 domain as the Kubernetes nodes.
- BGP Configuration - this method does not have the above L2 limitation, however it does rely on knowledge about your router’s BGP configuration.
$ kubectl describe nodes | grep Internal
In Prism, I see that the matching network is 10.45.100.0/22:
Karbon Kubernetes Cluster Network
This means that MetalLB in Layer 2 mode will only work if I designate available addresses from the 10.45.100.1 to 10.45.103.254 IP range. In my environment, I set aside address 10.45.100.11 through 10.45.100.15 for MetalLB to use. To configure MetalLB, we’ll need to create and apply a Kubernetes ConfigMap:
$ cat << EOF > metallb-config.yaml
- name: default
- 10.45.100.11-10.45.100.15 #substitute your range here
$ kubectl apply -f metallb-config.yaml
That’s it! After running three simple commands (MetalLB install, ConfigMap create, ConfigMap apply), any LoadBalancer type of Service will automatically be handed an IP from the available range by MetalLB. Let’s see this in action by deploying Consul with a Helm chart.
Consul Deployment with a LoadBalancer Service
As mentioned earlier, we’ll be adding on to this Tillerless Helm blog post. So if you haven’t already, be sure to:
- Install Helm on your workstation
- Initialize in client-only mode
- Install the helm-tiller plugin
- Start helm tiller
$ helm install --name consul --set uiService.type=LoadBalancer,HttpPort=80
The first set key value pair instructs helm to create a LoadBalancer Service instead of the default NodePort, while the second changes the default port 8500 to port 80, so we will not have to append any ports to our IP. In the output of that command, you should see the Service section call out a Service of type LoadBalancer:
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
consul ClusterIP None 80/TCP... 1s
consul-ui LoadBalancer 172.19.62.30 10.45.100.11 80:31263/TCP 1s
You can also view your Service and its corresponding IP with a kubectl command:
$ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
consul ClusterIP None 80/TCP... 117s
consul-ui LoadBalancer 172.19.62.30 10.45.100.11 80:31263/TCP 117s
kubernetes ClusterIP 172.19.0.1 443/TCP 3h40m
To verify our deployment, copy the External-IP of the consul-ui service (in my case 10.45.100.11) and paste it into your web browser. We should see the same Consul UI as was shown in Jose’s blog post:
Accessing Consul via a LoadBalancer
It’s worth noting that MetalLB in layer 2 mode, a particular service IP (10.45.100.11 in our example) is routed to a single Kubernetes worker node. From there, kube-proxy spreads the traffic to all of the underlying pods. While each pod will be utilized, total bandwidth of the Service is limited to the bandwidth of a single worker node. If this limitation is not acceptable, please look into using MetalLB in BGP mode.
ConclusionLoadBalancer Services allow you to expose your Kubernetes based applications to your end users with simple IPs or domain names, rather than relying on long, random port numbers. With just a couple of simple commands you can configure your Karbon Kubernetes clusters to automatically hand out IPs when a LoadBalancer Service is created. This allows for your on-prem Kubernetes clusters to have the same great end user experience of public cloud hosted clusters. Thanks for reading!
Disclaimer: This blog may contain links to external websites that are not part of Nutanix.com. Nutanix does not control these sites and disclaims all responsibility for the content or accuracy of any external site. Our decision to link to an external site should not be considered an endorsement of any content on such site.
️ 2019 Nutanix, Inc. All rights reserved. Nutanix, the Nutanix logo and the other Nutanix products and features mentioned herein are registered trademarks or trademarks of Nutanix, Inc. in the United States and other countries. All other brand names mentioned herein are for identification purposes only and may be the trademarks of their respective holder(s).