High Availability (HA) is crucial for ensuring your Kubernetes cluster remains operational even in the face of failures. In this blog, we'll walk through the steps to configure HA for your Kubernetes cluster.

Prerequisites

1. Basic knowledge of Kubernetes

2. A minimum of three master nodes (for HA)

3. A load balancer node (for distributing traffic to master nodes)

4. Two worker nodes.

Step 1: Set Up the Kubernetes Control Plane

To achieve HA, you need multiple master nodes. Here’s how to configure them:

1. Prepare Your Nodes

For a robust Kubernetes HA setup, you'll need to configure your nodes appropriately. Here's a detailed guide on preparing your infrastructure:

• 3 Master Nodes: These nodes will manage your Kubernetes control plane.

• 2 Worker Nodes: These nodes will run your application workloads.

• 1 Load Balancer Node: This node will handle the load balancing of API requests across your master nodes.

• Operating System: Install a compatible Linux distribution (e.g., Ubuntu 22.04).

• Disk Size: Allocate a minimum of 20 GB for the root volume. This ensures you have enough space for Kubernetes and etcd data.

• CPU/RAM: Depending on your workload, a minimum of 2 CPUs and 8 GB of RAM per master node is recommended.

Network Configuration

• Ensure that all master nodes can communicate with each other over the network.

• Open the necessary ports for Kubernetes communication as listed below.

Step 2: Prepare the Load Balancer Node

• Install HA-Proxy:

sudo apt-get update
sudo apt-get install -y haproxy

• Configure HAProxy: Edit the HAProxy configuration file (/etc/haproxy/haproxy.cfg):

sudo nano /etc/haproxy/haproxy.cfg

• Add the following configuration to the bottom of the HAProxy configuration file (/etc/haproxy/haproxy.cfg):

nano /etc/haproxy/haproxy.cfg

frontend kubernetes-frontend
    bind *:6443
    option tcplog
    mode tcp
    default_backend kubernetes-backend

backend kubernetes-backend
    mode tcp
    balance roundrobin
    option tcp-check
    server master1 <MASTER1_IP>:6443 check
    server master2 <MASTER2_IP>:6443 check
    server master3 <MASTER3_IP>:6443 check

• Save the file and restart the HAProxy service**.**

sudo systemctl restart haproxy

Step 3: Prepare All Nodes (Masters and Workers)

• Install Docker, kubeadm, kubelet, and kubectl:

1.     sudo apt-get update
       sudo apt install docker.io -y
       sudo chmod 666 /var/run/docker.sock
       sudo apt-get install -y apt-transport-https ca-certificates curl gnupg
       sudo mkdir -p -m 755 /etc/apt/keyrings
       curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.30/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
       echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.30/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list
       sudo apt update
       sudo apt install -y kubeadm=1.30.0-1.1 kubelet=1.30.0-1.1 kubectl=1.30.0-1.1
   

Step 4: Initialize the First Master Node

• Initialize the first master node: [Replace your HA-Load-Balancer Instance IP]

    sudo kubeadm init --control-plane-endpoint "LOAD_BALANCER_IP:6443" --upload-certs --pod-network-cidr=10.244.0.0/16
  

  

• Set up kubeconfig for the first master node:

    mkdir -p $HOME/.kube
    sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
    sudo chown $(id -u):$(id -g) $HOME/.kube/config
  

• Install Calico network plugin: (Install all Master Nodes)

    kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml
  

• Install Ingress-NGINX Controller: (Install all Master Nodes)

    kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v0.49.0/deploy/static/provider/baremetal/deploy.yaml
  

  • Join more control plane (master) nodes and worker nodes using the output from the "kubeadm init" command mentioned above screenshot.

For Example: #FOR JOINING THE MASTER NODES

    sudo kubeadm join 3.7.66.227:6443 --token r6egit.1ign78wlntd8tqos \
            --discovery-token-ca-cert-hash sha256:c2fa8b33f327f38c61e07e4ebc13498192b714b604700e12d26d592935865fbe \
            --control-plane --certificate-key 248baff77d6376ab51da993e09b6df2ee9dbaa09f264bf181c81b626d5920c70

For Join the Worker Nodes:

    sudo kubeadm join 3.7.66.227:6443 --token r6egit.1ign78wlntd8tqos \
            --discovery-token-ca-cert-hash sha256:c2fa8b33f327f38c61e07e4ebc13498192b714b604700e12d26d592935865fbe

Step 5: Join the Second & third Master Node

1. Get the join command and certificate key from the first master node:

    kubeadm token create --print-join-command --certificate-key $(kubeadm init phase upload-certs --upload-certs | tail -1)

2. Run the join command on the second master node:

    sudo kubeadm join LOAD_BALANCER_IP:6443 --token <token> --discovery-token-ca-cert-hash sha256:<hash> --control-plane --certificate-key <certificate-key>

3. Set up kubeconfig for the second master node:

    mkdir -p $HOME/.kube
    sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
    sudo chown $(id -u):$(id -g) $HOME/.kube/config

Step 6: Join the Worker Nodes

1. Get the join command from the first master node:

kubeadm token create --print-join-command

2. Run the join command on each worker node:

sudo kubeadm join LOAD_BALANCER_IP:6443 --token <token> --discovery-token-ca-cert-hash sha256:<hash>

Refer to the above screenshot for all the commands you need, you can see the output from the "kubeadm init" command.

Step 7: Verify the Cluster

1. Check the status of all nodes:

kubectl get nodes

2. Check the status of all pods:

kubectl get pods --all-namespaces

By following these steps, you'll successfully set up a highly available Kubernetes cluster with two master nodes and three worker nodes.

A load balancer will distribute traffic between the master nodes, ensuring that if one master node fails, the remaining nodes will continue to handle API requests.

This configuration enhances the resilience and availability of your Kubernetes cluster, providing uninterrupted service even in the event of a master node failure.

Verification

Install etcdctl

1. Install etcdctl using apt:

sudo apt-get update
sudo apt-get install -y etcd-client

Verify Etcd Cluster Health

Check the health of the etcd cluster: [Execute all Master Nodes]

sudo ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/peer.crt --key=/etc/kubernetes/pki/etcd/peer.key endpoint health

Check the cluster membership: [Execute all Master Nodes]

sudo ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/peer.crt --key=/etc/kubernetes/pki/etcd/peer.key member list

Verify HAProxy Configuration and Functionality

1. Configure HAProxy Stats:

   • Add the stats configuration to /etc/haproxy/haproxy.cfg:

listen stats
    bind *:8404
    mode http
    stats enable
    stats uri /
    stats refresh 10s
    stats admin if LOCALHOST

Restart HAProxy:

sudo systemctl restart haproxy

Check HAProxy Stats:

• Access the stats page at http://<LOAD_BALANCER_IP>:8404.

Test High Availability

Evaluate Master Node Failover:
Stop the kubelet service and Docker containers on one of the master nodes to simulate a failure:

sudo systemctl stop kubelet
sudo docker stop $(sudo docker ps -q)

OR --> You can terminate the one of the Master Node, for testing.

Verify Cluster Functionality:

kubectl get nodes
kubectl get pods --all-namespaces

• The cluster should still show the remaining nodes as Ready, and the Kubernetes API should be accessible.

You can see, have terminated the master 3 and HA-Proxy statistics report also showing as a Down.

You can deploy a web application on the worker nodes, and it should be accessible from all master nodes.

If one master node goes down, it will not affect your website's performance.

• Conclusion: Your high availability Kubernetes cluster is all set up with multiple master and worker nodes, backed by a load balancer. This setup keeps things running smoothly and resiliently, even if one master node goes down. The remaining nodes will take care of API requests. With this configuration, your cluster is ready for reliable and scalable app management.

  Thanks for following along!