Wednesday, September 30, 2020

CentOS 7 Kubernetes Installation Guide

Kubernetes has been a bit of a buzzword lately. I haven't had much exposure to it, focusing my energy on VMware products over the last few years. Today, we'll cover the Kubernetes install process on my go to Linux distribution of choice, CentOS 7.

Step -1: Why?

I like to practice exercises to get a better understanding. A lot of the documentation surrounding Kubernetes, like most technologies, covers the "what" but not the "why". To add some color to "why containers", we'll solve a problem that I would have for personal use: A Plex media server.

It used to be that Plex had to be installed on Windows or Linux bare metal. Virtualization helped by allowing a VM to be right sized for this purpose, and freed up other resources for other machines. Containers simplified this even more; instead of having to install a VM from scratch or from a template, update the OS, and then install the application, I can deploy Plex with a single command. Kubernetes takes this one step further, allowing for management and orchestration, similar to a command-line only version of vCenter managing ESXi hosts.

Step 0: Prerequisites

For this exercise, I'm going to be using an ESXi 7.0 host with an NFS datastore mounted. Any type of datastore will do, we just need a place to put VMDKs.

We'll be creating 3 virtual machines for this, one master node and two worker nodes. The minimum requirements for each VM is as follows:

    - 2 vCPUs for the master node, 1 for each worker node (official requirements are 1.5 and 0.7)

    - 2GB of memory for the master, 1GB for workers

    - I'm using 100GB of storage each but could probably get away with less

When in the installation environment, be sure to select "Virtualization Host" option. This should install docker automagically.

I'll include the step when the time comes in the guide, but a big callout that should be made is that installing Kubernetes requires disabling SELinux. I initially wasn't comfortable with this, if I have an issue caused by SELinux, journalctl usually gives me the reason why and the steps to fix it; however, kubeadm notes indicate that SELinux must be disabled so that containers can access the filesystem. I don't plan on putting anything else on these VMs, and neither should you.

While this isn't an official requirement, I would strongly recommend configuring NTP. Kubernetes requires the nodes to maintain the same time. NTP is a great way to maintain consistent time between VMs without much manual intervention. 

After installing CentOS 7, run the following commands on all three VMs:

Install ntp and ntpdate (ntp sync)

        yum install ntp ntpdate

Start the service

        systemctl start ntpd

Enable the service

        systemctl enable ntpd

Configure NTP servers

        ntpdate -u -s 0.centos.pool.ntp.org 1.centos.pool.ntp.org 2.centos.pool.ntp.org

Restart the service

        systemctl restart ntpd

Check time, ensure that it's the same on all three VMs

        timedatectl

Set the hardware clock to match current system time

        hwclock  -w


Step 1: Configure Kubernetes Repository

Kubernetes packages are not on the official CentOS 7 repo, so we'll add the repo. Skipping this will cause step 2 to fail.  Feel free to use the text editor of your choice to add the repository. In this tutorial, I'll be using vi. Be sure to carry out the following on all of the nodes.

    vi /etc/yum.repos.d/kubernetes.repo

    [kubernetes]
    name=Kubernetes
    baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
    enabled=1
    gpgcheck=1
    repo_gpgcheck=1
    gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg
    https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg

Save and quit.

Step 2: Install required packages

Install the following packages on each node as a super user:

    yum install -y kubelet kubeadm kubectl

Enable kubelet on each node:

    systemctl enable kubelet

And finally, start kubelet on each node:

    systemctl start kubelet

Step 3: Set hostname

Record the IP addresses of each VM (we'll need this later to update DNS records)

    ip addr

Choose one of the VMs to be the master node, and run the following command:

    hostnamectl set-hostname master-node

Choose which VMs will be worker-node1 and worker-node2. Run this on the first:

    hostnamectl set-hostname worker-node1

And this on the second:

    hostnamectl set-hostname worker-node2

Once complete, be sure to edit /etc/hosts to reflect these changes (using the IPs recorded previously):

    vi /etc/hosts
    192.168.0.10    master-node
    192.168.0.11    worker-node1
    192.168.0.12    worker-node2

Save and quit.

Step 4: Configure firewall-cmd

We'll need to open up some ports to allow the nodes to communicate with one another.

Run the following on the master node:

    firewall-cmd --permanent --add-port=6443/tcp
    firewall-cmd --permanent --add-port=2379-2380/tcp
    firewall-cmd --permanent --add-port=10250/tcp
    firewall-cmd --permanent --add-port=10251/tcp
    firewall-cmd --permanent --add-port=10252/tcp
    firewall-cmd --permanent --add-port=10255/tcp
    firewall-cmd --reload

On each of the worker nodes:

    firewall-cmd --permanent --add-port=6783/tcp
    firewall-cmd --permanent --add-port=10250/tcp
    firewall-cmd --permanent --add-port=10255/tcp
    firewall-cmd --permanent --add-port=30000-32767/tcp
    firewall-cmd --reload

Step 5: Update Kubernetes firewall configuration

Kubernetes uses iptables by default. To ensure that our firewalld rules work, we'll need to modify the sysctl.conf file. Run this command on each node:

    echo 'net.bridge.bridge-nf-call-iptables=1' | sudo tee -a /etc/sysctl.conf

This should persist through reboots. If for some reason you only wish to have it valid for this session, run:

    echo '1' > /proc/sys/net/bridge/bridge-nf-call-iptables

Step 6: Disable SELinux

As mentioned in Step 0 above, we need to disable SELinux (in other words, set it to permissive mode) so containers can access the host filesystem. Run the following on each node:

    setenforce 0
    sed -i ‘s/^SELINUX=enforcing$/SELINUX=permissive/’ /etc/selinux/config

Step 7: Disable swap

We'll need to disable swap on each node:
    
    swapoff -a

Step 8: Deploy the cluster

Use the following command to initialize the cluster:

    kubeadm init

This will take a few minutes, and should generate a token that we'll need to copy for future use, similar to:
    --token i5tx56.q5q6tx37m9j2acea --discovery-token-ca-cert-hash sha256:2a0c84ce92c6185009941730aac1955e7a445d5115b768c5955dab356e645fd5

Before joining the worker nodes, we need to install a pod network, which will allow nodes to communicate. There are several options to choose from, for this we'll use weave.

Create a regular expression that copies your current kubectl version:

    export kubever=$(kubectl version | base64 | tr -d '\n')

Then run this command to apply the weave network:

    kubectl apply -f “https://cloud.weave.works/k8s/net?k8s-version=$kubever"

Once complete, join the worker nodes by using the following command on each (using the token documented above as an example:

    kubeadm join 192.168.0.10:6443 --token i5tx56.q5q6tx37m9j2acea --discovery-token-ca-cert-hash sha256:2a0c84ce92c6185009941730aac1955e7a445d5115b768c5955dab356e645fd5

Once complete, you should be able to check the status of the cluster. Run this on the master node:

    kubectl get nodes

Finally, we need to enable a user to start the cluster. I ran all of the previous commands as root, and used:

    mkdir -p $HOME/.kube
    cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
    chown $(id -u):$(id -g) $HOME/.kube/config

If you're using a sudo enabled user, run:

    mkdir -p $HOME/.kube
    sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
    sudo chown $(id -u):$(id -g) $HOME/.kube/config

And that should do it! In the next blog post, I'll be checking out a project that goes back to the example used previously... can we get a Plex transcode running on a Kubernetes cluster?

Using Intel Optane for NVMe Tiering

 A series of unfortunate events occurred shortly after posting the previous blog post: DIMM H1 decided to fail Replacement was ordered Post ...