NOMAD Vis Kubernetes Install

By fawzi

November 20, 2018

NOMAD CoE

Kubernetes

Kubernetes is central to NOMAD archive and analytics (see /nomad/Kubernetes).

Every installation of kubernetes has been quite different (kubernetes evolves quickly). Here is the description of the latest (more manual) installation I did on CentOS for the remote visualization, mainly as a reference for me. This uses kubeadm something that has simplified the installation of kubernetes.

update kubeadm

(from https://kubernetes.io/docs/setup/independent/install-kubeadm/)

cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
exclude=kube*
EOF

# Set SELinux in permissive mode (effectively disabling it)
setenforce 0
sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config

yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes

systemctl enable kubelet && systemctl start kubelet

The following changes to sysctl were not needed (already performed)

if ! sysctl --system | grep "net.bridge.bridge-nf-call-ip6tables = 1" > /dev/null ; then
   cat <<EOF >  /etc/sysctl.d/90-k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
fi
sysctl --system

cleanup old version

    kubectl drain <node> --delete-local-data --force --ignore-daemonsets

    kubeadm reset

reinstall (with flannel, weave seems to officially support only 1.10?)

    kubeadm init --pod-network-cidr=10.244.0.0/16

Save join command printed by the command above to /etc/kubernetes/join.cmd to be able to join other nodes

get & install flannel

    curl https://raw.githubusercontent.com/coreos/flannel/bc79dd1505b0c8681ece4de4c0d86c5cd2643275/Documentation/kube-flannel.yml > kube-flannel.yml
    kubectl create -f kube-flannel.yml

fix coredns on older docker/SElinux versions

This is the issue I did hit this time: the dns would not start up. As discussed in kubeadm issue 998 this is caused by a bug in older docker/SElinux. One could disable SElinux, or upgrade docker, but these weren’t options for us, so

    kubectl -n kube-system edit deploy coredns
    # remove "allowPrivilegeEscalation: false"

allow pods on master node

This installation was a single node install, so allow pods on master node:

   kubectl taint nodes --all node-role.kubernetes.io/master-