KubeSpray部署k8s集群

Kubespray 是 Kubernetes incubator 中的項目，目標是提供 Production Ready Kubernetes 部署方案，該項目基礎是通過 Ansible Playbook 來定義系統與 Kubernetes 集群部署的任務，具有以下幾個特點：

1. l 可以部署在 AWS, GCE, Azure, OpenStack 以及裸機上.
2. l 部署 High Available Kubernetes 集群.
3. l 可組合性 (Composable)，可自行選擇 Network Plugin (flannel, calico, canal, weave) 來部署.
4. l 支持多種 Linux distributions(CoreOS, Debian Jessie, Ubuntu 16.04, CentOS/RHEL7).

GitHub地址：https://github.com/kubernetes-sigs/kubespray

環境准備

1）所以的主機都需要關閉selinux，執行的命令如下：

setenforce 0
sed -i --follow-symlinks 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/sysconfig/selinux

2）防火牆（可選）和網絡設置，所有的主機都執行以下命令：

systemctl stop firewalld & systemctl disable firewalld
modprobe br_netfilter
echo '1' > /proc/sys/net/bridge/bridge-nf-call-iptables
sysctl -w net.ipv4.ip_forward=1

3）#設置內核參數

sudo vim /etc/security/limits.conf
* soft nofile 32768
* hard nofile 65535
* soft nproc 32768
* hadr nproc 65535

4）設置k8s內核參數

sudo vim /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_nonlocal_bind = 1
net.ipv4.ip_forward = 1
vm.swappiness=0

5）重新加載生效

sudo sysctl --system
sudo sysctl -p

6) 安裝 python 及 epel (在Ansible主機上安裝並配置好與各node的免秘鑰登錄)

yum install -y epel-release python36 python36-pip git

部署k8s集群

1）#克隆項目

 wget https://github.com/kubernetes-sigs/kubespray/archive/v2.12.4.tar.gz

2）# Install dependencies from ``requirements.txt``

sudo /usr/bin/pip3.6 install -r requirements.txt

3）# Copy ``inventory/sample`` as ``inventory/mycluster``

cp -rfp inventory/sample inventory/mycluster

4）# Update Ansible inventory file with inventory builder

declare -a IPS=(10.10.1.3 10.10.1.4 10.10.1.5)

CONFIG_FILE=inventory/mycluster/hosts.yaml /usr/bin/python3.6 contrib/inventory_builder/inventory.py ${IPS[@]}

5）# Review and change parameters under ``inventory/mycluster/group_vars``

cat inventory/mycluster/group_vars/all/all.yml

cat inventory/mycluster/group_vars/k8s-cluster/k8s-cluster.yml

6）# Deploy Kubespray with Ansible Playbook - run the playbook as root

# The option `--become` is required, as for example writing SSL keys in /etc/,

# installing packages and interacting with various systemd daemons.

# Without --become the playbook will fail to run!

ansible-playbook -i inventory/mycluster/hosts.yaml  --become --become-user=root cluster.yml

 擴展

Adding node
1、Add the new worker node to your inventory in the appropriate group (or utilize a dynamic inventory).
2、Run the ansible-playbook command, substituting cluster.yml for scale.yml:

如下圖：node3為新增節點

ansible-playbook -i inventory/mycluster/hosts.yaml scale.yml -b -v

Remove nodes
1、hosts.yaml無需修改，運行命令--extra-vars指定node。

ansible-playbook -i inventory/mycluster/hosts.yaml remove-node.yml -b -v --extra-vars "node=node3"

地址：https://github.com/kubernetes-sigs/kubespray/blob/master/docs/getting-started.md

升級

ansible-playbook upgrade-cluster.yml -b -i inventory/sample/hosts.ini -e kube_version=v1.15.0

地址：https://github.com/kubernetes-sigs/kubespray/blob/master/docs/upgrades.md

 卸載

ansible-playbook -i inventory/mycluster/hosts.ini reset.yml

#每台node都要執行
rm -rf /etc/kubernetes/
rm -rf /var/lib/kubelet
rm -rf /var/lib/etcd
rm -rf /usr/local/bin/kubectl
rm -rf /etc/systemd/system/calico-node.service
rm -rf /etc/systemd/system/kubelet.service

reboot

 后記：

1、默認從國外下載鏡像因為node需要能上外網。

2、安裝需要點時間耐心等待，遇到報錯解決后繼續安裝。

3、當添加worker node或者升級集群版本時，會把原先手動修改的集群參數給還原這點要特別注意。