1. 安裝要求
在開始之前，部署Kubernetes集群機器需要滿足以下幾個條件：
-
一台或多台機器，操作系統 CentOS7.x-86_x64
-
硬件配置：2GB或更多RAM，2個CPU或更多CPU，硬盤30GB或更多
-
集群中所有機器之間網絡互通
-
可以訪問外網，需要拉取鏡像
-
禁止swap分區
2. 學習目標
-
在所有節點上安裝Docker和kubeadm
-
部署Kubernetes Master
-
部署容器網絡插件
-
部署 Kubernetes Node，將節點加入Kubernetes集群中
設置主機名 $ hostnamectl set-hostname k8s-master $ hostnamectl set-hostname k8s-node1 關閉防火牆： $ systemctl stop firewalld $ systemctl disable firewalld 關閉selinux： $ sed -i 's/enforcing/disabled/' /etc/selinux/config $ setenforce 0 關閉swap： $ swapoff -a $ 臨時 $ vim /etc/fstab $ 永久 添加主機名與IP對應關系（記得設置主機名）： cat <<EOF > /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 192.168.31.62 k8s-master 192.168.31.63 k8s-node1 EOF 將橋接的IPv4流量傳遞到iptables的鏈： $ cat > /etc/sysctl.d/k8s.conf << EOF net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1 EOF $ sysctl --system
4. 所有節點安裝Docker/kubeadm/kubelet
4.1 安裝Docker
Kubernetes默認CRI（容器運行時）為Docker，因此先安裝Docker。
$ wget https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo -O /etc/yum.repos.d/docker-ce.repo $ yum -y install docker-ce-18.06.1.ce-3.el7 $ systemctl enable docker && systemctl start docker $ docker --version Docker version 18.06.1-ce, build e68fc7a
關於docker遇到的坑可以參考此篇文章：https://blog.csdn.net/zhonglinzhang/article/details/85254035
docker 加速鏡像配置
$ mkdir /etc/docker/
cat > /etc/docker/daemon.json <<EOF
{
"exec-opts": ["native.cgroupdriver=cgroupfs"],
"registry-mirrors": ["http://f1361db2.m.daocloud.io"],
"log-driver": "json-file",
"log-opts": {
"max-size": "100m", "max-file":"3"
},
"storage-driver": "overlay2",
"storage-opts": [
"overlay2.override_kernel_check=true"
]
}
EOF
# Restart docker.
systemctl daemon-reload
systemctl restart docker
##max-file=3，意味着一個容器有三個日志，分別是id+.json、id+1.json、id+2.json
嘗試過修改kubelet的cgroup dirver(文件位置：/lib/systemd/system/kubelet.service.d/10-kubeadm.conf)，但是每次啟動kubeadm時會被覆蓋掉，導致kubelet啟動失敗，只能修改docker的cgroup dirver設置；
文檔中docker驅動是systemd
參考：https://kubernetes.io/docs/setup/production-environment/container-runtimes/
4.2 添加阿里雲YUM軟件源
$ cat > /etc/yum.repos.d/kubernetes.repo << EOF [kubernetes] name=Kubernetes baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64 enabled=1 gpgcheck=1 repo_gpgcheck=1 gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg EOF
4.3 安裝kubeadm，kubelet和kubectl
這里安裝最新版
$ yum install -y kubelet kubeadm kubectl $ systemctl enable kubelet
5. 部署Kubernetes Master
kubeadm init \ --apiserver-advertise-address=192.168.124.195 \ --image-repository registry.aliyuncs.com/google_containers \ --kubernetes-version v1.14.0 \ --service-cidr=10.1.0.0/16\ --pod-network-cidr=10.244.0.0/16
這個初始化過程需要幾分鍾，具體時間取決於你的網絡。
由於默認拉取鏡像地址k8s.gcr.io國內無法訪問，這里指定阿里雲鏡像倉庫地址。
安裝完成后根據提示操作
# 所有階段會打印出來 .... Your Kubernetes master has initialized successfully! To start using your cluster, you need to run the following as a regular user: mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config You should now deploy a pod network to the cluster. Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: https://kubernetes.io/docs/concepts/cluster-administration/addons/ You can now join any number of machines by running the following on each node as root: kubeadm join 192.168.124.194:6443 --token 4b3m3i.hrzetk4qb5uh74e4 --discovery-token-ca-cert-hash sha256:357b0fec02af903e66022019cea82ff3a95264479cb5d222ea8e938df2db3d20
給出一些下面要做的步驟，配置 kubectl, 配置網絡組件，以及 worker node 加入的命令。
下面就是跟着提示配置 kubectl
mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config
或者，如果您是 root 用戶，則可以運行：
export KUBECONFIG=/etc/kubernetes/admin.conf
可以看到 status 為 NotReady, 因為網絡組件還沒有配置, 下面配置 flannel。
[root@master ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION master NotReady master 1h v1.14.0
6. 安裝Pod網絡插件（CNI）
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
如果下載quay.io鏡像不成功可以修改為quay-mirror.qiniu.com 和 registry.aliyuncs.com
例如：
下拉鏡像：quay.io/coreos/flannel:v0.10.0-s390x 如果拉取較慢，可以改為：quay-mirror.qiniu.com/coreos/flannel:v0.10.0-s390x 下拉鏡像：gcr.io/google_containers/kube-proxy 可以改為： registry.aliyuncs.com/google_containers/kube-proxy
已修改的kube-flannel.yml
---
apiVersion: extensions/v1beta1
kind: PodSecurityPolicy
metadata:
name: psp.flannel.unprivileged
annotations:
seccomp.security.alpha.kubernetes.io/allowedProfileNames: docker/default
seccomp.security.alpha.kubernetes.io/defaultProfileName: docker/default
apparmor.security.beta.kubernetes.io/allowedProfileNames: runtime/default
apparmor.security.beta.kubernetes.io/defaultProfileName: runtime/default
spec:
privileged: false
volumes:
- configMap
- secret
- emptyDir
- hostPath
allowedHostPaths:
- pathPrefix: "/etc/cni/net.d"
- pathPrefix: "/etc/kube-flannel"
- pathPrefix: "/run/flannel"
readOnlyRootFilesystem: false
# Users and groups
runAsUser:
rule: RunAsAny
supplementalGroups:
rule: RunAsAny
fsGroup:
rule: RunAsAny
# Privilege Escalation
allowPrivilegeEscalation: false
defaultAllowPrivilegeEscalation: false
# Capabilities
allowedCapabilities: ['NET_ADMIN']
defaultAddCapabilities: []
requiredDropCapabilities: []
# Host namespaces
hostPID: false
hostIPC: false
hostNetwork: true
hostPorts:
- min: 0
max: 65535
# SELinux
seLinux:
# SELinux is unsed in CaaSP
rule: 'RunAsAny'
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: flannel
rules:
- apiGroups: ['extensions']
resources: ['podsecuritypolicies']
verbs: ['use']
resourceNames: ['psp.flannel.unprivileged']
- apiGroups:
- ""
resources:
- pods
verbs:
- get
- apiGroups:
- ""
resources:
- nodes
verbs:
- list
- watch
- apiGroups:
- ""
resources:
- nodes/status
verbs:
- patch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: flannel
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: flannel
subjects:
- kind: ServiceAccount
name: flannel
namespace: kube-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: flannel
namespace: kube-system
---
kind: ConfigMap
apiVersion: v1
metadata:
name: kube-flannel-cfg
namespace: kube-system
labels:
tier: node
app: flannel
data:
cni-conf.json: |
{
"name": "cbr0",
"plugins": [
{
"type": "flannel",
"delegate": {
"hairpinMode": true,
"isDefaultGateway": true
}
},
{
"type": "portmap",
"capabilities": {
"portMappings": true
}
}
]
}
net-conf.json: |
{
"Network": "10.244.0.0/16",
"Backend": {
"Type": "vxlan"
}
}
---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
name: kube-flannel-ds-amd64
namespace: kube-system
labels:
tier: node
app: flannel
spec:
template:
metadata:
labels:
tier: node
app: flannel
spec:
hostNetwork: true
nodeSelector:
beta.kubernetes.io/arch: amd64
tolerations:
- operator: Exists
effect: NoSchedule
serviceAccountName: flannel
initContainers:
- name: install-cni
image: quay-mirror.qiniu.com/coreos/flannel:v0.11.0-amd64
command:
- cp
args:
- -f
- /etc/kube-flannel/cni-conf.json
- /etc/cni/net.d/10-flannel.conflist
volumeMounts:
- name: cni
mountPath: /etc/cni/net.d
- name: flannel-cfg
mountPath: /etc/kube-flannel/
containers:
- name: kube-flannel
image: quay-mirror.qiniu.com/coreos/flannel:v0.11.0-amd64
command:
- /opt/bin/flanneld
args:
- --ip-masq
- --kube-subnet-mgr
resources:
requests:
cpu: "100m"
memory: "50Mi"
limits:
cpu: "100m"
memory: "50Mi"
securityContext:
privileged: false
capabilities:
add: ["NET_ADMIN"]
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
volumeMounts:
- name: run
mountPath: /run/flannel
- name: flannel-cfg
mountPath: /etc/kube-flannel/
volumes:
- name: run
hostPath:
path: /run/flannel
- name: cni
hostPath:
path: /etc/cni/net.d
- name: flannel-cfg
configMap:
name: kube-flannel-cfg
---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
name: kube-flannel-ds-arm64
namespace: kube-system
labels:
tier: node
app: flannel
spec:
template:
metadata:
labels:
tier: node
app: flannel
spec:
hostNetwork: true
nodeSelector:
beta.kubernetes.io/arch: arm64
tolerations:
- operator: Exists
effect: NoSchedule
serviceAccountName: flannel
initContainers:
- name: install-cni
image: quay-mirror.qiniu.com/coreos/flannel:v0.11.0-arm64
command:
- cp
args:
- -f
- /etc/kube-flannel/cni-conf.json
- /etc/cni/net.d/10-flannel.conflist
volumeMounts:
- name: cni
mountPath: /etc/cni/net.d
- name: flannel-cfg
mountPath: /etc/kube-flannel/
containers:
- name: kube-flannel
image: quay-mirror.qiniu.com/coreos/flannel:v0.11.0-arm64
command:
- /opt/bin/flanneld
args:
- --ip-masq
- --kube-subnet-mgr
resources:
requests:
cpu: "100m"
memory: "50Mi"
limits:
cpu: "100m"
memory: "50Mi"
securityContext:
privileged: false
capabilities:
add: ["NET_ADMIN"]
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
volumeMounts:
- name: run
mountPath: /run/flannel
- name: flannel-cfg
mountPath: /etc/kube-flannel/
volumes:
- name: run
hostPath:
path: /run/flannel
- name: cni
hostPath:
path: /etc/cni/net.d
- name: flannel-cfg
configMap:
name: kube-flannel-cfg
---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
name: kube-flannel-ds-arm
namespace: kube-system
labels:
tier: node
app: flannel
spec:
template:
metadata:
labels:
tier: node
app: flannel
spec:
hostNetwork: true
nodeSelector:
beta.kubernetes.io/arch: arm
tolerations:
- operator: Exists
effect: NoSchedule
serviceAccountName: flannel
initContainers:
- name: install-cni
image: quay-mirror.qiniu.com/coreos/flannel:v0.11.0-arm
command:
- cp
args:
- -f
- /etc/kube-flannel/cni-conf.json
- /etc/cni/net.d/10-flannel.conflist
volumeMounts:
- name: cni
mountPath: /etc/cni/net.d
- name: flannel-cfg
mountPath: /etc/kube-flannel/
containers:
- name: kube-flannel
image: quay-mirror.qiniu.com/coreos/flannel:v0.11.0-arm
command:
- /opt/bin/flanneld
args:
- --ip-masq
- --kube-subnet-mgr
resources:
requests:
cpu: "100m"
memory: "50Mi"
limits:
cpu: "100m"
memory: "50Mi"
securityContext:
privileged: false
capabilities:
add: ["NET_ADMIN"]
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
volumeMounts:
- name: run
mountPath: /run/flannel
- name: flannel-cfg
mountPath: /etc/kube-flannel/
volumes:
- name: run
hostPath:
path: /run/flannel
- name: cni
hostPath:
path: /etc/cni/net.d
- name: flannel-cfg
configMap:
name: kube-flannel-cfg
---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
name: kube-flannel-ds-ppc64le
namespace: kube-system
labels:
tier: node
app: flannel
spec:
template:
metadata:
labels:
tier: node
app: flannel
spec:
hostNetwork: true
nodeSelector:
beta.kubernetes.io/arch: ppc64le
tolerations:
- operator: Exists
effect: NoSchedule
serviceAccountName: flannel
initContainers:
- name: install-cni
image: quay-mirror.qiniu.com/coreos/flannel:v0.11.0-ppc64le
command:
- cp
args:
- -f
- /etc/kube-flannel/cni-conf.json
- /etc/cni/net.d/10-flannel.conflist
volumeMounts:
- name: cni
mountPath: /etc/cni/net.d
- name: flannel-cfg
mountPath: /etc/kube-flannel/
containers:
- name: kube-flannel
image: quay-mirror.qiniu.com/coreos/flannel:v0.11.0-ppc64le
command:
- /opt/bin/flanneld
args:
- --ip-masq
- --kube-subnet-mgr
resources:
requests:
cpu: "100m"
memory: "50Mi"
limits:
cpu: "100m"
memory: "50Mi"
securityContext:
privileged: false
capabilities:
add: ["NET_ADMIN"]
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
volumeMounts:
- name: run
mountPath: /run/flannel
- name: flannel-cfg
mountPath: /etc/kube-flannel/
volumes:
- name: run
hostPath:
path: /run/flannel
- name: cni
hostPath:
path: /etc/cni/net.d
- name: flannel-cfg
configMap:
name: kube-flannel-cfg
---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
name: kube-flannel-ds-s390x
namespace: kube-system
labels:
tier: node
app: flannel
spec:
template:
metadata:
labels:
tier: node
app: flannel
spec:
hostNetwork: true
nodeSelector:
beta.kubernetes.io/arch: s390x
tolerations:
- operator: Exists
effect: NoSchedule
serviceAccountName: flannel
initContainers:
- name: install-cni
image: quay-mirror.qiniu.com/coreos/flannel:v0.11.0-s390x
command:
- cp
args:
- -f
- /etc/kube-flannel/cni-conf.json
- /etc/cni/net.d/10-flannel.conflist
volumeMounts:
- name: cni
mountPath: /etc/cni/net.d
- name: flannel-cfg
mountPath: /etc/kube-flannel/
containers:
- name: kube-flannel
image: quay-mirror.qiniu.com/coreos/flannel:v0.11.0-s390x
command:
- /opt/bin/flanneld
args:
- --ip-masq
- --kube-subnet-mgr
resources:
requests:
cpu: "100m"
memory: "50Mi"
limits:
cpu: "100m"
memory: "50Mi"
securityContext:
privileged: false
capabilities:
add: ["NET_ADMIN"]
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
volumeMounts:
- name: run
mountPath: /run/flannel
- name: flannel-cfg
mountPath: /etc/kube-flannel/
volumes:
- name: run
hostPath:
path: /run/flannel
- name: cni
hostPath:
path: /etc/cni/net.d
- name: flannel-cfg
configMap:
name: kube-flannel-cfg
再來看狀態
$ kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-78fcdf6894-4744c 1/1 Running 0 5m
kube-system coredns-78fcdf6894-jbvhd 1/1 Running 0 5m
kube-system kube-apiserver-master1 1/1 Running 0 5m
kube-system kube-controller-manager-master1 1/1 Running 0 5m
kube-system kube-flannel-ds-amd64-kp7cr 1/1 Running 0 11s
kube-system kube-proxy-6778v 1/1 Running 0 5m
kube-system kube-scheduler-master1 1/1 Running 0 5m
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
master1 Ready master 1h v1.14.0
$ kubectl get cs
NAME STATUS MESSAGE ERROR
scheduler Healthy ok
controller-manager Healthy ok
etcd-0 Healthy {"health": "true"}
在k8s最新版本 v1.16.3 你看到的是這樣的 $ kubectl get cs NAME AGE controller-manager <unknown> scheduler <unknown> etcd-0 <unknown>
開啟master節點允許部署pod: kubectl taint nodes --all node-role.kubernetes.io/master-
不執行這條命令 創建會報錯誤：nodes are available: 1 node(s) had taints that the pod didn't tolerate.
master 節點的配置基本完成了。
Node 配置
就是執行之前 init 最后打出的那個命令
sudo kubeadm join 192.168.124.195:6443 --token qn1tr1.3xpu3qhu7ettn4gv \
--discovery-token-ca-cert-hash sha256:932f99ce7b294b63dd0f511da047c6ea9cf56fa9d8b4b1df9be70013b0c049c9
如果不幸，你忘記了上面的命令並且找不到了，可以通過 kubeadm token create --print-join-command 來獲取
master復制admin文件到node節點 scp /etc/kubernetes/admin.conf root@node1:/etc/kubernetes/admin.conf
node操作完之后，再來 master 節點看看
$ kubectl get nodes NAME STATUS ROLES AGE VERSION master1 Ready master 7m v1.14.0 node1 Ready <none> 18s v1.14.0
用一個簡單pod測試下
kubectl run -i --tty busybox --image=busybox --restart=Never -- sh # 正常執行，再看看調度情況 $ kubectl get pod --show-all -o wide NAME READY STATUS RESTARTS AGE IP NODE busybox 0/1 Completed 0 48s 10.244.1.2 node1
其他
master節點的token 24小時過期后，可以通過命令產生新的token： kubeadm token list 創建永不過期的token kubeadm token create --ttl 0 master節點上運行命令，可查詢discovery-token-ca-cert-hash值： openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //' 重新加入節點 kubeadm join 192.168.124.195:6443 --token 8xwg8u.lkj382k9ox58qkw9 \ --discovery-token-ca-cert-hash sha256:86291bed442dd1dcd6c26f2213208e10cab0f87763f44e0edf01fa670cd9e8b
查看 kubeadm 的配置
kubeadm config view