一、軟硬件環境
采用CentOS7.4 minimual，docker 1.12，kubeadm 1.7.5，etcd 3.0， k8s 1.7.6
本章節以下配置內容需要在全部節點上都做配置。我們這里僅選用兩個節點搭建一個實驗環境。
設置主機節點的主機名，在/etc/hosts中配置好映射關系：
10.0.2.15 gqtest1.future
10.0.2.4 gqtest2.future
配置系統防火牆策略，使以上兩個主機在同網段間的通信不受限制：
firewall-cmd --permanent --zone=public --add-rich-rule="rule family="ipv4" source address="10.0.2.0/24" accept"
firewall-cmd --reload
firewall-cmd --list-all
注：對於實驗環境，不妨直接永久關停firewalld。
關閉selinux。
配置系統內核參數使流過網橋的流量也進入iptables/netfilter框架中，在/etc/sysctl.conf中添加以下配置：
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
sysctl -p
注：如果上面執行sysctl -p時報錯，可以先執行一下modprobe br_netfilter，然后再重新執行sysctl -p
二、使用kubeadm工具快速安裝Kubernetes集群
Kubeadm到目前為止還是用於初學者快速安裝和學習k8s使用，不適合用在生產環境中。
1、安裝kubeadm和相關工具
本小節的配置內容需要在所有節點上進行配置。
添加k8s yum 阿里源
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
enabled=1
gpgcheck=0
EOF
yum -y install epel-release
yum clean all
yum makecache
安裝kubeadm和相關工具包：
# yum -y install docker kubelet kubeadm kubectl kubernetes-cni
啟動Docker與kubelet服務：
systemctl enable docker && systemctl start docker
systemctl enable kubelet && systemctl start kubelet
注：此時kubelet的服務運行狀態是異常的，因為缺少主配置文件kubelet.conf。但可以暫不處理，因為在完成Master節點的初始化后才會生成這個配置文件。
2、下載Kubernetes的相關鏡像
本小節的配置內容需要在所有節點上進行配置。
（1）因為無法直接訪問gcr.io下載鏡像，所以需要配置一個國內的容器鏡像加速器
配置一個阿里雲的加速器：
- 登錄 https://cr.console.aliyun.com/
- 在頁面中找到並點擊鏡像加速按鈕，即可看到屬於自己的專屬加速鏈接，選擇centos版本后即可看到類似上面的配置方法提示信息
在系統中執行以下命令（mirror的地址需要更新）：
tee /etc/docker/daemon.json <<-'EOF'
{
"registry-mirrors": ["https://jzv3xt7h.mirror.aliyuncs.com"]
}
EOF
重啟docker服務：
systemctl daemon-reload
systemctl restart docker
（2）下載k8s相關鏡像，下載后將鏡像名改為gcr.io/google_container開頭的名字，以供kubeadm使用。
下面的shell腳本主要做了3件事，下載各種需要用到的容器鏡像、重新打標記為符合google命令規范的版本名稱、清除舊的容器鏡像：
[root@gqtest1 ~]# more get-images.sh
#!/bin/bash
images=(kube-proxy-amd64:v1.7.6 kube-scheduler-amd64:v1.7.6 kube-controller-manager-amd64:v1.7.6 kube-apiserver-amd64:v1.7.6 etcd-amd64:3.0.17 pause-amd64:3.0 kubernetes-
dashboard-amd64:v1.6.1 k8s-dns-sidecar-amd64:1.14.4 k8s-dns-kube-dns-amd64:1.14.4 k8s-dns-dnsmasq-nanny-amd64:1.14.4)
for imageName in ${images[@]} ; do
docker pull cloudnil/$imageName
docker tag cloudnil/$imageName gcr.io/google_containers/$imageName
docker rmi cloudnil/$imageName
done
執行上述shell腳本，等待下載完成后，查看一下下載容器鏡像的結果：
[root@gqtest1 ~]# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
gcr.io/google_containers/kube-apiserver-amd64 v1.7.6 fd35bbc17508 5 months ago 186.1 MB
gcr.io/google_containers/kube-scheduler-amd64 v1.7.6 15c1d3eed0e7 5 months ago 77.2 MB
gcr.io/google_containers/kube-controller-manager-amd64 v1.7.6 41cbd335ed40 5 months ago 138 MB
gcr.io/google_containers/kube-proxy-amd64 v1.7.6 fbb7fbc5b300 5 months ago 114.7 MB
gcr.io/google_containers/k8s-dns-kube-dns-amd64 1.14.4 2d6a3bea02c4 7 months ago 49.38 MB
gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64 1.14.4 13117b1d461f 7 months ago 41.41 MB
gcr.io/google_containers/k8s-dns-sidecar-amd64 1.14.4 c413c7235eb4 7 months ago 41.81 MB
gcr.io/google_containers/etcd-amd64 3.0.17 393e48d05c4e 7 months ago 168.9 MB
gcr.io/google_containers/kubernetes-dashboard-amd64 v1.6.1 c14ffb751676 7 months ago 134.4 MB
gcr.io/google_containers/pause-amd64 3.0 66c684b679d2 7 months ago 746.9 kB
3、運行kubeadm init安裝Master
[root@gqtest1 ~]# kubeadm init --kubernetes-version=v1.7.6
[kubeadm] WARNING: kubeadm is in beta, please do not use it for production clusters.
[init] Using Kubernetes version: v1.7.6
[init] Using Authorization modes: [Node RBAC]
[preflight] Running pre-flight checks
[kubeadm] WARNING: starting in 1.8, tokens expire after 24 hours by default (if you require a non-expiring token use --token-ttl 0)
[certificates] Generated CA certificate and key.
[certificates] Generated API server certificate and key.
[certificates] API Server serving cert is signed for DNS names [gqtest1.future kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 10.0.2.15]
[certificates] Generated API server kubelet client certificate and key.
[certificates] Generated service account token signing key and public key.
[certificates] Generated front-proxy CA certificate and key.
[certificates] Generated front-proxy client certificate and key.
[certificates] Valid certificates and keys now exist in "/etc/kubernetes/pki"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/admin.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/controller-manager.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/scheduler.conf"
[apiclient] Created API client, waiting for the control plane to become ready
[apiclient] All control plane components are healthy after 37.006294 seconds
[token] Using token: 320116.d14b1964f47178bc
[apiconfig] Created RBAC rules
[addons] Applied essential addon: kube-proxy
[addons] Applied essential addon: kube-dns
Your Kubernetes master has initialized successfully!
To start using your cluster, you need to run (as a regular user):
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
http://kubernetes.io/docs/admin/addons/
You can now join any number of machines by running the following on each node
as root:
kubeadm join --token 320116.d14b1964f47178bc 10.0.2.15:6443
注：選項--kubernetes-version=v1.7.6是必須的，否則會因為訪問google網站被牆而無法執行命令。這里使用v1.7.6版本，與上面下載的相關容器鏡像的版本有關。
上面的命令大約需要1分鍾的過程，期間可以觀察下tail -f /var/log/message日志文件的輸出，掌握該配置過程和進度。
上面的輸出信息建議保存一份，后續添加工作節點還要用到。
Kubernetes Master初始化成功后，按提示執行以下操作：
[root@gqtest1 ~]# mkdir -p $HOME/.kube
[root@gqtest1 ~]# cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
[root@gqtest1 ~]# chown $(id -u):$(id -g) $HOME/.kube/config
[root@gqtest1 ~]# kubectl get nodes
NAME STATUS AGE VERSION
gqtest1.future NotReady 32m v1.7.5
[root@gqtest1 ~]# kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system etcd-gqtest1.future 1/1 Running 0 32m
kube-system kube-apiserver-gqtest1.future 1/1 Running 0 32m
kube-system kube-controller-manager-gqtest1.future 1/1 Running 0 32m
kube-system kube-dns-2425271678-gps35 0/3 Pending 0 33m
kube-system kube-proxy-6m2z7 1/1 Running 0 33m
kube-system kube-scheduler-gqtest1.future 1/1 Running 0 32m
[root@gqtest1 ~]# kubectl get nodes
NAME STATUS AGE VERSION
gqtest1.future NotReady 32m v1.7.5
至此完成了Master節點上k8s軟件的安裝，但集群內還沒有可用的工作Node，也缺少容器網絡的配置。
查看pods狀態信息，可以看到還有一個dns的pod處於Pending狀態，這是受缺少容器網絡支持的影響而造成的。
查看nodes狀態信息，看到gqtest1節點的狀態為NotReady 。
4、安裝網絡插件
再詳細看一下Master節點初始化時輸出的提示信息，包括了網絡插件的安裝建議：
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
這里是選擇安裝weave插件，在Master節點上執行：
[root@gqtest1 ~]# kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')"
serviceaccount "weave-net" created
clusterrole "weave-net" created
clusterrolebinding "weave-net" created
role "weave-net" created
rolebinding "weave-net" created
daemonset "weave-net" created
Weave以透明而可靠的方式實現了簡單、安全的網絡。關於k8s網絡插件的介紹詳見本文末尾。
等待一會，再觀察pods的運行狀態，可以看到已經全部處於正常狀態了：
[root@gqtest1 ~]# kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system etcd-gqtest1.future 1/1 Running 0 34m
kube-system kube-apiserver-gqtest1.future 1/1 Running 0 34m
kube-system kube-controller-manager-gqtest1.future 1/1 Running 0 34m
kube-system kube-dns-2425271678-gps35 3/3 Running 0 35m
kube-system kube-proxy-6m2z7 1/1 Running 0 35m
kube-system kube-scheduler-gqtest1.future 1/1 Running 0 34m
kube-system weave-net-hd7k2 2/2 Running 0 1m
安裝一個weave網絡管理工具：
[root@gqtest1 ~]# curl -L git.io/weave -o /usr/local/bin/weave
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
0 0 0 0 0 0 0 0 --:--:-- 0:00:02 --:--:-- 0
0 0 0 595 0 0 93 0 --:--:-- 0:00:06 --:--:-- 220
100 50382 100 50382 0 0 5268 0 0:00:09 0:00:09 --:--:-- 17671
[root@gqtest1 ~]# chmod a+x /usr/local/bin/weave
查看weave網絡服務運行狀態信息：
[root@gqtest1 ~]# weave status
Version: 2.2.0 (failed to check latest version - see logs; next check at 2018/02/23 16:26:07)
Service: router
Protocol: weave 1..2
Name: ea:0f:53:f9:2f:f0(gqtest1.future)
Encryption: disabled
PeerDiscovery: enabled
Targets: 1
Connections: 1 (1 failed)
Peers: 1
TrustedSubnets: none
Service: ipam
Status: ready
Range: 10.32.0.0/12
DefaultSubnet: 10.32.0.0/12
5、安裝Node並加入集群
在工作節點上執行kubeadm join命令，加入集群：
[root@gqtest2 ~]# kubeadm join --token 320116.d14b1964f47178bc 10.0.2.15:6443
[kubeadm] WARNING: kubeadm is in beta, please do not use it for production clusters.
[preflight] Running pre-flight checks
[discovery] Trying to connect to API Server "10.0.2.15:6443"
[discovery] Created cluster-info discovery client, requesting info from "https://10.0.2.15:6443"
[discovery] Cluster info signature and contents are valid, will use API Server "https://10.0.2.15:6443"
[discovery] Successfully established connection with API Server "10.0.2.15:6443"
[bootstrap] Detected server version: v1.7.6
[bootstrap] The server supports the Certificates API (certificates.k8s.io/v1beta1)
[csr] Created API client to obtain unique certificate for this node, generating keys and certificate signing request
[csr] Received signed certificate from the API server, generating KubeConfig...
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf"
Node join complete:
* Certificate signing request sent to master and response
received.
* Kubelet informed of new secure connection details.
Run 'kubectl get nodes' on the master to see this machine join.
默認情況下，Master節點不參與工作負載，但如果希望安裝出一個All-In-One的k8s環境，則可以執行以下命令，讓Master節點也成為一個Node節點：
# kubectl taint nodes --all node-role.kubernetes.io/master-
注：相當於是刪除Node的Label"node-role.kubernetes.io/master"
6、驗證k8s集群是否成功安裝完成
再觀察擴容了一個工作節點后的完整集群的pods運行狀態信息：
[root@gqtest1 ~]# kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system etcd-gqtest1.future 1/1 Running 1 1h
kube-system kube-apiserver-gqtest1.future 1/1 Running 1 1h
kube-system kube-controller-manager-gqtest1.future 1/1 Running 1 1h
kube-system kube-dns-2425271678-gps35 3/3 Running 3 1h
kube-system kube-proxy-0pc5d 1/1 Running 0 43m
kube-system kube-proxy-6m2z7 1/1 Running 1 1h
kube-system kube-scheduler-gqtest1.future 1/1 Running 1 1h
kube-system weave-net-3fh66 2/2 Running 0 43m
kube-system weave-net-hd7k2 2/2 Running 3 1h
查看nodes信息：
[root@gqtest1 ~]# kubectl get nodes
NAME STATUS AGE VERSION
gqtest1.future Ready 1h v1.7.5
gqtest2.future Ready 43m v1.7.5
查看k8s集群狀態信息：
[root@gqtest1 ~]# kubectl get cs
NAME STATUS MESSAGE ERROR
scheduler Healthy ok
controller-manager Healthy ok
etcd-0 Healthy {"health": "true"}
查看k8s集群中的Services狀態信息：
[root@gqtest1 ~]# kubectl get svc kubernetes
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes 10.96.0.1 <none> 443/TCP 2h
[root@gqtest1 ~]# kubectl get svc -n kube-system
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kube-dns 10.96.0.10 <none> 53/UDP,53/TCP 2h
或者直接查看全部Services的完整信息：
[root@gqtest1 ~]# kubectl get svc --all-namespaces -o wide
NAMESPACE NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
default kubernetes 10.96.0.1 <none> 443/TCP 2h <none>
kube-system kube-dns 10.96.0.10 <none> 53/UDP,53/TCP 2h k8s-app=kube-dns
三、k8s部署的故障排錯與調試方法
（1）先掌握有哪些命名空間，有哪些pods，確認每個pod的運行狀態
[root@gqtest1 ~]# kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system etcd-gqtest1.future 1/1 Running 1 2h
kube-system kube-apiserver-gqtest1.future 1/1 Running 1 2h
kube-system kube-controller-manager-gqtest1.future 1/1 Running 1 2h
kube-system kube-dns-2425271678-gps35 3/3 Running 3 2h
kube-system kube-proxy-0pc5d 1/1 Running 0 54m
kube-system kube-proxy-6m2z7 1/1 Running 1 2h
kube-system kube-scheduler-gqtest1.future 1/1 Running 1 2h
kube-system weave-net-3fh66 2/2 Running 0 54m
kube-system weave-net-hd7k2 2/2 Running 3 1h
（2）查看一個指定的pod的詳細配置信息
[root@gqtest1 ~]# kubectl --namespace=kube-system describe pod kube-dns-2425271678-gps35
Name: kube-dns-2425271678-gps35
Namespace: kube-system
Node: gqtest1.future/10.0.2.15
Start Time: Fri, 23 Feb 2018 18:27:04 +0800
Labels: k8s-app=kube-dns
pod-template-hash=2425271678
Annotations: kubernetes.io/created-by={"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"ReplicaSet","namespace":"kube-system","name":"kube-dns-2425271678","uid":"386100c7-187f-11e8-9a79-08002770...
scheduler.alpha.kubernetes.io/critical-pod=
Status: Running
IP: 10.32.0.2
Created By: ReplicaSet/kube-dns-2425271678
Controlled By: ReplicaSet/kube-dns-2425271678
Containers:
kubedns:
Container ID: docker://53ba0a56e18ea8130c414f42983d89100e80646b3ee20557bb47e58079a97745
Image: gcr.io/google_containers/k8s-dns-kube-dns-amd64:1.14.4
Image ID: docker://sha256:2d6a3bea02c4f469c117aaae0ac51668585024a2c9e174403076cc1c5f79860e
Ports: 10053/UDP, 10053/TCP, 10055/TCP
Args:
--domain=cluster.local.
--dns-port=10053
--config-dir=/kube-dns-config
--v=2
State: Running
Started: Fri, 23 Feb 2018 18:49:12 +0800
Last State: Terminated
Reason: Error
Exit Code: 137
Started: Fri, 23 Feb 2018 18:27:05 +0800
Finished: Fri, 23 Feb 2018 18:43:35 +0800
Ready: True
Restart Count: 1
Limits:
memory: 170Mi
Requests:
cpu: 100m
memory: 70Mi
Liveness: http-get http://:10054/healthcheck/kubedns delay=60s timeout=5s period=10s #success=1 #failure=5
Readiness: http-get http://:8081/readiness delay=3s timeout=5s period=10s #success=1 #failure=3
Environment:
PROMETHEUS_PORT: 10055
Mounts:
/kube-dns-config from kube-dns-config (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-dns-token-trvkm (ro)
dnsmasq:
Container ID: docker://8baceefac0a5475d932aa77cc2bd2350a28a046ea2a27313cbac42303d96817d
Image: gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64:1.14.4
Image ID: docker://sha256:13117b1d461f84c5ff47adeaff5b016922e1baab83f47de3320cf4a6f3c4e911
Ports: 53/UDP, 53/TCP
Args:
-v=2
-logtostderr
-configDir=/etc/k8s/dns/dnsmasq-nanny
-restartDnsmasq=true
--
-k
--cache-size=1000
--log-facility=-
--server=/cluster.local/127.0.0.1#10053
--server=/in-addr.arpa/127.0.0.1#10053
--server=/ip6.arpa/127.0.0.1#10053
State: Running
Started: Fri, 23 Feb 2018 18:49:13 +0800
Last State: Terminated
Reason: Error
Exit Code: 137
Started: Fri, 23 Feb 2018 18:27:06 +0800
Finished: Fri, 23 Feb 2018 18:43:35 +0800
Ready: True
Restart Count: 1
Requests:
cpu: 150m
memory: 20Mi
Liveness: http-get http://:10054/healthcheck/dnsmasq delay=60s timeout=5s period=10s #success=1 #failure=5
Environment: <none>
Mounts:
/etc/k8s/dns/dnsmasq-nanny from kube-dns-config (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-dns-token-trvkm (ro)
sidecar:
Container ID: docker://5284f8602b574560a673e55d5c57ed094344016067c1531c3c803267c6a36b2b
Image: gcr.io/google_containers/k8s-dns-sidecar-amd64:1.14.4
Image ID: docker://sha256:c413c7235eb4ba8165ec953c0e886e22bd94f72dd360de7ab42ce340fda6550e
Port: 10054/TCP
Args:
--v=2
--logtostderr
--probe=kubedns,127.0.0.1:10053,kubernetes.default.svc.cluster.local,5,A
--probe=dnsmasq,127.0.0.1:53,kubernetes.default.svc.cluster.local,5,A
State: Running
Started: Fri, 23 Feb 2018 18:49:14 +0800
Last State: Terminated
Reason: Error
Exit Code: 2
Started: Fri, 23 Feb 2018 18:27:07 +0800
Finished: Fri, 23 Feb 2018 18:43:25 +0800
Ready: True
Restart Count: 1
Requests:
cpu: 10m
memory: 20Mi
Liveness: http-get http://:10054/metrics delay=60s timeout=5s period=10s #success=1 #failure=5
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-dns-token-trvkm (ro)
Conditions:
Type Status
Initialized True
Ready True
PodScheduled True
Volumes:
kube-dns-config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: kube-dns
Optional: true
kube-dns-token-trvkm:
Type: Secret (a volume populated by a Secret)
SecretName: kube-dns-token-trvkm
Optional: false
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: CriticalAddonsOnly
node-role.kubernetes.io/master:NoSchedule
node.alpha.kubernetes.io/notReady:NoExecute for 300s
node.alpha.kubernetes.io/unreachable:NoExecute for 300s
Events: <none>
按selector分組，查看service和pod的詳細運行狀態：
[root@gqtest1 ~]# kubectl get svc -n kube-system -l k8s-app=kube-dns -o wide
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
kube-dns 10.96.0.10 <none> 53/UDP,53/TCP 2h k8s-app=kube-dns
[root@gqtest1 ~]# kubectl get pods -n kube-system -l name=weave-net -o wide
NAME READY STATUS RESTARTS AGE IP NODE
weave-net-3fh66 2/2 Running 0 1h 10.0.2.4 gqtest2.future
weave-net-hd7k2 2/2 Running 3 1h 10.0.2.15 gqtest1.future
這是weave status之外，另一種查看weave網絡服務狀態的方法：
[root@gqtest1 ~]# kubectl exec -n kube-system weave-net-hd7k2 -c weave -- /home/weave/weave --local status
Version: 2.2.0 (failed to check latest version - see logs; next check at 2018/02/23 16:26:07)
Service: router
Protocol: weave 1..2
Name: ea:0f:53:f9:2f:f0(gqtest1.future)
Encryption: disabled
PeerDiscovery: enabled
Targets: 1
Connections: 2 (1 established, 1 failed)
Peers: 2 (with 2 established connections)
TrustedSubnets: none
Service: ipam
Status: ready
Range: 10.32.0.0/12
DefaultSubnet: 10.32.0.0/12
查看kubelet產生的事件日志信息，在排錯時很有用：
journalctl -xeu kubelet
查看一個主機節點的配置詳情：
[root@gqtest1 ~]# kubectl describe node gqtest2.future
Name: gqtest2.future
Role:
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/os=linux
kubernetes.io/hostname=gqtest2.future
Annotations: node.alpha.kubernetes.io/ttl=0
volumes.kubernetes.io/controller-managed-attach-detach=true
Taints: <none>
CreationTimestamp: Fri, 23 Feb 2018 19:08:04 +0800
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
OutOfDisk False Fri, 23 Feb 2018 20:14:03 +0800 Fri, 23 Feb 2018 19:08:05 +0800 KubeletHasSufficientDisk kubelet has sufficient disk space available
MemoryPressure False Fri, 23 Feb 2018 20:14:03 +0800 Fri, 23 Feb 2018 19:08:05 +0800 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Fri, 23 Feb 2018 20:14:03 +0800 Fri, 23 Feb 2018 19:08:05 +0800 KubeletHasNoDiskPressure kubelet has no disk pressure
Ready True Fri, 23 Feb 2018 20:14:03 +0800 Fri, 23 Feb 2018 19:50:40 +0800 KubeletReady kubelet is posting ready status
Addresses:
InternalIP: 10.0.2.4
Hostname: gqtest2.future
Capacity:
cpu: 2
memory: 1883376Ki
pods: 110
Allocatable:
cpu: 2
memory: 1780976Ki
pods: 110
System Info:
Machine ID: 53e312c62f2942908f2035d576b42b51
System UUID: B7ADF3E2-298A-47BC-86A3-F11038C80119
Boot ID: cbecf64b-e172-4b12-b9b4-db9646f49e1d
Kernel Version: 3.10.0-693.17.1.el7.x86_64
OS Image: CentOS Linux 7 (Core)
Operating System: linux
Architecture: amd64
Container Runtime Version: docker://1.12.6
Kubelet Version: v1.7.5
Kube-Proxy Version: v1.7.5
ExternalID: gqtest2.future
Non-terminated Pods: (2 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits
--------- ---- ------------ ---------- --------------- -------------
kube-system kube-proxy-0pc5d 0 (0%) 0 (0%) 0 (0%) 0 (0%)
kube-system weave-net-3fh66 20m (1%) 0 (0%) 0 (0%) 0 (0%)
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
CPU Requests CPU Limits Memory Requests Memory Limits
------------ ---------- --------------- -------------
20m (1%) 0 (0%) 0 (0%) 0 (0%)
Events:
FirstSeen LastSeen Count From SubObjectPath Type Reason Message
--------- -------- ----- ---- ------------- -------- ------ -------
24m 24m 1 kube-proxy, gqtest2.future Normal Starting Starting kube-proxy.
23m 23m 1 kubelet, gqtest2.future Normal NodeReady Node gqtest2.future status is now: NodeReady
查看提供dns服務的pod中3個容器中的應用運行日志信息：
[root@gqtest1 ~]# kubectl logs -f kube-dns-2425271678-gps35 -n kube-system -c kubedns
I0223 10:49:12.712082 1 dns.go:48] version: 1.14.3-4-gee838f6
I0223 10:49:12.726504 1 server.go:70] Using configuration read from directory: /kube-dns-config with period 10s
I0223 10:49:12.726545 1 server.go:113] FLAG: --alsologtostderr="false"
I0223 10:49:12.726553 1 server.go:113] FLAG: --config-dir="/kube-dns-config"
I0223 10:49:12.726559 1 server.go:113] FLAG: --config-map=""
I0223 10:49:12.726563 1 server.go:113] FLAG: --config-map-namespace="kube-system"
I0223 10:49:12.726567 1 server.go:113] FLAG: --config-period="10s"
I0223 10:49:12.726571 1 server.go:113] FLAG: --dns-bind-address="0.0.0.0"
I0223 10:49:12.726575 1 server.go:113] FLAG: --dns-port="10053"
I0223 10:49:12.726581 1 server.go:113] FLAG: --domain="cluster.local."
I0223 10:49:12.726588 1 server.go:113] FLAG: --federations=""
I0223 10:49:12.726595 1 server.go:113] FLAG: --healthz-port="8081"
I0223 10:49:12.726599 1 server.go:113] FLAG: --initial-sync-timeout="1m0s"
I0223 10:49:12.726603 1 server.go:113] FLAG: --kube-master-url=""
I0223 10:49:12.726608 1 server.go:113] FLAG: --kubecfg-file=""
I0223 10:49:12.726611 1 server.go:113] FLAG: --log-backtrace-at=":0"
I0223 10:49:12.726617 1 server.go:113] FLAG: --log-dir=""
I0223 10:49:12.726622 1 server.go:113] FLAG: --log-flush-frequency="5s"
I0223 10:49:12.726625 1 server.go:113] FLAG: --logtostderr="true"
I0223 10:49:12.726629 1 server.go:113] FLAG: --nameservers=""
I0223 10:49:12.726633 1 server.go:113] FLAG: --stderrthreshold="2"
I0223 10:49:12.726637 1 server.go:113] FLAG: --v="2"
I0223 10:49:12.726640 1 server.go:113] FLAG: --version="false"
I0223 10:49:12.726646 1 server.go:113] FLAG: --vmodule=""
I0223 10:49:12.726916 1 server.go:176] Starting SkyDNS server (0.0.0.0:10053)
I0223 10:49:12.727218 1 server.go:198] Skydns metrics enabled (/metrics:10055)
I0223 10:49:12.727230 1 dns.go:147] Starting endpointsController
I0223 10:49:12.727234 1 dns.go:150] Starting serviceController
I0223 10:49:12.727490 1 logs.go:41] skydns: ready for queries on cluster.local. for tcp://0.0.0.0:10053 [rcache 0]
I0223 10:49:12.727497 1 logs.go:41] skydns: ready for queries on cluster.local. for udp://0.0.0.0:10053 [rcache 0]
I0223 10:49:13.230199 1 dns.go:171] Initialized services and endpoints from apiserver
I0223 10:49:13.230215 1 server.go:129] Setting up Healthz Handler (/readiness)
I0223 10:49:13.230223 1 server.go:134] Setting up cache handler (/cache)
I0223 10:49:13.230229 1 server.go:120] Status HTTP port 8081
^C
[root@gqtest1 ~]# kubectl logs -f kube-dns-2425271678-gps35 -n kube-system -c sidecar
ERROR: logging before flag.Parse: I0223 10:49:14.751622 1 main.go:48] Version v1.14.3-4-gee838f6
ERROR: logging before flag.Parse: I0223 10:49:14.751973 1 server.go:45] Starting server (options {DnsMasqPort:53 DnsMasqAddr:127.0.0.1 DnsMasqPollIntervalMs:5000 Probes:[{Label:kubedns Server:127.0.0.1:10053 Name:kubernetes.default.svc.cluster.local. Interval:5s Type:1} {Label:dnsmasq Server:127.0.0.1:53 Name:kubernetes.default.svc.cluster.local. Interval:5s Type:1}] PrometheusAddr:0.0.0.0 PrometheusPort:10054 PrometheusPath:/metrics PrometheusNamespace:kubedns})
ERROR: logging before flag.Parse: I0223 10:49:14.751997 1 dnsprobe.go:75] Starting dnsProbe {Label:kubedns Server:127.0.0.1:10053 Name:kubernetes.default.svc.cluster.local. Interval:5s Type:1}
ERROR: logging before flag.Parse: I0223 10:49:14.752105 1 dnsprobe.go:75] Starting dnsProbe {Label:dnsmasq Server:127.0.0.1:53 Name:kubernetes.default.svc.cluster.local. Interval:5s Type:1}
^C
[root@gqtest1 ~]# kubectl logs -f kube-dns-2425271678-gps35 -n kube-system -c dnsmasq
I0223 10:49:13.799678 1 main.go:76] opts: {{/usr/sbin/dnsmasq [-k --cache-size=1000 --log-facility=- --server=/cluster.local/127.0.0.1#10053 --server=/in-addr.arpa/127.0.0.1#10053 --server=/ip6.arpa/127.0.0.1#10053] true} /etc/k8s/dns/dnsmasq-nanny 10000000000}
I0223 10:49:13.800884 1 nanny.go:86] Starting dnsmasq [-k --cache-size=1000 --log-facility=- --server=/cluster.local/127.0.0.1#10053 --server=/in-addr.arpa/127.0.0.1#10053 --server=/ip6.arpa/127.0.0.1#10053]
I0223 10:49:14.638909 1 nanny.go:111]
W0223 10:49:14.639013 1 nanny.go:112] Got EOF from stdout
I0223 10:49:14.639280 1 nanny.go:108] dnsmasq[10]: started, version 2.76 cachesize 1000
I0223 10:49:14.639308 1 nanny.go:108] dnsmasq[10]: compile time options: IPv6 GNU-getopt no-DBus no-i18n no-IDN DHCP DHCPv6 no-Lua TFTP no-conntrack ipset auth no-DNSSEC loop-detect inotify
I0223 10:49:14.639314 1 nanny.go:108] dnsmasq[10]: using nameserver 127.0.0.1#10053 for domain ip6.arpa
I0223 10:49:14.639318 1 nanny.go:108] dnsmasq[10]: using nameserver 127.0.0.1#10053 for domain in-addr.arpa
I0223 10:49:14.639321 1 nanny.go:108] dnsmasq[10]: using nameserver 127.0.0.1#10053 for domain cluster.local
I0223 10:49:14.639328 1 nanny.go:108] dnsmasq[10]: reading /etc/resolv.conf
I0223 10:49:14.639332 1 nanny.go:108] dnsmasq[10]: using nameserver 127.0.0.1#10053 for domain ip6.arpa
I0223 10:49:14.639336 1 nanny.go:108] dnsmasq[10]: using nameserver 127.0.0.1#10053 for domain in-addr.arpa
I0223 10:49:14.639339 1 nanny.go:108] dnsmasq[10]: using nameserver 127.0.0.1#10053 for domain cluster.local
I0223 10:49:14.639343 1 nanny.go:108] dnsmasq[10]: using nameserver 192.168.5.66#53
I0223 10:49:14.639346 1 nanny.go:108] dnsmasq[10]: read /etc/hosts - 7 addresses
^C
注：名為sidecar的容器中應用輸出了一些錯誤日志，據稱是功能bug，已經在后續新版本中修復。
進入到kubedns容器系統中做檢查的方法：
[root@gqtest1 ~]# docker exec -it 53ba0a56e18ea8130c414f42983d89100e80646b3ee20557bb47e58079a97745 /bin/sh
/ # ls
bin etc kube-dns lib mnt root sbin sys usr
dev home kube-dns-config media proc run srv tmp var
/ # ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
12: eth0@if13: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1376 qdisc noqueue state UP
link/ether d6:25:ca:25:47:ac brd ff:ff:ff:ff:ff:ff
inet 10.32.0.2/12 brd 10.47.255.255 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::d425:caff:fe25:47ac/64 scope link tentative flags 08
valid_lft forever preferred_lft forever
注：這進入的是一個名為kube-dns-2425271678-gps35的pod中的一個名為kubedns的容器。
使用kubeadm搭建k8s集群失敗后，怎么重新來過，初始化失敗后的清理命令如下：
kubeadm reset
ifconfig weave down
ip link delete weave
rm -rf /var/lib/cni/
iptables -P INPUT ACCEPT
iptables -P FORWARD ACCEPT
iptables -P OUTPUT ACCEPT
iptables -t nat -F
iptables -t mangle -F
iptables -F
iptables -X
reboot
四、k8s 網絡插件知識掃盲
下面列出的插件是專門為Kubernetes開發的。
Kubenet
Kubenet是專門用於單節點環境，它可以通過與設定規則的雲平台一起使用來實現節點間的通信。Kubenet是一個非常基本的網絡插件，如果你正在尋找跨節點的網絡策略，Kubenet沒有任何幫助。
Flannel
Flannel是一個被CoreOS開發的，專門為k8s設計的overlay網絡方案。Flannel的主要優點是它經過了良好的測試並且成本很低。Flannel 為整個集群的提供分布式處理。k8s 為正確的通信和服務，進行端口映射和分配唯一的ip地址給每個pod。如果你用Google Compute，它將非常兼容。然而，如果你用其他的雲服務商，可能會遇到很多困難。Flannel正解決這個問題。
Weave
Weave是由Weavenetwork開發，用於連接，監視，可視化和監控Kubernetes。通過Weave，可以創建網絡，更快的部署防火牆，並通過自動化的故障排除，提高網絡運維效率。
Weave創建的網絡可以連通在不同位置的容器，比如公有雲、私有雲，虛擬機和裸金屬設備，容器網絡可以承載二層和三層的流量，並支持多播；內建的加密功能讓容器隔離更加容易實現； Weave網絡還可以自動選擇最快的路徑路由容器流量，保證容器的網絡速度。每台運行weave的主機都需要運行幾個必須的容器，透過這些容器實現跨主機通訊。在一個weave網絡中，會有多個運行在不同主機的peer，這些peer起到路由的作用。
在weave routers間會創建TCP或UDP連接，工作的流程是：
- 先執行handshake
- 隨后交換拓撲信息
如果用戶啟用了加密(啟用加密的方法會在后面說明)，這些全雙工的連接會使用UDP協議承載封裝好的網包，並且可以透過防火牆。
在實現上，weave會在主機上創建一個網橋，容器會通過veth peer連接到網橋，一般情況下由weave自帶的地址分配工具自動分配為容器分配地址，如果用戶進行干預，則以用戶設置優先。
因為起到路由作用的weave容器也會連接到上述網橋，所以，weave routers會借助pcap，透過設置為混雜模式的接入網橋的接口捕捉以太網包，但是對於直接透過內核轉發的本地容器間流量或是宿主機與本地容器間的流量則會被排除。
被捕捉的數據包通過UDP協議轉發到其他Host上的weave router peer上，一旦收到這些包，路由會把包通過pcap注入到它的網橋接口或轉發到其他的peers。
weave路由會通過mac地址識別不同的peer所在的位置，連同拓撲信息去決定轉發路徑，避免采取像泛洪般的手段，把每個包都發到每個peer上，以提高網絡性能。
用GRE/VXLAN 的 OpenVSwitch
OpenVSwitch 用於跨節點建立網絡。隧道類型可以是VxLAN或GRE（通用的路由封裝）。GRE用於在IP網絡上進行數據幀的隧道化。在VXLAN的一幀數據中，包括了原來的2層數據包頭，被封裝的一個IP包頭，一個UDP包頭和一個VXLAN包頭。VXLAN更適用於要進行大規模網絡隔離的大型數據中心。值得注意的是，OpenVSwitch也是Xen默認的網絡方案，同樣也適用於像KVM, VIrtualBox, Proxmox VE 或 OpenStack 等平台。
Calico
從k8s 1.0 開始， Calico 為k8s pod提供了三層網絡。Calico提供了簡單的，可擴展的，安全的虛擬網絡。它用邊界網關協議（BGP）為每個Pod提供根分布，並可使用IT基礎設施集成Kubernetes集群。Calico可以幾乎與所有的雲平台兼容，在Kubernetes環境下，具有高可擴展性。除了Kubernetes, Calico 還支持 OpenStack, Mesos, and Docker。
參考1：《Kubernetes權威指南——從Docker到Kubernetes實踐全接觸》第2章。
參考2：大約上百篇的各類技術博客、github.com issues或stackoverflow.com questions等！