碼上快樂

相關內容    簡體    繁體

k8s安裝部署問題、解決方案匯總

本文轉載自   查看原文   2019-06-20 10:16   2385 

 

角色 節點名 節點ip
master n1 192.168.14.11
節點1 n2 192.168.14.12
節點2 n3 192.168.14.13

https://raw.githubusercontent.com/lannyMa/scripts/master/k8s/

准備token.csv和bootstrap.kubeconfig文件

- 在master生成token.csv
BOOTSTRAP_TOKEN="41f7e4ba8b7be874fcff18bf5cf41a7c" cat > token.csv<<EOF 41f7e4ba8b7be874fcff18bf5cf41a7c,kubelet-bootstrap,10001,"system:kubelet-bootstrap" EOF - 將bootstrap.kubeconfig同步到所有節點 設置集群參數 kubectl config set-cluster kubernetes \ --certificate-authority=/root/ssl/ca.crt \ --embed-certs=true \ --server=http://192.168.14.11:8080 \ --kubeconfig=bootstrap.kubeconfig 設置客戶端認證參數 kubectl config set-credentials kubelet-bootstrap \ --token="41f7e4ba8b7be874fcff18bf5cf41a7c" \ --kubeconfig=bootstrap.kubeconfig 設置上下文參數 kubectl config set-context default \ --cluster=kubernetes \ --user=kubelet-bootstrap \ --kubeconfig=bootstrap.kubeconfig 設置默認上下文 kubectl config use-context default --kubeconfig=bootstrap.kubeconfig 兩個文件我都放在了/root下.

coredns+dashboard(heapster)+kube-router yaml

https://github.com/lannyMa/scripts/tree/master/k8s

k8s 1.9 on the fly啟動

etcd --advertise-client-urls=http://192.168.14.11:2379 --listen-client-urls=http://0.0.0.0:2379 --debug kube-apiserver --service-cluster-ip-range=10.254.0.0/16 --etcd-servers=http://127.0.0.1:2379 --insecure-bind-address=0.0.0.0 --admission-control=ServiceAccount --service-account-key-file=/root/ssl/ca.key --client-ca-file=/root/ssl/ca.crt --tls-cert-file=/root/ssl/server.crt --tls-private-key-file=/root/ssl/server.key --allow-privileged=true --storage-backend=etcd2 --v=2 --enable-bootstrap-token-auth --token-auth-file=/root/token.csv kube-controller-manager --master=http://127.0.0.1:8080 --service-account-private-key-file=/root/ssl/ca.key --cluster-signing-cert-file=/root/ssl/ca.crt --cluster-signing-key-file=/root/ssl/ca.key --root-ca-file=/root/ssl/ca.crt --v=2 kube-scheduler --master=http://127.0.0.1:8080 --v=2 kubelet --allow-privileged=true --cluster-dns=10.254.0.2 --cluster-domain=cluster.local --v=2 --experimental-bootstrap-kubeconfig=/root/bootstrap.kubeconfig --kubeconfig=/root/kubelet.kubeconfig --fail-swap-on=false kube-proxy --master=http://192.168.14.11:8080 --v=2 kubectl get csr | grep Pending | awk '{print $1}' | xargs kubectl certificate approve

api相對1.7的變化:
https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG.md#before-upgrading
https://mritd.me/2017/10/09/set-up-kubernetes-1.8-ha-cluster/

  • kubelet沒了--api-servers參數,必須用bootstrap方式去連api
  • 移除了 --runtime-config=rbac.authorization.k8s.io/v1beta1 配置,因為 RBAC 已經穩定，被納入了 v1 api，不再需要指定開啟
  • --authorization-mode 授權模型增加了 Node 參數，因為 1.8 后默認 system:node role 不會自動授予 system:nodes 組
  • 增加 --audit-policy-file 參數用於指定高級審計配置
  • 移除 --experimental-bootstrap-token-auth 參數，更換為 --enable-bootstrap-token-auth

k8s1.9 cni(flannel) on the fly啟動

注: HostPort不能使用CNI網絡插件(docker run -p 8081:8080)。這意味着pod中所有HostPort屬性將被簡單地忽略。

mkdir -p /etc/cni/net.d /opt/cni/bin wget https://github.com/containernetworking/plugins/releases/download/v0.6.0/cni-plugins-amd64-v0.6.0.tgz tar xf cni-plugins-amd64-v0.6.0.tgz -C /opt/cni/bin cat > /etc/cni/net.d/10-flannel.conflist<<EOF { "name":"cni0", "cniVersion":"0.3.1", "plugins":[ { "type":"flannel", "delegate":{ "forceAddress":true, "isDefaultGateway":true } }, { "type":"portmap", "capabilities":{ "portMappings":true } } ] } EOF
etcd --advertise-client-urls=http://192.168.14.11:2379 --listen-client-urls=http://0.0.0.0:2379 --debug kube-apiserver --service-cluster-ip-range=10.254.0.0/16 --etcd-servers=http://127.0.0.1:2379 --insecure-bind-address=0.0.0.0 --admission-control=ServiceAccount --service-account-key-file=/root/ssl/ca.key --client-ca-file=/root/ssl/ca.crt --tls-cert-file=/root/ssl/server.crt --tls-private-key-file=/root/ssl/server.key --allow-privileged=true --storage-backend=etcd2 --v=2 --enable-bootstrap-token-auth --token-auth-file=/root/token.csv kube-controller-manager --master=http://127.0.0.1:8080 --service-account-private-key-file=/root/ssl/ca.key --cluster-signing-cert-file=/root/ssl/ca.crt --cluster-signing-key-file=/root/ssl/ca.key --root-ca-file=/root/ssl/ca.crt --v=2 --allocate-node-cidrs=true --cluster-cidr=10.244.0.0/16 kube-scheduler --master=http://127.0.0.1:8080 --v=2 kubelet --allow-privileged=true --cluster-dns=10.254.0.2 --cluster-domain=cluster.local --v=2 --experimental-bootstrap-kubeconfig=/root/bootstrap.kubeconfig --kubeconfig=/root/kubelet.kubeconfig --fail-swap-on=false --network-plugin=cni kube-proxy --master=http://192.168.14.11:8080 --v=2
 kubectl apply -f https://raw.githubusercontent.com/lannyMa/scripts/master/k8s/cni-flannel.yml

查看pod確實從cni0分到了地址

[root@n1 ~]# kk NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE LABELS default b1 1/1 Running 0 6m 10.244.0.2 n2.ma.com <none> default b2 1/1 Running 0 6m 10.244.1.2 n3.ma.com <none> [root@n1 ~]# kubectl exec -it b1 sh / # ping 10.244.1.2 PING 10.244.1.2 (10.244.1.2): 56 data bytes 64 bytes from 10.244.1.2: seq=0 ttl=62 time=6.292 ms 64 bytes from 10.244.1.2: seq=1 ttl=62 time=0.981 ms

遇到的報錯

kubectl apply -f cni-flannel.yml沒提示報錯,但get pod無顯示

kubectl apply -f https://raw.githubusercontent.com/lannyMa/scripts/master/k8s/cni-flannel.yml 原因: yaml用到了sa未創建 --- apiVersion: v1 kind: ServiceAccount metadata:  name: flannel  namespace: kube-system

pod0無cni0網卡,創建pod后分到的是172.17.x.x即docker0分配的地址

原因: kubelet未加cni啟動參數
--network-plugin=cni

創建成功后提示.kubectl create成功后一直pending,kubelet報錯

因為ipv6沒開,參考how-to-disable-ipv6

ifconfig -a | grep inet6
I1231 23:22:08.343188 15369 kubelet.go:1881] SyncLoop (PLEG): "busybox_default(5a16fa0c-ee3e-11e7-9220-000c29bfdc52)", event: &pleg.PodLifecycleEvent{ID:"5a16fa0c-ee3e-11e7-9220-000c29bfdc52", Type:"ContainerDied", Data:"76e584c83f8dd3d54c759ac637bb47aa2a94de524372e282abde004d1cfbcd1b"} W1231 23:22:08.343338 15369 pod_container_deletor.go:77] Container "76e584c83f8dd3d54c759ac637bb47aa2a94de524372e282abde004d1cfbcd1b" not found in pod's containers I1231 23:22:08.644401 15369 kuberuntime_manager.go:403] No ready sandbox for pod "busybox_default(5a16fa0c-ee3e-11e7-9220-000c29bfdc52)" can be found. Need to start a new one E1231 23:22:08.857121 15369 cni.go:259] Error adding network: open /proc/sys/net/ipv6/conf/eth0/accept_dad: no such file or directory E1231 23:22:08.857144 15369 cni.go:227] Error while adding to cni network: open /proc/sys/net/ipv6/conf/eth0/accept_dad: no such file or directory E1231 23:22:08.930343 15369 remote_runtime.go:92] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = NetworkPlugin cni failed to set up pod "busybox_default" network: open /proc/sys/net/ipv6/conf/eth0/accept_dad: no such file or directory

多次實驗,導致node的網絡配置污染,所以新驗證時一定要reboot,清理環境

rm -rf /var/lib/kubelet/ docker rm $(docker ps -a -q)

controller需要加這兩個參數,且cluster-cidr的地址和cni-flannel.yaml里的地址要一致.和svc網段的地址(--service-cluster-ip-range)不要相同.

--allocate-node-cidrs=true --cluster-cidr=10.244.0.0/16

no IP addresses available in range set: 10.244.0.1-10.244.0.254

沒地址了,pod一直在創建中....
參考: https://github.com/kubernetes/kubernetes/issues/57280

現象:

- kubelet報錯
E0101 00:06:38.629105 1109 kuberuntime_manager.go:647] createPodSandbox for pod "busybox2_default(7fa06467-ee44-11e7-a440-000c29bfdc52)" failed: rpc error: code = Unknown desc = NetworkPlugin cni failed to set up pod "busybox2_default" network: failed to allocate for range 0: no IP addresses available in range set: 10.244.0.1-10.244.0.254 E0101 00:06:38.629143 1109 pod_workers.go:186] Error syncing pod 7fa06467-ee44-11e7-a440-000c29bfdc52 ("busybox2_default(7fa06467-ee44-11e7-a440-000c29bfdc52)"), skipping: failed to "CreatePodSandbox" for "busybox2_default(7fa06467-ee44-11e7-a440-000c29bfdc52)" with CreatePodSandboxError: "CreatePodSandbox for pod \"busybox2_default(7fa06467-ee44-11e7-a440-000c29bfdc52)\" failed: rpc error: code = Unknown desc = NetworkPlugin cni failed to set up pod \"busybox2_default\" network: failed to allocate for range 0: no IP addresses available in range set: 10.244.0.1-10.244.0.254" - ip地址 /var/lib/cni/networks# ls cbr0/ 10.244.0.10 10.244.0.123 10.244.0.147 10.244.0.170 10.244.0.194 10.244.0.217 10.244.0.240 10.244.0.35 10.244.0.59 10.244.0.82 10.244.0.100 10.244.0.124 10.244.0.148 10.244.0.171 10.244.0.195 10.244.0.218 10.244.0.241 10.244.0.36 10.244.0.6 10.244.0.83 10.244.0.101 10.244.0.125 10.244.0.149 10.244.0.172 10.244.0.196 10.244.0.219 10.244.0.242 10.244.0.37 10.244.0.60 10.244.0.84 10.244.0.102 10.244.0.126 10.244.0.15 10.244.0.173 10.244.0.197 10.244.0.22 10.244.0.243 10.244.0.38 10.244.0.61 10.244.0.85 10.244.0.103 10.244.0.127 10.244.0.150 10.244.0.174 10.244.0.198 10.244.0.220 10.244.0.244 10.244.0.39 10.244.0.62 10.244.0.86 10.244.0.104 10.244.0.128 10.244.0.151 10.244.0.175 10.244.0.199 10.244.0.221 10.244.0.245 10.244.0.4 10.244.0.63 10.244.0.87 10.244.0.105 10.244.0.129 10.244.0.152 10.244.0.176 10.244.0.2 10.244.0.222 10.244.0.246 10.244.0.40 10.244.0.64 10.244.0.88 10.244.0.106 10.244.0.13 10.244.0.153 10.244.0.177 10.244.0.20 10.244.0.223 10.244.0.247 10.244.0.41 10.244.0.65 10.244.0.89 10.244.0.107 10.244.0.130 10.244.0.154 10.244.0.178 10.244.0.200 10.244.0.224 10.244.0.248 10.244.0.42 10.244.0.66 10.244.0.9 10.244.0.108 10.244.0.131 10.244.0.155 10.244.0.179 10.244.0.201 10.244.0.225 10.244.0.249 10.244.0.43 10.244.0.67 10.244.0.90 10.244.0.109 10.244.0.132 10.244.0.156 10.244.0.18 10.244.0.202 10.244.0.226 10.244.0.25 10.244.0.44 10.244.0.68 10.244.0.91 10.244.0.11 10.244.0.133 10.244.0.157 10.244.0.180 10.244.0.203 10.244.0.227 10.244.0.250 10.244.0.45 10.244.0.69 10.244.0.92 10.244.0.110 10.244.0.134 10.244.0.158 10.244.0.181 10.244.0.204 10.244.0.228 10.244.0.251 10.244.0.46 10.244.0.7 10.244.0.93 10.244.0.111 10.244.0.135 10.244.0.159 10.244.0.182 10.244.0.205 10.244.0.229 10.244.0.252 10.244.0.47 10.244.0.70 10.244.0.94 10.244.0.112 10.244.0.136 10.244.0.16 10.244.0.183 10.244.0.206 10.244.0.23 10.244.0.253 10.244.0.48 10.244.0.71 10.244.0.95 10.244.0.113 10.244.0.137 10.244.0.160 10.244.0.184 10.244.0.207 10.244.0.230 10.244.0.254 10.244.0.49 10.244.0.72 10.244.0.96 10.244.0.114 10.244.0.138 10.244.0.161 10.244.0.185 10.244.0.208 10.244.0.231 10.244.0.26 10.244.0.5 10.244.0.73 10.244.0.97 10.244.0.115 10.244.0.139 10.244.0.162 10.244.0.186 10.244.0.209 10.244.0.232 10.244.0.27 10.244.0.50 10.244.0.74 10.244.0.98 10.244.0.116 10.244.0.14 10.244.0.163 10.244.0.187 10.244.0.21 10.244.0.233 10.244.0.28 10.244.0.51 10.244.0.75 10.244.0.99 10.244.0.117 10.244.0.140 10.244.0.164 10.244.0.188 10.244.0.210 10.244.0.234 10.244.0.29 10.244.0.52 10.244.0.76 last_reserved_ip.0 10.244.0.118 10.244.0.141 10.244.0.165 10.244.0.189 10.244.0.211 10.244.0.235 10.244.0.3 10.244.0.53 10.244.0.77 10.244.0.119 10.244.0.142 10.244.0.166 10.244.0.19 10.244.0.212 10.244.0.236 10.244.0.30 10.244.0.54 10.244.0.78 10.244.0.12 10.244.0.143 10.244.0.167 10.244.0.190 10.244.0.213 10.244.0.237 10.244.0.31 10.244.0.55 10.244.0.79 10.244.0.120 10.244.0.144 10.244.0.168 10.244.0.191 10.244.0.214 10.244.0.238 10.244.0.32 10.244.0.56 10.244.0.8 10.244.0.121 10.244.0.145 10.244.0.169 10.244.0.192 10.244.0.215 10.244.0.239 10.244.0.33 10.244.0.57 10.244.0.80 10.244.0.122 10.244.0.146 10.244.0.17 10.244.0.193 10.244.0.216 10.244.0.24 10.244.0.34 10.244.0.58 10.244.0.81 - flannel創建了很多文件 /var/lib/cni/flannel# ls | wc ; date

解決:

kubeadm reset
systemctl stop kubelet
systemctl stop docker
rm -rf /var/lib/cni/
rm -rf /var/lib/kubelet/*
rm -rf /etc/cni/
ifconfig cni0 down
ifconfig flannel.1 down
ifconfig docker0 down
ip link delete cni0
ip link delete flannel.1
service docker restart 
service kubelet restart
kubeadm join {你的參數}

  • 推薦打開,不打開我沒發現什么問題
echo 'net.bridge.bridge-nf-call-iptables=1' >> /etc/sysctl.conf sysctl -p # 打開ip轉發,下面4條都干上去 net.ipv4.ip_forward = 1 # Disable netfilter on bridges. net.bridge.bridge-nf-call-iptables = 1 net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-arptables = 1

參考:
https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
http://cizixs.com/2017/05/23/container-network-cni
https://k8smeetup.github.io/docs/concepts/cluster-administration/network-plugins/
https://mritd.me/2017/09/20/set-up-ha-kubernetes-cluster-on-aliyun-ecs/
https://coreos.com/flannel/docs/latest/kubernetes.html (不要用legency的那個,那個是kubelet模式)
https://feisky.gitbooks.io/kubernetes/network/flannel/#cni集成
http://blog.csdn.net/idea77/article/details/78793318

kube-proxy ipvs模式

目前還是測試版,打開玩一玩.
參考: https://jicki.me/2017/12/20/kubernetes-1.9-ipvs/#%E5%90%AF%E5%8A%A8-kube-proxy
https://mritd.me/2017/10/10/kube-proxy-use-ipvs-on-kubernetes-1.8/

確保內核有rr模塊
[root@n2 ~]# lsmod | grep ip_vs ip_vs_sh 12688 0 ip_vs_wrr 12697 0 ip_vs_rr 12600 2 ip_vs 141092 8 ip_vs_rr,ip_vs_sh,ip_vs_wrr nf_conntrack 133387 9 ip_vs,nf_nat,nf_nat_ipv4,nf_nat_ipv6,xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_netlink,nf_conntrack_ipv4,nf_conntrack_ipv6 libcrc32c 12644 4 xfs,ip_vs,nf_nat,nf_conntrack 
啟用 ipvs 后與 1.7 版本的配置差異如下： 增加 --feature-gates=SupportIPVSProxyMode=true 選項，用於告訴 kube-proxy 開啟 ipvs 支持，因為目前 ipvs 並未穩定 增加 ipvs-min-sync-period、--ipvs-sync-period、--ipvs-scheduler 三個參數用於調整 ipvs，具體參數值請自行查閱 ipvs 文檔 增加 --masquerade-all 選項，以確保反向流量通過 重點說一下 --masquerade-all 選項: kube-proxy ipvs 是基於 NAT 實現的，當創建一個 service 后，kubernetes 會在每個節點上創建一個網卡，同時幫你將 Service IP(VIP) 綁定上，此時相當於每個 Node 都是一個 ds，而其他任何 Node 上的 Pod，甚至是宿主機服務(比如 kube-apiserver 的 6443)都可能成為 rs；按照正常的 lvs nat 模型，所有 rs 應該將 ds 設置成為默認網關，以便數據包在返回時能被 ds 正確修改；在 kubernetes 將 vip 設置到每個 Node 后，默認路由顯然不可行，所以要設置 --masquerade-all 選項，以便反向數據包能通過 注意:--masquerade-all 選項與 Calico 安全策略控制不兼容,請酌情使用
kube-proxy  --master=http://192.168.14.11:8080 --v=2 --feature-gates=SupportIPVSProxyMode=true --masquerade-all --proxy-mode=ipvs --masquerade-all 注意: 1.需要打開 –feature-gates=SupportIPVSProxyMode=true,官方 –feature-gates=SupportIPVSProxyMode=false 默認是 false 2.–masquerade-all 必須添加這項配置，否則 創建 svc 在 ipvs 不會添加規則 3.打開 ipvs 需要安裝 ipvsadm 軟件， 在 node 中安裝 yum install ipvsadm -y ipvsadm -L -n
[root@n2 ~]# ipvsadm -L -n IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 10.254.0.1:443 rr persistent 10800 -> 192.168.14.11:6443 Masq 1 0 0 TCP 10.254.12.188:80 rr -> 10.244.0.3:80 Masq 1 0 0 -> 10.244.1.4:80 Masq 1 0 0

 

 
 


免責聲明！

本站轉載的文章為個人學習借鑒使用，本站對版權不負任何法律責任。如果侵犯了您的隱私權益，請聯系本站郵箱yoyou2525@163.com刪除。



猜您在找 K8s 常見問題及解決方案 k8s時區問題解決方案 istio(1)：ServiceMesh解決方案-k8s安裝istio k8s的網絡解決方案 kubeadm部署k8s集群node一直是notready狀態的解決方案 K8S部署遇到的問題處理匯總 K8S LoadBalance 私有環境解決方案 == metallb 工具安裝和使用介紹 k8s日志收集解決方案 安裝k8s出現問題匯總 終於解決 k8s 集群中部署 nodelocaldns 的問題
 
粵ICP備18138465號   © 2018-2025 CODEPRJ.COM