k8s学习笔记
本文是我在k8s学习道路的笔记。
环境介绍
- winver win10 1909 18363.1016
- VMware® Workstation 15 Pro 15.5.1 build-15018445
- Ubuntu 20.04.1 LTS * 2
- kubeadm version: &version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.8", GitCommit:"9f2892aab98fe339f3bd70e3c470144299398ace", GitTreeState:"clean", BuildDate:"2020-08-13T16:10:16Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}
- trojan-qt5 version 0.0.9 Early Access
apt-get 源
使用清华的Ubuntu镜像进行加速, 见其官方文档-Ubuntu 镜像使用帮助
代理设置
在win10上安装了一个trojan-qt5，打开【设置】-【高级设置】，打开【局域网共享】
在ubuntu上安装privoxy，这是一个可以将sock5代理转换为http代理的软件
# install
sudo apt-get install privoxy
# config，注意讲192.168.31.34:1080替换为你的trojan或者sock5代理软件的地址+端口，同时后面还跟着一个点
sudo vim /etc/privoxy/config
forward-socks5t / 192.168.31.34:1080 . #添加到最后面一行
sudo systemctl restart privoxy # 重启privoxy
docker代理安装
mkdir -p /etc/systemd/system/docker.service.d
cat << EOF > /etc/systemd/system/docker.service.d/http-proxy.conf
Environment="HTTP_PROXY=http://127.0.0.1:8118" "HTTPS_PROXY=http://127.0.0.1:8118" "NO_PROXY=localhost,127.0.0.1,192.168.31.0/24"
EOF
systemctl daemon-reload
终端代理设置
export PROXY_PORT=8118
export PROXY_IP=127.0.0.1
export http_proxy=http://$PROXY_IP:$PROXY_PORT
export HTTP_PROXY=$http_proxy
export https_proxy=$http_proxy
export HTTPS_PROXY=$http_proxy
export no_proxy="localhost,127.0.0.1,localaddress,.localdomain.com,example.com,10.224.0.0/16,192.168.0.0/16"
docker安装
docker官方文档 Install Docker Engine on Ubuntu
kubeadmin安装
sudo apt-get update && sudo apt-get install -y apt-transport-https curl
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
cat <<EOF | sudo tee /etc/apt/sources.list.d/kubernetes.list
deb https://apt.kubernetes.io/ kubernetes-xenial main
EOF
sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl
关闭交换空间
sudo swapoff -a
初步测试
aside@ubuntu:~$ sudo kubeadm config images pull --v=5
I0815 06:55:30.520668 23171 initconfiguration.go:103] detected and using CRI socket: /var/run/dockershim.sock
I0815 06:55:30.521172 23171 interface.go:400] Looking for default routes with IPv4 addresses
I0815 06:55:30.521375 23171 interface.go:405] Default route transits interface "ens33"
I0815 06:55:30.521843 23171 interface.go:208] Interface ens33 is up
I0815 06:55:30.522201 23171 interface.go:256] Interface "ens33" has 2 addresses :[192.168.31.110/24 fe80::d875:c7db:da80:8b02/64].
I0815 06:55:30.522342 23171 interface.go:223] Checking addr 192.168.31.110/24.
I0815 06:55:30.522602 23171 interface.go:230] IP found 192.168.31.110
I0815 06:55:30.522760 23171 interface.go:262] Found valid IPv4 address 192.168.31.110 for interface "ens33".
I0815 06:55:30.522934 23171 interface.go:411] Found active IP 192.168.31.110
I0815 06:55:30.523129 23171 version.go:183] fetching Kubernetes version from URL: https://dl.k8s.io/release/stable-1.txt
W0815 06:55:33.518557 23171 configset.go:202] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
kubeadm[config/images] Pulled k8s.gcr.io/kube-apiserver:v1.18.8
[config/images] Pulled k8s.gcr.io/kube-controller-manager:v1.18.8
[config/images] Pulled k8s.gcr.io/kube-scheduler:v1.18.8
[config/images] Pulled k8s.gcr.io/kube-proxy:v1.18.8
[config/images] Pulled k8s.gcr.io/pause:3.2
[config/images] Pulled k8s.gcr.io/etcd:3.4.3-0
[config/images] Pulled k8s.gcr.io/coredns:1.6.7
kubeadm init
aside@ubuntu:~$ sudo kubeadm init --pod-network-cidr=10.244.0.0/16 # 加上网络插件
W0815 07:02:55.166496 24701 configset.go:202] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
[init] Using Kubernetes version: v1.18.8
[preflight] Running pre-flight checks
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [ubuntu kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.31.110]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [ubuntu localhost] and IPs [192.168.31.110 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [ubuntu localhost] and IPs [192.168.31.110 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
W0815 07:02:59.112011 24701 manifests.go:225] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"
[control-plane] Creating static Pod manifest for "kube-scheduler"
W0815 07:02:59.113016 24701 manifests.go:225] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 20.502074 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.18" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node ubuntu as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-control-plane] Marking the node ubuntu as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: fgjdxz.autaz335lpp16ue5
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 192.168.31.110:6443 --token p6a11n.l5ywmn8dznnwv9hi \
--discovery-token-ca-cert-hash sha256:1b8203febb2cfffe5bbb9e4e4543c3bcab1751558ccbb13f2275bf7dc0ae524b
kubectl 安装
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
flannel 安装
aside@ubuntu:~$ kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
podsecuritypolicy.policy/psp.flannel.unprivileged created
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
serviceaccount/flannel created
configmap/kube-flannel-cfg created
daemonset.apps/kube-flannel-ds-amd64 created
daemonset.apps/kube-flannel-ds-arm64 created
daemonset.apps/kube-flannel-ds-arm created
daemonset.apps/kube-flannel-ds-ppc64le created
daemonset.apps/kube-flannel-ds-s390x created
验证
aside@ubuntu:~$ kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
default frontend-797f47d685-pfmbl 1/1 Running 0 3d6h
kube-system coredns-66bff467f8-ndxnv 1/1 Running 45 10d
kube-system coredns-66bff467f8-pz4b4 1/1 Running 44 10d
kube-system etcd-ubuntu 1/1 Running 2 10d
kube-system kube-apiserver-ubuntu 1/1 Running 2 10d
kube-system kube-controller-manager-ubuntu 1/1 Running 2 10d
kube-system kube-flannel-ds-amd64-6mw8b 1/1 Running 2 10d
kube-system kube-flannel-ds-amd64-phsh8 1/1 Running 1 10d
kube-system kube-proxy-fjqx8 1/1 Running 2 10d
kube-system kube-proxy-wt6tp 1/1 Running 0 10d
kube-system kube-scheduler-ubuntu 1/1 Running 2 10d
重启发现k8s集群挂了
在master上，查看kubelet启动的日志
kubelet.service - kubelet: The Kubernetes Node Agent
Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/kubelet.service.d
└─10-kubeadm.conf
Active: activating (auto-restart) (Result: exit-code) since Tue 2020-08-25 08:32:12 PDT; 511ms ago
Docs: https://kubernetes.io/docs/home/
Process: 6387 ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARG>
Main PID: 6387 (code=exited, status=255/EXCEPTION)
Aug 25 08:32:12 ubuntu systemd[1]: kubelet.service: Main process exited, code=exited, status=255/EXCEPTION
Aug 25 08:32:12 ubuntu systemd[1]: kubelet.service: Failed with result 'exit-code'.
继续查看日志
aside@ubuntu:~$ journalctl -xeu kubelet
Aug 25 08:32:32 ubuntu kubelet[6555]: I0825 08:32:32.823225 6555 server.go:417] Version: v1.18.8
Aug 25 08:32:32 ubuntu kubelet[6555]: I0825 08:32:32.824147 6555 plugins.go:100] No cloud provider specified.
Aug 25 08:32:32 ubuntu kubelet[6555]: I0825 08:32:32.824187 6555 server.go:838] Client rotation is on, will bootstrap in background
Aug 25 08:32:32 ubuntu kubelet[6555]: I0825 08:32:32.832697 6555 certificate_store.go:130] Loading cert/key pair from "/var/lib/kubelet/pki/kubelet-client-current.pem".
Aug 25 08:32:32 ubuntu kubelet[6555]: I0825 08:32:32.833623 6555 dynamic_cafile_content.go:167] Starting client-ca-bundle::/etc/kubernetes/pki/ca.crt
Aug 25 08:32:32 ubuntu kubelet[6555]: I0825 08:32:32.889005 6555 server.go:647] --cgroups-per-qos enabled, but --cgroup-root was not specified. defaulting to /
Aug 25 08:32:32 ubuntu kubelet[6555]: F0825 08:32:32.889209 6555 server.go:274] failed to run Kubelet: running with swap on is not supported, please disable swap! or set >
Aug 25 08:32:32 ubuntu systemd[1]: kubelet.service: Main process exited, code=exited, status=255/EXCEPTION
-- Subject: Unit process exited
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- An ExecStart= process belonging to unit kubelet.service has exited.
--
-- The process' exit code is 'exited' and its exit status is 255.
Aug 25 08:32:32 ubuntu systemd[1]: kubelet.service: Failed with result 'exit-code'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- The unit kubelet.service has entered the 'failed' state with result 'exit-code'.
~
journalctl -xeu kubectl
原来是重启后交换分区又被开启了
sudo swapoff -a
vim /etc/fstab
# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point> <type> <options> <dump> <pass>
# / was on /dev/sda5 during installation
UUID=56f764cf-f113-4ece-aee8-94dada48c731 / ext4 errors=remount-ro 0 1
# /boot/efi was on /dev/sda1 during installation
UUID=E021-9DE7 /boot/efi vfat umask=0077 0 1
# 请注释下面这行
# /swapfile none swap sw 0 0
/dev/fd0 /media/floppy0 auto rw,user,noauto,exec,utf8 0 0
kubelet就会重启成功
aside@ubuntu:~$ sudo systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/kubelet.service.d
└─10-kubeadm.conf
Active: active (running) since Tue 2020-08-25 08:35:37 PDT; 1min 47s ago
Docs: https://kubernetes.io/docs/home/
Main PID: 8112 (kubelet)
Tasks: 19 (limit: 9455)
Memory: 55.1M
CGroup: /system.slice/kubelet.service
└─8112 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config>
Aug 25 08:35:54 ubuntu kubelet[8112]: E0825 08:35:54.882536 8112 remote_runtime.go:105] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed>
Aug 25 08:35:54 ubuntu kubelet[8112]: E0825 08:35:54.882585 8112 kuberuntime_sandbox.go:68] CreatePodSandbox for pod "coredns-66bff467f8-pz4b4_kube-system(19f4af18-5ef1-4>
Aug 25 08:35:54 ubuntu kubelet[8112]: E0825 08:35:54.882598 8112 kuberuntime_manager.go:727] createPodSandbox for pod "coredns-66bff467f8-pz4b4_kube-system(19f4af18-5ef1->
Aug 25 08:35:54 ubuntu kubelet[8112]: E0825 08:35:54.882645 8112 pod_workers.go:191] Error syncing pod 19f4af18-5ef1-4920-8c95-b9752debcd8e ("coredns-66bff467f8-pz4b4_kub>
Aug 25 08:35:55 ubuntu kubelet[8112]: W0825 08:35:55.206927 8112 docker_sandbox.go:400] failed to read pod IP from plugin/docker: networkPlugin cni failed on the status h>
Aug 25 08:35:55 ubuntu kubelet[8112]: W0825 08:35:55.212089 8112 pod_container_deletor.go:77] Container "7e6e9bede668c8f74e490409ad7ea38a917c8aedbc86dcadac27c782e2caeae8">
Aug 25 08:35:55 ubuntu kubelet[8112]: W0825 08:35:55.213647 8112 cni.go:331] CNI failed to retrieve network namespace path: cannot find network namespace for the terminat>
Aug 25 08:35:55 ubuntu kubelet[8112]: W0825 08:35:55.219142 8112 docker_sandbox.go:400] failed to read pod IP from plugin/docker: networkPlugin cni failed on the status h>
Aug 25 08:35:55 ubuntu kubelet[8112]: W0825 08:35:55.225485 8112 pod_container_deletor.go:77] Container "d6d19f6e622e051432423f91e070bd575aa7bf542a90fce5a6f1244f76c45396">
Aug 25 08:35:55 ubuntu kubelet[8112]: W0825 08:35:55.226680 8112 cni.go:331] CNI failed to retrieve network namespace path: cannot find network namespace for the terminat>
再次验证
aside@ubuntu:~$ kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
default frontend-797f47d685-pfmbl 1/1 Running 0 3d6h
kube-system coredns-66bff467f8-ndxnv 1/1 Running 45 10d
kube-system coredns-66bff467f8-pz4b4 1/1 Running 44 10d
kube-system etcd-ubuntu 1/1 Running 2 10d
kube-system kube-apiserver-ubuntu 1/1 Running 2 10d
kube-system kube-controller-manager-ubuntu 1/1 Running 2 10d
kube-system kube-flannel-ds-amd64-6mw8b 1/1 Running 2 10d
kube-system kube-flannel-ds-amd64-phsh8 1/1 Running 1 10d
kube-system kube-proxy-fjqx8 1/1 Running 2 10d
kube-system kube-proxy-wt6tp 1/1 Running 0 10d
kube-system kube-scheduler-ubuntu 1/1 Running 2 10d