CentOS 7.6使用kubeadm部署k8s 1.17.2測試集群實戰篇
作者：尹正傑
版權聲明：原創作品，謝絕轉載！否則將追究法律責任。
kubernetes技術已經稱為原生雲技術的事實標准，它是目前基礎軟件領域最為熱門的分布式調度和管理平台。於是,kubernetes也幾乎成了時下開發工程師和運維工程師必備的技能之一。今天我們就來一起搭建一個Mini版本的kubernetes集群來簡單體驗一下吧。
一.主機基礎環境准備
1>.測試環境說明
測試使用的kubernetes集群可由一個master主機及一個以上(建議至少兩個)node主機組成，這些主機可以是物理服務器,也可以是vmware,virtualbox或kvm等虛擬化平台上的虛擬機，甚至是公有雲上的VPS主機。 本測試環境將由master200,node201,node202,node203四個獨立的主機組成，它們分別擁有4核心的CPU及4G的內存資源，操作系統均為"CentOS Linux release 7.6.1810 (Core)",域名為"yinzhengjie.org.cn",具體配置如下圖所示。
此外,各主機需要預設的系統環境如下:
(1)借助NTP服務設置節點時間精確同步;
(2)通過DNS完成各節點的主機名解析，測試環境主機數量較少時也可以使用hosts文件進行;
(3)關閉各節點的iptables或firewalld服務,並確保它們被禁止隨系統引導過程啟動;
(4)各節點禁用Selinux(否則在運行容器時可以會遇到各種奇葩報錯);
(5)各節點禁用所有的swap設備(生產環境中強烈建議禁用,雖說K8S也支持配置參數來啟用swap但這樣會降低集群性能，使用"swapoff -a"只是臨時關閉交換分區使用,永久關閉需要編輯"/etc/fstab"文件將掛載swap哪一行前面加一個"#"進行注釋);
(6)若要使用ipvs模型的proxy，各節點還需要載入ipvs相關的各模塊;
2>.搭建內網的時間服務器(設定集群各節點時鍾同步)
[root@master200.yinzhengjie.org.cn ~]# yum -y install chrony Loaded plugins: fastestmirror Determining fastest mirrors * base: mirror.bit.edu.cn * extras: mirrors.tuna.tsinghua.edu.cn * updates: mirrors.tuna.tsinghua.edu.cn base | 3.6 kB 00:00:00 extras | 2.9 kB 00:00:00 updates | 2.9 kB 00:00:00 (1/2): extras/7/x86_64/primary_db | 159 kB 00:00:00 (2/2): updates/7/x86_64/primary_db | 5.9 MB 00:00:00 Resolving Dependencies --> Running transaction check ---> Package chrony.x86_64 0:3.4-1.el7 will be installed --> Processing Dependency: libseccomp.so.2()(64bit) for package: chrony-3.4-1.el7.x86_64 --> Running transaction check ---> Package libseccomp.x86_64 0:2.3.1-3.el7 will be installed --> Finished Dependency Resolution Dependencies Resolved ============================================================================================================================================================================================================================================================================== Package Arch Version Repository Size ============================================================================================================================================================================================================================================================================== Installing: chrony x86_64 3.4-1.el7 base 251 k Installing for dependencies: libseccomp x86_64 2.3.1-3.el7 base 56 k Transaction Summary ============================================================================================================================================================================================================================================================================== Install 1 Package (+1 Dependent package) Total download size: 306 k Installed size: 788 k Downloading packages: (1/2): chrony-3.4-1.el7.x86_64.rpm | 251 kB 00:00:00 (2/2): libseccomp-2.3.1-3.el7.x86_64.rpm | 56 kB 00:00:00 ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Total 1.7 MB/s | 306 kB 00:00:00 Running transaction check Running transaction test Transaction test succeeded Running transaction Installing : libseccomp-2.3.1-3.el7.x86_64 1/2 Installing : chrony-3.4-1.el7.x86_64 2/2 Verifying : libseccomp-2.3.1-3.el7.x86_64 1/2 Verifying : chrony-3.4-1.el7.x86_64 2/2 Installed: chrony.x86_64 0:3.4-1.el7 Dependency Installed: libseccomp.x86_64 0:2.3.1-3.el7 Complete! [root@master200.yinzhengjie.org.cn ~]#
[root@master200.yinzhengjie.org.cn ~]# cp /etc/chrony.conf /etc/chrony.conf-`date +%F` [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# sed -r -i 's@(^server)@#\1@g' /etc/chrony.conf [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# sed -r -i 's@#(allow) 192.168.0.0/16@\1 172.200.0.0/21@' /etc/chrony.conf [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# sed -r -i 's@#(local)@\1@' /etc/chrony.conf [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# vim /etc/chrony.conf [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# egrep -v "^#|^$" /etc/chrony.conf server master200.yinzhengjie.org.cn iburst driftfile /var/lib/chrony/drift makestep 1.0 3 rtcsync allow 172.200.0.0/21 allow 127.0.0.0/8 local stratum 10 logdir /var/log/chrony [root@master200.yinzhengjie.org.cn ~]#
[root@master200.yinzhengjie.org.cn ~]# systemctl restart chronyd.service [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# systemctl status chronyd.service ● chronyd.service - NTP client/server Loaded: loaded (/usr/lib/systemd/system/chronyd.service; enabled; vendor preset: enabled) Active: active (running) since Tue 2020-02-04 17:25:31 CST; 4s ago Docs: man:chronyd(8) man:chrony.conf(5) Process: 5658 ExecStartPost=/usr/libexec/chrony-helper update-daemon (code=exited, status=0/SUCCESS) Process: 5654 ExecStart=/usr/sbin/chronyd $OPTIONS (code=exited, status=0/SUCCESS) Main PID: 5656 (chronyd) CGroup: /system.slice/chronyd.service └─5656 /usr/sbin/chronyd Feb 04 17:25:31 master200.yinzhengjie.org.cn systemd[1]: Starting NTP client/server... Feb 04 17:25:31 master200.yinzhengjie.org.cn chronyd[5656]: chronyd version 3.4 starting (+CMDMON +NTP +REFCLOCK +RTC +PRIVDROP +SCFILTER +SIGND +ASYNCDNS +SECHASH +IPV6 +DEBUG) Feb 04 17:25:31 master200.yinzhengjie.org.cn systemd[1]: Started NTP client/server. Feb 04 17:25:35 master200.yinzhengjie.org.cn chronyd[5656]: Selected source 172.200.1.200 [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# systemctl enable chronyd.service [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# systemctl list-unit-files | grep chronyd chronyd.service enabled [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]#
[root@node201.yinzhengjie.org.cn ~]# vim /etc/chrony.conf [root@node201.yinzhengjie.org.cn ~]# [root@node201.yinzhengjie.org.cn ~]# egrep -v "^#|^$" /etc/chrony.conf server master200.yinzhengjie.org.cn iburst driftfile /var/lib/chrony/drift makestep 1.0 3 rtcsync logdir /var/log/chrony [root@node201.yinzhengjie.org.cn ~]# [root@node201.yinzhengjie.org.cn ~]# systemctl restart chronyd.service [root@node201.yinzhengjie.org.cn ~]# [root@node201.yinzhengjie.org.cn ~]# systemctl enable chronyd.service [root@node201.yinzhengjie.org.cn ~]# [root@node201.yinzhengjie.org.cn ~]# systemctl list-unit-files | grep chronyd chronyd.service enabled [root@node201.yinzhengjie.org.cn ~]# [root@node201.yinzhengjie.org.cn ~]#
[root@node201.yinzhengjie.org.cn ~]# chronyc sources 210 Number of sources = 1 MS Name/IP address Stratum Poll Reach LastRx Last sample =============================================================================== ^* master200.yinzhengjie.or> 11 6 37 48 +75ns[+2097ns] +/- 87ms [root@node201.yinzhengjie.org.cn ~]#
[root@node201.yinzhengjie.org.cn ~]# chronyc sourcestats -v 210 Number of sources = 1 .- Number of sample points in measurement set. / .- Number of residual runs with same sign. | / .- Length of measurement set (time). | | / .- Est. clock freq error (ppm). | | | / .- Est. error in freq. | | | | / .- Est. offset. | | | | | | On the -. | | | | | | samples. \ | | | | | | | Name/IP Address NP NR Span Frequency Freq Skew Offset Std Dev ============================================================================== master200.yinzhengjie.or> 6 6 136 -0.002 0.471 -23ns 4950ns [root@node201.yinzhengjie.org.cn ~]#
3>.啟用ipvs內核模塊
[root@master200.yinzhengjie.org.cn ~]# vim /etc/sysconfig/modules/ipvs.modules [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# cat /etc/sysconfig/modules/ipvs.modules #!/bin/bash # #******************************************************************** #Author: yinzhengjie #QQ: 1053419035 #Date: 2019-11-30 #URL: http://www.cnblogs.com/yinzhengjie #Description： enable ipvs script #Copyright notice: original works, no reprint! Otherwise, legal liability will be investigated. #******************************************************************** ipvs_mods_dir="/usr/lib/modules/$(uname -r)/kernel/net/netfilter/ipvs" for mod in $(ls $ipvs_mods_dir | grep -o "^[^.]*");do /usr/sbin/modinfo -F filename $mod &> /dev/null if [ $? -eq 0 ];then /sbin/modprobe $mod fi done [root@master200.yinzhengjie.org.cn ~]#
[root@master200.yinzhengjie.org.cn ~]# lsmod | grep ip_vs [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# chmod +x /etc/sysconfig/modules/ipvs.modules [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# /etc/sysconfig/modules/ipvs.modules [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# lsmod | grep ip_vs ip_vs_wrr 12697 0 ip_vs_wlc 12519 0 ip_vs_sh 12688 0 ip_vs_sed 12519 0 ip_vs_rr 12600 0 ip_vs_pe_sip 12740 0 nf_conntrack_sip 33860 1 ip_vs_pe_sip ip_vs_nq 12516 0 ip_vs_lc 12516 0 ip_vs_lblcr 12922 0 ip_vs_lblc 12819 0 ip_vs_ftp 13079 0 nf_nat 26787 1 ip_vs_ftp ip_vs_dh 12688 0 ip_vs 145497 24 ip_vs_dh,ip_vs_lc,ip_vs_nq,ip_vs_rr,ip_vs_sh,ip_vs_ftp,ip_vs_sed,ip_vs_wlc,ip_vs_wrr,ip_vs_pe_sip,ip_vs_lblcr,ip_vs_lblc nf_conntrack 133095 3 ip_vs,nf_nat,nf_conntrack_sip libcrc32c 12644 4 xfs,ip_vs,nf_nat,nf_conntrack [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]#
[root@master200.yinzhengjie.org.cn ~]# ll /etc/sysconfig/modules/ipvs.modules -rwxr-xr-x 1 root root 680 Feb 4 10:08 /etc/sysconfig/modules/ipvs.modules [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# scp -p /etc/sysconfig/modules/ipvs.modules node201.yinzhengjie.org.cn:/etc/sysconfig/modules/ root@node201.yinzhengjie.org.cn's password: ipvs.modules 100% 680 521.4KB/s 00:00 [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]#
[root@master200.yinzhengjie.org.cn ~]# ll /etc/sysconfig/modules/ipvs.modules -rwxr-xr-x 1 root root 680 Feb 4 10:08 /etc/sysconfig/modules/ipvs.modules [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# scp -p /etc/sysconfig/modules/ipvs.modules node202.yinzhengjie.org.cn:/etc/sysconfig/modules/ root@node202.yinzhengjie.org.cn's password: ipvs.modules 100% 680 521.4KB/s 00:00 [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]#
[root@master200.yinzhengjie.org.cn ~]# ll /etc/sysconfig/modules/ipvs.modules -rwxr-xr-x 1 root root 680 Feb 4 10:08 /etc/sysconfig/modules/ipvs.modules [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# scp -p /etc/sysconfig/modules/ipvs.modules node203.yinzhengjie.org.cn:/etc/sysconfig/modules/ root@node203.yinzhengjie.org.cn's password: ipvs.modules 100% 680 521.4KB/s 00:00 [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]#
二.安裝docker並啟動(每個節點都需要安裝docker環境)
1>.下載docker阿里雲的軟件源文件
[root@master200.yinzhengjie.org.cn ~]# cd /etc/yum.repos.d/ [root@master200.yinzhengjie.org.cn /etc/yum.repos.d]# [root@master200.yinzhengjie.org.cn /etc/yum.repos.d]# ll total 32 -rw-r--r--. 1 root root 1664 Nov 23 2018 CentOS-Base.repo -rw-r--r--. 1 root root 1309 Nov 23 2018 CentOS-CR.repo -rw-r--r--. 1 root root 649 Nov 23 2018 CentOS-Debuginfo.repo -rw-r--r--. 1 root root 314 Nov 23 2018 CentOS-fasttrack.repo -rw-r--r--. 1 root root 630 Nov 23 2018 CentOS-Media.repo -rw-r--r--. 1 root root 1331 Nov 23 2018 CentOS-Sources.repo -rw-r--r--. 1 root root 5701 Nov 23 2018 CentOS-Vault.repo [root@master200.yinzhengjie.org.cn /etc/yum.repos.d]# [root@master200.yinzhengjie.org.cn /etc/yum.repos.d]# wget https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo --2020-02-04 10:44:27-- https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo Resolving mirrors.aliyun.com (mirrors.aliyun.com)... 61.240.147.118, 27.221.92.111, 61.240.147.114, ... Connecting to mirrors.aliyun.com (mirrors.aliyun.com)|61.240.147.118|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 2640 (2.6K) [application/octet-stream] Saving to: ‘docker-ce.repo’ 100%[===================================================================================================================================================================================================================>] 2,640 --.-K/s in 0s 2020-02-04 10:44:27 (69.9 MB/s) - ‘docker-ce.repo’ saved [2640/2640] [root@master200.yinzhengjie.org.cn /etc/yum.repos.d]# [root@master200.yinzhengjie.org.cn /etc/yum.repos.d]# ll total 36 -rw-r--r--. 1 root root 1664 Nov 23 2018 CentOS-Base.repo -rw-r--r--. 1 root root 1309 Nov 23 2018 CentOS-CR.repo -rw-r--r--. 1 root root 649 Nov 23 2018 CentOS-Debuginfo.repo -rw-r--r--. 1 root root 314 Nov 23 2018 CentOS-fasttrack.repo -rw-r--r--. 1 root root 630 Nov 23 2018 CentOS-Media.repo -rw-r--r--. 1 root root 1331 Nov 23 2018 CentOS-Sources.repo -rw-r--r--. 1 root root 5701 Nov 23 2018 CentOS-Vault.repo -rw-r--r-- 1 root root 2640 Feb 3 16:23 docker-ce.repo [root@master200.yinzhengjie.org.cn /etc/yum.repos.d]# [root@master200.yinzhengjie.org.cn /etc/yum.repos.d]#
2>.安裝docker
[root@master200.yinzhengjie.org.cn ~]# yum -y install docker-ce Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile * base: mirror.bit.edu.cn * extras: mirrors.tuna.tsinghua.edu.cn * updates: mirrors.tuna.tsinghua.edu.cn docker-ce-stable | 3.5 kB 00:00:00 (1/2): docker-ce-stable/x86_64/primary_db | 37 kB 00:00:00 (2/2): docker-ce-stable/x86_64/updateinfo | 55 B 00:00:00 Resolving Dependencies --> Running transaction check ---> Package docker-ce.x86_64 3:19.03.5-3.el7 will be installed --> Processing Dependency: container-selinux >= 2:2.74 for package: 3:docker-ce-19.03.5-3.el7.x86_64 --> Processing Dependency: containerd.io >= 1.2.2-3 for package: 3:docker-ce-19.03.5-3.el7.x86_64 --> Processing Dependency: docker-ce-cli for package: 3:docker-ce-19.03.5-3.el7.x86_64 --> Processing Dependency: libcgroup for package: 3:docker-ce-19.03.5-3.el7.x86_64 --> Running transaction check ---> Package container-selinux.noarch 2:2.107-3.el7 will be installed --> Processing Dependency: policycoreutils-python for package: 2:container-selinux-2.107-3.el7.noarch ---> Package containerd.io.x86_64 0:1.2.10-3.2.el7 will be installed ---> Package docker-ce-cli.x86_64 1:19.03.5-3.el7 will be installed ---> Package libcgroup.x86_64 0:0.41-21.el7 will be installed --> Running transaction check ---> Package policycoreutils-python.x86_64 0:2.5-33.el7 will be installed --> Processing Dependency: policycoreutils = 2.5-33.el7 for package: policycoreutils-python-2.5-33.el7.x86_64 --> Processing Dependency: setools-libs >= 3.3.8-4 for package: policycoreutils-python-2.5-33.el7.x86_64 --> Processing Dependency: libsemanage-python >= 2.5-14 for package: policycoreutils-python-2.5-33.el7.x86_64 --> Processing Dependency: audit-libs-python >= 2.1.3-4 for package: policycoreutils-python-2.5-33.el7.x86_64 --> Processing Dependency: python-IPy for package: policycoreutils-python-2.5-33.el7.x86_64 --> Processing Dependency: libqpol.so.1(VERS_1.4)(64bit) for package: policycoreutils-python-2.5-33.el7.x86_64 --> Processing Dependency: libqpol.so.1(VERS_1.2)(64bit) for package: policycoreutils-python-2.5-33.el7.x86_64 --> Processing Dependency: libapol.so.4(VERS_4.0)(64bit) for package: policycoreutils-python-2.5-33.el7.x86_64 --> Processing Dependency: checkpolicy for package: policycoreutils-python-2.5-33.el7.x86_64 --> Processing Dependency: libqpol.so.1()(64bit) for package: policycoreutils-python-2.5-33.el7.x86_64 --> Processing Dependency: libapol.so.4()(64bit) for package: policycoreutils-python-2.5-33.el7.x86_64 --> Running transaction check ---> Package audit-libs-python.x86_64 0:2.8.5-4.el7 will be installed --> Processing Dependency: audit-libs(x86-64) = 2.8.5-4.el7 for package: audit-libs-python-2.8.5-4.el7.x86_64 ---> Package checkpolicy.x86_64 0:2.5-8.el7 will be installed ---> Package libsemanage-python.x86_64 0:2.5-14.el7 will be installed ---> Package policycoreutils.x86_64 0:2.5-29.el7 will be updated ---> Package policycoreutils.x86_64 0:2.5-33.el7 will be an update ---> Package python-IPy.noarch 0:0.75-6.el7 will be installed ---> Package setools-libs.x86_64 0:3.3.8-4.el7 will be installed --> Running transaction check ---> Package audit-libs.x86_64 0:2.8.4-4.el7 will be updated --> Processing Dependency: audit-libs(x86-64) = 2.8.4-4.el7 for package: audit-2.8.4-4.el7.x86_64 ---> Package audit-libs.x86_64 0:2.8.5-4.el7 will be an update --> Running transaction check ---> Package audit.x86_64 0:2.8.4-4.el7 will be updated ---> Package audit.x86_64 0:2.8.5-4.el7 will be an update --> Finished Dependency Resolution Dependencies Resolved ========================================================================================================================================================================================================================================================= Package Arch Version Repository Size ========================================================================================================================================================================================================================================================= Installing: docker-ce x86_64 3:19.03.5-3.el7 docker-ce-stable 24 M Installing for dependencies: audit-libs-python x86_64 2.8.5-4.el7 base 76 k checkpolicy x86_64 2.5-8.el7 base 295 k container-selinux noarch 2:2.107-3.el7 extras 39 k containerd.io x86_64 1.2.10-3.2.el7 docker-ce-stable 23 M docker-ce-cli x86_64 1:19.03.5-3.el7 docker-ce-stable 39 M libcgroup x86_64 0.41-21.el7 base 66 k libsemanage-python x86_64 2.5-14.el7 base 113 k policycoreutils-python x86_64 2.5-33.el7 base 457 k python-IPy noarch 0.75-6.el7 base 32 k setools-libs x86_64 3.3.8-4.el7 base 620 k Updating for dependencies: audit x86_64 2.8.5-4.el7 base 256 k audit-libs x86_64 2.8.5-4.el7 base 102 k policycoreutils x86_64 2.5-33.el7 base 916 k Transaction Summary ========================================================================================================================================================================================================================================================= Install 1 Package (+10 Dependent packages) Upgrade ( 3 Dependent packages) Total download size: 90 M Downloading packages: Delta RPMs disabled because /usr/bin/applydeltarpm not installed. (1/14): audit-libs-2.8.5-4.el7.x86_64.rpm | 102 kB 00:00:00 (2/14): audit-2.8.5-4.el7.x86_64.rpm | 256 kB 00:00:00 (3/14): audit-libs-python-2.8.5-4.el7.x86_64.rpm | 76 kB 00:00:00 (4/14): checkpolicy-2.5-8.el7.x86_64.rpm | 295 kB 00:00:00 (5/14): container-selinux-2.107-3.el7.noarch.rpm | 39 kB 00:00:00 warning: /var/cache/yum/x86_64/7/docker-ce-stable/packages/containerd.io-1.2.10-3.2.el7.x86_64.rpm: Header V4 RSA/SHA512 Signature, key ID 621e9f35: NOKEY ] 3.6 MB/s | 23 MB 00:00:18 ETA Public key for containerd.io-1.2.10-3.2.el7.x86_64.rpm is not installed (6/14): containerd.io-1.2.10-3.2.el7.x86_64.rpm | 23 MB 00:00:02 (7/14): libcgroup-0.41-21.el7.x86_64.rpm | 66 kB 00:00:01 (8/14): libsemanage-python-2.5-14.el7.x86_64.rpm | 113 kB 00:00:03 (9/14): python-IPy-0.75-6.el7.noarch.rpm | 32 kB 00:00:00 (10/14): docker-ce-cli-19.03.5-3.el7.x86_64.rpm | 39 MB 00:00:05 (11/14): setools-libs-3.3.8-4.el7.x86_64.rpm | 620 kB 00:00:00 (12/14): policycoreutils-2.5-33.el7.x86_64.rpm | 916 kB 00:00:03 (13/14): policycoreutils-python-2.5-33.el7.x86_64.rpm | 457 kB 00:00:03 (14/14): docker-ce-19.03.5-3.el7.x86_64.rpm | 24 MB 00:00:09 --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Total 9.4 MB/s | 90 MB 00:00:09 Retrieving key from https://mirrors.aliyun.com/docker-ce/linux/centos/gpg Importing GPG key 0x621E9F35: Userid : "Docker Release (CE rpm) <docker@docker.com>" Fingerprint: 060a 61c5 1b55 8a7f 742b 77aa c52f eb6b 621e 9f35 From : https://mirrors.aliyun.com/docker-ce/linux/centos/gpg Running transaction check Running transaction test Transaction test succeeded Running transaction Updating : audit-libs-2.8.5-4.el7.x86_64 1/17 Updating : policycoreutils-2.5-33.el7.x86_64 2/17 Installing : libcgroup-0.41-21.el7.x86_64 3/17 Installing : audit-libs-python-2.8.5-4.el7.x86_64 4/17 Installing : setools-libs-3.3.8-4.el7.x86_64 5/17 Installing : 1:docker-ce-cli-19.03.5-3.el7.x86_64 6/17 Installing : checkpolicy-2.5-8.el7.x86_64 7/17 Installing : python-IPy-0.75-6.el7.noarch 8/17 Installing : libsemanage-python-2.5-14.el7.x86_64 9/17 Installing : policycoreutils-python-2.5-33.el7.x86_64 10/17 Installing : 2:container-selinux-2.107-3.el7.noarch 11/17 setsebool: SELinux is disabled. Installing : containerd.io-1.2.10-3.2.el7.x86_64 12/17 Installing : 3:docker-ce-19.03.5-3.el7.x86_64 13/17 Updating : audit-2.8.5-4.el7.x86_64 14/17 Cleanup : policycoreutils-2.5-29.el7.x86_64 15/17 Cleanup : audit-2.8.4-4.el7.x86_64 16/17 Cleanup : audit-libs-2.8.4-4.el7.x86_64 17/17 Verifying : audit-libs-2.8.5-4.el7.x86_64 1/17 Verifying : policycoreutils-python-2.5-33.el7.x86_64 2/17 Verifying : audit-2.8.5-4.el7.x86_64 3/17 Verifying : 3:docker-ce-19.03.5-3.el7.x86_64 4/17 Verifying : audit-libs-python-2.8.5-4.el7.x86_64 5/17 Verifying : libsemanage-python-2.5-14.el7.x86_64 6/17 Verifying : 2:container-selinux-2.107-3.el7.noarch 7/17 Verifying : python-IPy-0.75-6.el7.noarch 8/17 Verifying : checkpolicy-2.5-8.el7.x86_64 9/17 Verifying : policycoreutils-2.5-33.el7.x86_64 10/17 Verifying : containerd.io-1.2.10-3.2.el7.x86_64 11/17 Verifying : 1:docker-ce-cli-19.03.5-3.el7.x86_64 12/17 Verifying : setools-libs-3.3.8-4.el7.x86_64 13/17 Verifying : libcgroup-0.41-21.el7.x86_64 14/17 Verifying : policycoreutils-2.5-29.el7.x86_64 15/17 Verifying : audit-2.8.4-4.el7.x86_64 16/17 Verifying : audit-libs-2.8.4-4.el7.x86_64 17/17 Installed: docker-ce.x86_64 3:19.03.5-3.el7 Dependency Installed: audit-libs-python.x86_64 0:2.8.5-4.el7 checkpolicy.x86_64 0:2.5-8.el7 container-selinux.noarch 2:2.107-3.el7 containerd.io.x86_64 0:1.2.10-3.2.el7 docker-ce-cli.x86_64 1:19.03.5-3.el7 libcgroup.x86_64 0:0.41-21.el7 libsemanage-python.x86_64 0:2.5-14.el7 policycoreutils-python.x86_64 0:2.5-33.el7 python-IPy.noarch 0:0.75-6.el7 setools-libs.x86_64 0:3.3.8-4.el7 Dependency Updated: audit.x86_64 0:2.8.5-4.el7 audit-libs.x86_64 0:2.8.5-4.el7 policycoreutils.x86_64 0:2.5-33.el7 Complete! [root@master200.yinzhengjie.org.cn ~]#
[root@node201.yinzhengjie.org.cn ~]# yum -y install docker-ce Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile * base: mirrors.aliyun.com * extras: mirrors.aliyun.com * updates: mirrors.aliyun.com docker-ce-stable | 3.5 kB 00:00:00 (1/2): docker-ce-stable/x86_64/primary_db | 37 kB 00:00:00 (2/2): docker-ce-stable/x86_64/updateinfo | 55 B 00:00:00 Resolving Dependencies --> Running transaction check ---> Package docker-ce.x86_64 3:19.03.5-3.el7 will be installed --> Processing Dependency: container-selinux >= 2:2.74 for package: 3:docker-ce-19.03.5-3.el7.x86_64 --> Processing Dependency: containerd.io >= 1.2.2-3 for package: 3:docker-ce-19.03.5-3.el7.x86_64 --> Processing Dependency: docker-ce-cli for package: 3:docker-ce-19.03.5-3.el7.x86_64 --> Processing Dependency: libcgroup for package: 3:docker-ce-19.03.5-3.el7.x86_64 --> Running transaction check ---> Package container-selinux.noarch 2:2.107-3.el7 will be installed --> Processing Dependency: policycoreutils-python for package: 2:container-selinux-2.107-3.el7.noarch ---> Package containerd.io.x86_64 0:1.2.10-3.2.el7 will be installed ---> Package docker-ce-cli.x86_64 1:19.03.5-3.el7 will be installed ---> Package libcgroup.x86_64 0:0.41-21.el7 will be installed --> Running transaction check ---> Package policycoreutils-python.x86_64 0:2.5-33.el7 will be installed --> Processing Dependency: policycoreutils = 2.5-33.el7 for package: policycoreutils-python-2.5-33.el7.x86_64 --> Processing Dependency: setools-libs >= 3.3.8-4 for package: policycoreutils-python-2.5-33.el7.x86_64 --> Processing Dependency: libsemanage-python >= 2.5-14 for package: policycoreutils-python-2.5-33.el7.x86_64 --> Processing Dependency: audit-libs-python >= 2.1.3-4 for package: policycoreutils-python-2.5-33.el7.x86_64 --> Processing Dependency: python-IPy for package: policycoreutils-python-2.5-33.el7.x86_64 --> Processing Dependency: libqpol.so.1(VERS_1.4)(64bit) for package: policycoreutils-python-2.5-33.el7.x86_64 --> Processing Dependency: libqpol.so.1(VERS_1.2)(64bit) for package: policycoreutils-python-2.5-33.el7.x86_64 --> Processing Dependency: libapol.so.4(VERS_4.0)(64bit) for package: policycoreutils-python-2.5-33.el7.x86_64 --> Processing Dependency: checkpolicy for package: policycoreutils-python-2.5-33.el7.x86_64 --> Processing Dependency: libqpol.so.1()(64bit) for package: policycoreutils-python-2.5-33.el7.x86_64 --> Processing Dependency: libapol.so.4()(64bit) for package: policycoreutils-python-2.5-33.el7.x86_64 --> Running transaction check ---> Package audit-libs-python.x86_64 0:2.8.5-4.el7 will be installed --> Processing Dependency: audit-libs(x86-64) = 2.8.5-4.el7 for package: audit-libs-python-2.8.5-4.el7.x86_64 ---> Package checkpolicy.x86_64 0:2.5-8.el7 will be installed ---> Package libsemanage-python.x86_64 0:2.5-14.el7 will be installed ---> Package policycoreutils.x86_64 0:2.5-29.el7 will be updated ---> Package policycoreutils.x86_64 0:2.5-33.el7 will be an update ---> Package python-IPy.noarch 0:0.75-6.el7 will be installed ---> Package setools-libs.x86_64 0:3.3.8-4.el7 will be installed --> Running transaction check ---> Package audit-libs.x86_64 0:2.8.4-4.el7 will be updated --> Processing Dependency: audit-libs(x86-64) = 2.8.4-4.el7 for package: audit-2.8.4-4.el7.x86_64 ---> Package audit-libs.x86_64 0:2.8.5-4.el7 will be an update --> Running transaction check ---> Package audit.x86_64 0:2.8.4-4.el7 will be updated ---> Package audit.x86_64 0:2.8.5-4.el7 will be an update --> Finished Dependency Resolution Dependencies Resolved ========================================================================================================================================================================================================================================================= Package Arch Version Repository Size ========================================================================================================================================================================================================================================================= Installing: docker-ce x86_64 3:19.03.5-3.el7 docker-ce-stable 24 M Installing for dependencies: audit-libs-python x86_64 2.8.5-4.el7 base 76 k checkpolicy x86_64 2.5-8.el7 base 295 k container-selinux noarch 2:2.107-3.el7 extras 39 k containerd.io x86_64 1.2.10-3.2.el7 docker-ce-stable 23 M docker-ce-cli x86_64 1:19.03.5-3.el7 docker-ce-stable 39 M libcgroup x86_64 0.41-21.el7 base 66 k libsemanage-python x86_64 2.5-14.el7 base 113 k policycoreutils-python x86_64 2.5-33.el7 base 457 k python-IPy noarch 0.75-6.el7 base 32 k setools-libs x86_64 3.3.8-4.el7 base 620 k Updating for dependencies: audit x86_64 2.8.5-4.el7 base 256 k audit-libs x86_64 2.8.5-4.el7 base 102 k policycoreutils x86_64 2.5-33.el7 base 916 k Transaction Summary ========================================================================================================================================================================================================================================================= Install 1 Package (+10 Dependent packages) Upgrade ( 3 Dependent packages) Total download size: 90 M Downloading packages: Delta RPMs disabled because /usr/bin/applydeltarpm not installed. (1/14): audit-libs-2.8.5-4.el7.x86_64.rpm | 102 kB 00:00:00 (2/14): audit-libs-python-2.8.5-4.el7.x86_64.rpm | 76 kB 00:00:00 (3/14): audit-2.8.5-4.el7.x86_64.rpm | 256 kB 00:00:00 (4/14): checkpolicy-2.5-8.el7.x86_64.rpm | 295 kB 00:00:00 (5/14): container-selinux-2.107-3.el7.noarch.rpm | 39 kB 00:00:00 warning: /var/cache/yum/x86_64/7/docker-ce-stable/packages/containerd.io-1.2.10-3.2.el7.x86_64.rpm: Header V4 RSA/SHA512 Signature, key ID 621e9f35: NOKEY ] 4.7 MB/s | 22 MB 00:00:14 ETA Public key for containerd.io-1.2.10-3.2.el7.x86_64.rpm is not installed (6/14): containerd.io-1.2.10-3.2.el7.x86_64.rpm | 23 MB 00:00:03 (7/14): libcgroup-0.41-21.el7.x86_64.rpm | 66 kB 00:00:00 (8/14): libsemanage-python-2.5-14.el7.x86_64.rpm | 113 kB 00:00:00 (9/14): python-IPy-0.75-6.el7.noarch.rpm | 32 kB 00:00:00 (10/14): docker-ce-cli-19.03.5-3.el7.x86_64.rpm | 39 MB 00:00:04 (11/14): setools-libs-3.3.8-4.el7.x86_64.rpm | 620 kB 00:00:03 (12/14): policycoreutils-2.5-33.el7.x86_64.rpm | 916 kB 00:00:06 (13/14): policycoreutils-python-2.5-33.el7.x86_64.rpm | 457 kB 00:00:08 (14/14): docker-ce-19.03.5-3.el7.x86_64.rpm | 24 MB 00:00:20 --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Total 4.3 MB/s | 90 MB 00:00:20 Retrieving key from https://mirrors.aliyun.com/docker-ce/linux/centos/gpg Importing GPG key 0x621E9F35: Userid : "Docker Release (CE rpm) <docker@docker.com>" Fingerprint: 060a 61c5 1b55 8a7f 742b 77aa c52f eb6b 621e 9f35 From : https://mirrors.aliyun.com/docker-ce/linux/centos/gpg Running transaction check Running transaction test Transaction test succeeded Running transaction Updating : audit-libs-2.8.5-4.el7.x86_64 1/17 Updating : policycoreutils-2.5-33.el7.x86_64 2/17 Installing : libcgroup-0.41-21.el7.x86_64 3/17 Installing : audit-libs-python-2.8.5-4.el7.x86_64 4/17 Installing : setools-libs-3.3.8-4.el7.x86_64 5/17 Installing : 1:docker-ce-cli-19.03.5-3.el7.x86_64 6/17 Installing : checkpolicy-2.5-8.el7.x86_64 7/17 Installing : python-IPy-0.75-6.el7.noarch 8/17 Installing : libsemanage-python-2.5-14.el7.x86_64 9/17 Installing : policycoreutils-python-2.5-33.el7.x86_64 10/17 Installing : 2:container-selinux-2.107-3.el7.noarch 11/17 setsebool: SELinux is disabled. Installing : containerd.io-1.2.10-3.2.el7.x86_64 12/17 Installing : 3:docker-ce-19.03.5-3.el7.x86_64 13/17 Updating : audit-2.8.5-4.el7.x86_64 14/17 Cleanup : policycoreutils-2.5-29.el7.x86_64 15/17 Cleanup : audit-2.8.4-4.el7.x86_64 16/17 Cleanup : audit-libs-2.8.4-4.el7.x86_64 17/17 Verifying : audit-libs-2.8.5-4.el7.x86_64 1/17 Verifying : policycoreutils-python-2.5-33.el7.x86_64 2/17 Verifying : audit-2.8.5-4.el7.x86_64 3/17 Verifying : 3:docker-ce-19.03.5-3.el7.x86_64 4/17 Verifying : audit-libs-python-2.8.5-4.el7.x86_64 5/17 Verifying : libsemanage-python-2.5-14.el7.x86_64 6/17 Verifying : 2:container-selinux-2.107-3.el7.noarch 7/17 Verifying : python-IPy-0.75-6.el7.noarch 8/17 Verifying : checkpolicy-2.5-8.el7.x86_64 9/17 Verifying : policycoreutils-2.5-33.el7.x86_64 10/17 Verifying : containerd.io-1.2.10-3.2.el7.x86_64 11/17 Verifying : 1:docker-ce-cli-19.03.5-3.el7.x86_64 12/17 Verifying : setools-libs-3.3.8-4.el7.x86_64 13/17 Verifying : libcgroup-0.41-21.el7.x86_64 14/17 Verifying : policycoreutils-2.5-29.el7.x86_64 15/17 Verifying : audit-2.8.4-4.el7.x86_64 16/17 Verifying : audit-libs-2.8.4-4.el7.x86_64 17/17 Installed: docker-ce.x86_64 3:19.03.5-3.el7 Dependency Installed: audit-libs-python.x86_64 0:2.8.5-4.el7 checkpolicy.x86_64 0:2.5-8.el7 container-selinux.noarch 2:2.107-3.el7 containerd.io.x86_64 0:1.2.10-3.2.el7 docker-ce-cli.x86_64 1:19.03.5-3.el7 libcgroup.x86_64 0:0.41-21.el7 libsemanage-python.x86_64 0:2.5-14.el7 policycoreutils-python.x86_64 0:2.5-33.el7 python-IPy.noarch 0:0.75-6.el7 setools-libs.x86_64 0:3.3.8-4.el7 Dependency Updated: audit.x86_64 0:2.8.5-4.el7 audit-libs.x86_64 0:2.8.5-4.el7 policycoreutils.x86_64 0:2.5-33.el7 Complete! [root@node201.yinzhengjie.org.cn ~]#
[root@node202.yinzhengjie.org.cn ~]# yum -y install docker-ce Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile * base: mirror.bit.edu.cn * extras: mirrors.aliyun.com * updates: mirrors.aliyun.com docker-ce-stable | 3.5 kB 00:00:00 (1/2): docker-ce-stable/x86_64/updateinfo | 55 B 00:00:00 (2/2): docker-ce-stable/x86_64/primary_db | 37 kB 00:00:00 Resolving Dependencies --> Running transaction check ---> Package docker-ce.x86_64 3:19.03.5-3.el7 will be installed --> Processing Dependency: container-selinux >= 2:2.74 for package: 3:docker-ce-19.03.5-3.el7.x86_64 --> Processing Dependency: containerd.io >= 1.2.2-3 for package: 3:docker-ce-19.03.5-3.el7.x86_64 --> Processing Dependency: docker-ce-cli for package: 3:docker-ce-19.03.5-3.el7.x86_64 --> Processing Dependency: libcgroup for package: 3:docker-ce-19.03.5-3.el7.x86_64 --> Running transaction check ---> Package container-selinux.noarch 2:2.107-3.el7 will be installed --> Processing Dependency: policycoreutils-python for package: 2:container-selinux-2.107-3.el7.noarch ---> Package containerd.io.x86_64 0:1.2.10-3.2.el7 will be installed ---> Package docker-ce-cli.x86_64 1:19.03.5-3.el7 will be installed ---> Package libcgroup.x86_64 0:0.41-21.el7 will be installed --> Running transaction check ---> Package policycoreutils-python.x86_64 0:2.5-33.el7 will be installed --> Processing Dependency: policycoreutils = 2.5-33.el7 for package: policycoreutils-python-2.5-33.el7.x86_64 --> Processing Dependency: setools-libs >= 3.3.8-4 for package: policycoreutils-python-2.5-33.el7.x86_64 --> Processing Dependency: libsemanage-python >= 2.5-14 for package: policycoreutils-python-2.5-33.el7.x86_64 --> Processing Dependency: audit-libs-python >= 2.1.3-4 for package: policycoreutils-python-2.5-33.el7.x86_64 --> Processing Dependency: python-IPy for package: policycoreutils-python-2.5-33.el7.x86_64 --> Processing Dependency: libqpol.so.1(VERS_1.4)(64bit) for package: policycoreutils-python-2.5-33.el7.x86_64 --> Processing Dependency: libqpol.so.1(VERS_1.2)(64bit) for package: policycoreutils-python-2.5-33.el7.x86_64 --> Processing Dependency: libapol.so.4(VERS_4.0)(64bit) for package: policycoreutils-python-2.5-33.el7.x86_64 --> Processing Dependency: checkpolicy for package: policycoreutils-python-2.5-33.el7.x86_64 --> Processing Dependency: libqpol.so.1()(64bit) for package: policycoreutils-python-2.5-33.el7.x86_64 --> Processing Dependency: libapol.so.4()(64bit) for package: policycoreutils-python-2.5-33.el7.x86_64 --> Running transaction check ---> Package audit-libs-python.x86_64 0:2.8.5-4.el7 will be installed --> Processing Dependency: audit-libs(x86-64) = 2.8.5-4.el7 for package: audit-libs-python-2.8.5-4.el7.x86_64 ---> Package checkpolicy.x86_64 0:2.5-8.el7 will be installed ---> Package libsemanage-python.x86_64 0:2.5-14.el7 will be installed ---> Package policycoreutils.x86_64 0:2.5-29.el7 will be updated ---> Package policycoreutils.x86_64 0:2.5-33.el7 will be an update ---> Package python-IPy.noarch 0:0.75-6.el7 will be installed ---> Package setools-libs.x86_64 0:3.3.8-4.el7 will be installed --> Running transaction check ---> Package audit-libs.x86_64 0:2.8.4-4.el7 will be updated --> Processing Dependency: audit-libs(x86-64) = 2.8.4-4.el7 for package: audit-2.8.4-4.el7.x86_64 ---> Package audit-libs.x86_64 0:2.8.5-4.el7 will be an update --> Running transaction check ---> Package audit.x86_64 0:2.8.4-4.el7 will be updated ---> Package audit.x86_64 0:2.8.5-4.el7 will be an update --> Finished Dependency Resolution Dependencies Resolved ========================================================================================================================================================================================================================================================= Package Arch Version Repository Size ========================================================================================================================================================================================================================================================= Installing: docker-ce x86_64 3:19.03.5-3.el7 docker-ce-stable 24 M Installing for dependencies: audit-libs-python x86_64 2.8.5-4.el7 base 76 k checkpolicy x86_64 2.5-8.el7 base 295 k container-selinux noarch 2:2.107-3.el7 extras 39 k containerd.io x86_64 1.2.10-3.2.el7 docker-ce-stable 23 M docker-ce-cli x86_64 1:19.03.5-3.el7 docker-ce-stable 39 M libcgroup x86_64 0.41-21.el7 base 66 k libsemanage-python x86_64 2.5-14.el7 base 113 k policycoreutils-python x86_64 2.5-33.el7 base 457 k python-IPy noarch 0.75-6.el7 base 32 k setools-libs x86_64 3.3.8-4.el7 base 620 k Updating for dependencies: audit x86_64 2.8.5-4.el7 base 256 k audit-libs x86_64 2.8.5-4.el7 base 102 k policycoreutils x86_64 2.5-33.el7 base 916 k Transaction Summary ========================================================================================================================================================================================================================================================= Install 1 Package (+10 Dependent packages) Upgrade ( 3 Dependent packages) Total download size: 90 M Downloading packages: Delta RPMs disabled because /usr/bin/applydeltarpm not installed. (1/14): audit-libs-2.8.5-4.el7.x86_64.rpm | 102 kB 00:00:01 (2/14): audit-2.8.5-4.el7.x86_64.rpm | 256 kB 00:00:01 (3/14): container-selinux-2.107-3.el7.noarch.rpm | 39 kB 00:00:00 (4/14): audit-libs-python-2.8.5-4.el7.x86_64.rpm | 76 kB 00:00:01 (5/14): checkpolicy-2.5-8.el7.x86_64.rpm | 295 kB 00:00:03 warning: /var/cache/yum/x86_64/7/docker-ce-stable/packages/containerd.io-1.2.10-3.2.el7.x86_64.rpm: Header V4 RSA/SHA512 Signature, key ID 621e9f35: NOKEY ] 3.8 MB/s | 34 MB 00:00:14 ETA Public key for containerd.io-1.2.10-3.2.el7.x86_64.rpm is not installed (6/14): containerd.io-1.2.10-3.2.el7.x86_64.rpm | 23 MB 00:00:15 (7/14): libcgroup-0.41-21.el7.x86_64.rpm | 66 kB 00:00:00 (8/14): libsemanage-python-2.5-14.el7.x86_64.rpm | 113 kB 00:00:00 (9/14): python-IPy-0.75-6.el7.noarch.rpm | 32 kB 00:00:00 (10/14): policycoreutils-python-2.5-33.el7.x86_64.rpm | 457 kB 00:00:00 (11/14): setools-libs-3.3.8-4.el7.x86_64.rpm | 620 kB 00:00:00 (12/14): policycoreutils-2.5-33.el7.x86_64.rpm | 916 kB 00:00:02 (13/14): docker-ce-19.03.5-3.el7.x86_64.rpm | 24 MB 00:00:21 (14/14): docker-ce-cli-19.03.5-3.el7.x86_64.rpm | 39 MB 00:00:07 --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Total 3.7 MB/s | 90 MB 00:00:24 Retrieving key from https://mirrors.aliyun.com/docker-ce/linux/centos/gpg Importing GPG key 0x621E9F35: Userid : "Docker Release (CE rpm) <docker@docker.com>" Fingerprint: 060a 61c5 1b55 8a7f 742b 77aa c52f eb6b 621e 9f35 From : https://mirrors.aliyun.com/docker-ce/linux/centos/gpg Running transaction check Running transaction test Transaction test succeeded Running transaction Updating : audit-libs-2.8.5-4.el7.x86_64 1/17 Updating : policycoreutils-2.5-33.el7.x86_64 2/17 Installing : libcgroup-0.41-21.el7.x86_64 3/17 Installing : audit-libs-python-2.8.5-4.el7.x86_64 4/17 Installing : setools-libs-3.3.8-4.el7.x86_64 5/17 Installing : 1:docker-ce-cli-19.03.5-3.el7.x86_64 6/17 Installing : checkpolicy-2.5-8.el7.x86_64 7/17 Installing : python-IPy-0.75-6.el7.noarch 8/17 Installing : libsemanage-python-2.5-14.el7.x86_64 9/17 Installing : policycoreutils-python-2.5-33.el7.x86_64 10/17 Installing : 2:container-selinux-2.107-3.el7.noarch 11/17 setsebool: SELinux is disabled. Installing : containerd.io-1.2.10-3.2.el7.x86_64 12/17 Installing : 3:docker-ce-19.03.5-3.el7.x86_64 13/17 Updating : audit-2.8.5-4.el7.x86_64 14/17 Cleanup : policycoreutils-2.5-29.el7.x86_64 15/17 Cleanup : audit-2.8.4-4.el7.x86_64 16/17 Cleanup : audit-libs-2.8.4-4.el7.x86_64 17/17 Verifying : audit-libs-2.8.5-4.el7.x86_64 1/17 Verifying : policycoreutils-python-2.5-33.el7.x86_64 2/17 Verifying : audit-2.8.5-4.el7.x86_64 3/17 Verifying : 3:docker-ce-19.03.5-3.el7.x86_64 4/17 Verifying : audit-libs-python-2.8.5-4.el7.x86_64 5/17 Verifying : libsemanage-python-2.5-14.el7.x86_64 6/17 Verifying : 2:container-selinux-2.107-3.el7.noarch 7/17 Verifying : python-IPy-0.75-6.el7.noarch 8/17 Verifying : checkpolicy-2.5-8.el7.x86_64 9/17 Verifying : policycoreutils-2.5-33.el7.x86_64 10/17 Verifying : containerd.io-1.2.10-3.2.el7.x86_64 11/17 Verifying : 1:docker-ce-cli-19.03.5-3.el7.x86_64 12/17 Verifying : setools-libs-3.3.8-4.el7.x86_64 13/17 Verifying : libcgroup-0.41-21.el7.x86_64 14/17 Verifying : policycoreutils-2.5-29.el7.x86_64 15/17 Verifying : audit-2.8.4-4.el7.x86_64 16/17 Verifying : audit-libs-2.8.4-4.el7.x86_64 17/17 Installed: docker-ce.x86_64 3:19.03.5-3.el7 Dependency Installed: audit-libs-python.x86_64 0:2.8.5-4.el7 checkpolicy.x86_64 0:2.5-8.el7 container-selinux.noarch 2:2.107-3.el7 containerd.io.x86_64 0:1.2.10-3.2.el7 docker-ce-cli.x86_64 1:19.03.5-3.el7 libcgroup.x86_64 0:0.41-21.el7 libsemanage-python.x86_64 0:2.5-14.el7 policycoreutils-python.x86_64 0:2.5-33.el7 python-IPy.noarch 0:0.75-6.el7 setools-libs.x86_64 0:3.3.8-4.el7 Dependency Updated: audit.x86_64 0:2.8.5-4.el7 audit-libs.x86_64 0:2.8.5-4.el7 policycoreutils.x86_64 0:2.5-33.el7 Complete! [root@node202.yinzhengjie.org.cn ~]#
[root@node203.yinzhengjie.org.cn ~]# yum -y install docker-ce Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile * base: mirrors.aliyun.com * extras: mirrors.aliyun.com * updates: mirrors.aliyun.com docker-ce-stable | 3.5 kB 00:00:00 (1/2): docker-ce-stable/x86_64/updateinfo | 55 B 00:00:00 (2/2): docker-ce-stable/x86_64/primary_db | 37 kB 00:00:01 Resolving Dependencies --> Running transaction check ---> Package docker-ce.x86_64 3:19.03.5-3.el7 will be installed --> Processing Dependency: container-selinux >= 2:2.74 for package: 3:docker-ce-19.03.5-3.el7.x86_64 --> Processing Dependency: containerd.io >= 1.2.2-3 for package: 3:docker-ce-19.03.5-3.el7.x86_64 --> Processing Dependency: docker-ce-cli for package: 3:docker-ce-19.03.5-3.el7.x86_64 --> Processing Dependency: libcgroup for package: 3:docker-ce-19.03.5-3.el7.x86_64 --> Running transaction check ---> Package container-selinux.noarch 2:2.107-3.el7 will be installed --> Processing Dependency: policycoreutils-python for package: 2:container-selinux-2.107-3.el7.noarch ---> Package containerd.io.x86_64 0:1.2.10-3.2.el7 will be installed ---> Package docker-ce-cli.x86_64 1:19.03.5-3.el7 will be installed ---> Package libcgroup.x86_64 0:0.41-21.el7 will be installed --> Running transaction check ---> Package policycoreutils-python.x86_64 0:2.5-33.el7 will be installed --> Processing Dependency: policycoreutils = 2.5-33.el7 for package: policycoreutils-python-2.5-33.el7.x86_64 --> Processing Dependency: setools-libs >= 3.3.8-4 for package: policycoreutils-python-2.5-33.el7.x86_64 --> Processing Dependency: libsemanage-python >= 2.5-14 for package: policycoreutils-python-2.5-33.el7.x86_64 --> Processing Dependency: audit-libs-python >= 2.1.3-4 for package: policycoreutils-python-2.5-33.el7.x86_64 --> Processing Dependency: python-IPy for package: policycoreutils-python-2.5-33.el7.x86_64 --> Processing Dependency: libqpol.so.1(VERS_1.4)(64bit) for package: policycoreutils-python-2.5-33.el7.x86_64 --> Processing Dependency: libqpol.so.1(VERS_1.2)(64bit) for package: policycoreutils-python-2.5-33.el7.x86_64 --> Processing Dependency: libapol.so.4(VERS_4.0)(64bit) for package: policycoreutils-python-2.5-33.el7.x86_64 --> Processing Dependency: checkpolicy for package: policycoreutils-python-2.5-33.el7.x86_64 --> Processing Dependency: libqpol.so.1()(64bit) for package: policycoreutils-python-2.5-33.el7.x86_64 --> Processing Dependency: libapol.so.4()(64bit) for package: policycoreutils-python-2.5-33.el7.x86_64 --> Running transaction check ---> Package audit-libs-python.x86_64 0:2.8.5-4.el7 will be installed --> Processing Dependency: audit-libs(x86-64) = 2.8.5-4.el7 for package: audit-libs-python-2.8.5-4.el7.x86_64 ---> Package checkpolicy.x86_64 0:2.5-8.el7 will be installed ---> Package libsemanage-python.x86_64 0:2.5-14.el7 will be installed ---> Package policycoreutils.x86_64 0:2.5-29.el7 will be updated ---> Package policycoreutils.x86_64 0:2.5-33.el7 will be an update ---> Package python-IPy.noarch 0:0.75-6.el7 will be installed ---> Package setools-libs.x86_64 0:3.3.8-4.el7 will be installed --> Running transaction check ---> Package audit-libs.x86_64 0:2.8.4-4.el7 will be updated --> Processing Dependency: audit-libs(x86-64) = 2.8.4-4.el7 for package: audit-2.8.4-4.el7.x86_64 ---> Package audit-libs.x86_64 0:2.8.5-4.el7 will be an update --> Running transaction check ---> Package audit.x86_64 0:2.8.4-4.el7 will be updated ---> Package audit.x86_64 0:2.8.5-4.el7 will be an update --> Finished Dependency Resolution Dependencies Resolved ========================================================================================================================================================================================================================================================= Package Arch Version Repository Size ========================================================================================================================================================================================================================================================= Installing: docker-ce x86_64 3:19.03.5-3.el7 docker-ce-stable 24 M Installing for dependencies: audit-libs-python x86_64 2.8.5-4.el7 base 76 k checkpolicy x86_64 2.5-8.el7 base 295 k container-selinux noarch 2:2.107-3.el7 extras 39 k containerd.io x86_64 1.2.10-3.2.el7 docker-ce-stable 23 M docker-ce-cli x86_64 1:19.03.5-3.el7 docker-ce-stable 39 M libcgroup x86_64 0.41-21.el7 base 66 k libsemanage-python x86_64 2.5-14.el7 base 113 k policycoreutils-python x86_64 2.5-33.el7 base 457 k python-IPy noarch 0.75-6.el7 base 32 k setools-libs x86_64 3.3.8-4.el7 base 620 k Updating for dependencies: audit x86_64 2.8.5-4.el7 base 256 k audit-libs x86_64 2.8.5-4.el7 base 102 k policycoreutils x86_64 2.5-33.el7 base 916 k Transaction Summary ========================================================================================================================================================================================================================================================= Install 1 Package (+10 Dependent packages) Upgrade ( 3 Dependent packages) Total download size: 90 M Downloading packages: Delta RPMs disabled because /usr/bin/applydeltarpm not installed. (1/14): audit-libs-2.8.5-4.el7.x86_64.rpm | 102 kB 00:00:00 (2/14): audit-libs-python-2.8.5-4.el7.x86_64.rpm | 76 kB 00:00:00 (3/14): container-selinux-2.107-3.el7.noarch.rpm | 39 kB 00:00:00 (4/14): audit-2.8.5-4.el7.x86_64.rpm | 256 kB 00:00:01 (5/14): checkpolicy-2.5-8.el7.x86_64.rpm | 295 kB 00:00:00 warning: /var/cache/yum/x86_64/7/docker-ce-stable/packages/docker-ce-19.03.5-3.el7.x86_64.rpm: Header V4 RSA/SHA512 Signature, key ID 621e9f35: NOKEY ] 3.8 MB/s | 24 MB 00:00:17 ETA Public key for docker-ce-19.03.5-3.el7.x86_64.rpm is not installed (6/14): docker-ce-19.03.5-3.el7.x86_64.rpm | 24 MB 00:00:03 (7/14): libcgroup-0.41-21.el7.x86_64.rpm | 66 kB 00:00:00 (8/14): libsemanage-python-2.5-14.el7.x86_64.rpm | 113 kB 00:00:00 (9/14): python-IPy-0.75-6.el7.noarch.rpm | 32 kB 00:00:00 (10/14): policycoreutils-python-2.5-33.el7.x86_64.rpm | 457 kB 00:00:01 (11/14): policycoreutils-2.5-33.el7.x86_64.rpm | 916 kB 00:00:01 (12/14): setools-libs-3.3.8-4.el7.x86_64.rpm | 620 kB 00:00:01 (13/14): docker-ce-cli-19.03.5-3.el7.x86_64.rpm | 39 MB 00:00:08 (14/14): containerd.io-1.2.10-3.2.el7.x86_64.rpm | 23 MB 00:00:19 --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Total 4.5 MB/s | 90 MB 00:00:20 Retrieving key from https://mirrors.aliyun.com/docker-ce/linux/centos/gpg Importing GPG key 0x621E9F35: Userid : "Docker Release (CE rpm) <docker@docker.com>" Fingerprint: 060a 61c5 1b55 8a7f 742b 77aa c52f eb6b 621e 9f35 From : https://mirrors.aliyun.com/docker-ce/linux/centos/gpg Running transaction check Running transaction test Transaction test succeeded Running transaction Updating : audit-libs-2.8.5-4.el7.x86_64 1/17 Updating : policycoreutils-2.5-33.el7.x86_64 2/17 Installing : libcgroup-0.41-21.el7.x86_64 3/17 Installing : audit-libs-python-2.8.5-4.el7.x86_64 4/17 Installing : setools-libs-3.3.8-4.el7.x86_64 5/17 Installing : 1:docker-ce-cli-19.03.5-3.el7.x86_64 6/17 Installing : checkpolicy-2.5-8.el7.x86_64 7/17 Installing : python-IPy-0.75-6.el7.noarch 8/17 Installing : libsemanage-python-2.5-14.el7.x86_64 9/17 Installing : policycoreutils-python-2.5-33.el7.x86_64 10/17 Installing : 2:container-selinux-2.107-3.el7.noarch 11/17 setsebool: SELinux is disabled. Installing : containerd.io-1.2.10-3.2.el7.x86_64 12/17 Installing : 3:docker-ce-19.03.5-3.el7.x86_64 13/17 Updating : audit-2.8.5-4.el7.x86_64 14/17 Cleanup : policycoreutils-2.5-29.el7.x86_64 15/17 Cleanup : audit-2.8.4-4.el7.x86_64 16/17 Cleanup : audit-libs-2.8.4-4.el7.x86_64 17/17 Verifying : audit-libs-2.8.5-4.el7.x86_64 1/17 Verifying : policycoreutils-python-2.5-33.el7.x86_64 2/17 Verifying : audit-2.8.5-4.el7.x86_64 3/17 Verifying : 3:docker-ce-19.03.5-3.el7.x86_64 4/17 Verifying : audit-libs-python-2.8.5-4.el7.x86_64 5/17 Verifying : libsemanage-python-2.5-14.el7.x86_64 6/17 Verifying : 2:container-selinux-2.107-3.el7.noarch 7/17 Verifying : python-IPy-0.75-6.el7.noarch 8/17 Verifying : checkpolicy-2.5-8.el7.x86_64 9/17 Verifying : policycoreutils-2.5-33.el7.x86_64 10/17 Verifying : containerd.io-1.2.10-3.2.el7.x86_64 11/17 Verifying : 1:docker-ce-cli-19.03.5-3.el7.x86_64 12/17 Verifying : setools-libs-3.3.8-4.el7.x86_64 13/17 Verifying : libcgroup-0.41-21.el7.x86_64 14/17 Verifying : policycoreutils-2.5-29.el7.x86_64 15/17 Verifying : audit-2.8.4-4.el7.x86_64 16/17 Verifying : audit-libs-2.8.4-4.el7.x86_64 17/17 Installed: docker-ce.x86_64 3:19.03.5-3.el7 Dependency Installed: audit-libs-python.x86_64 0:2.8.5-4.el7 checkpolicy.x86_64 0:2.5-8.el7 container-selinux.noarch 2:2.107-3.el7 containerd.io.x86_64 0:1.2.10-3.2.el7 docker-ce-cli.x86_64 1:19.03.5-3.el7 libcgroup.x86_64 0:0.41-21.el7 libsemanage-python.x86_64 0:2.5-14.el7 policycoreutils-python.x86_64 0:2.5-33.el7 python-IPy.noarch 0:0.75-6.el7 setools-libs.x86_64 0:3.3.8-4.el7 Dependency Updated: audit.x86_64 0:2.8.5-4.el7 audit-libs.x86_64 0:2.8.5-4.el7 policycoreutils.x86_64 0:2.5-33.el7 Complete! [root@node203.yinzhengjie.org.cn ~]#
3>.修改docker的默認策略,重新將FORWARD鏈的默認策略設置為ACCEPT
docker自1.13版起會自動設置iptable的FORWARD默認策略為DROP，這可能會影響kubernetes集群依賴的報文轉發功能,因此，需要在docker服務啟動后，重新將FORWARD鏈的默認策略設置為ACCEPT。 如下圖所示,修改"/usr/lib/systemd/system/docker.service"文件，在"ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock"一行之后新增一行"ExecStartPost=/usr/sbin/iptables -P FORWARD ACCEPT"(意思是docker服務器啟動成功后會執行執行該命令)
4>.為docker設置代理(一旦我們為docker服務器設置代理后再去配置阿里雲加速就沒有多大意義了,因此我這里並沒有配置鏡像加速，因為默認會走代理服務器)
若要通過默認的k8s.gcr.io鏡像倉庫(實際上國內的網絡是無法直接訪問Google的資源)獲取kubernetes系統組件的相關鏡像,需要配置docker Unit File(/usr/lib/systemd/system/docker.service)中的Environment變量,為其定義合理的HTTPS_PROXY(說白了就是你自己的VPN服務器，如果沒有VPN的話還是乖乖的使用阿里雲鏡像吧),當然我們訪問跟本地網絡或者node的IP時則無需代理，使用NO_PROXY指定不代理的網段，格式如下(具體配置參考下圖): Environment="HTTPS_PROXY=PROTOCOL://HOST:PORT" Environment="NO_PROXY=127.0.0.0/8,172.200.0.0/21" 溫馨提示: 如果你不想通過k8s.gcr.io鏡像倉庫下載那就可以直接忽略此步驟，因為阿里雲已經幫咱們提前下載好k8s鏡像啦，我們直接去阿里雲下載即可，使用阿里雲下載需要配置阿里雲加速，具體步驟可參考:"https://www.cnblogs.com/yinzhengjie/p/12182645.html"
5>.啟動docker
[root@master200.yinzhengjie.org.cn ~]# systemctl daemon-reload [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# systemctl start docker.service [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# systemctl enable docker.service Created symlink from /etc/systemd/system/multi-user.target.wants/docker.service to /usr/lib/systemd/system/docker.service. [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# systemctl list-unit-files | grep docker docker.service enabled docker.socket disabled [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# systemctl status docker.service ● docker.service - Docker Application Container Engine Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled) Active: active (running) since Tue 2020-02-04 11:30:27 CST; 1min 4s ago Docs: https://docs.docker.com Main PID: 6182 (dockerd) CGroup: /system.slice/docker.service └─6182 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock Feb 04 11:30:27 master200.yinzhengjie.org.cn dockerd[6182]: time="2020-02-04T11:30:27.706191791+08:00" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc Feb 04 11:30:27 master200.yinzhengjie.org.cn dockerd[6182]: time="2020-02-04T11:30:27.706203607+08:00" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///run/containerd/containerd.sock 0 <nil>}] <nil>}" module=grpc Feb 04 11:30:27 master200.yinzhengjie.org.cn dockerd[6182]: time="2020-02-04T11:30:27.706210087+08:00" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc Feb 04 11:30:27 master200.yinzhengjie.org.cn dockerd[6182]: time="2020-02-04T11:30:27.732793779+08:00" level=info msg="Loading containers: start." Feb 04 11:30:27 master200.yinzhengjie.org.cn dockerd[6182]: time="2020-02-04T11:30:27.819420862+08:00" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a...rred IP address" Feb 04 11:30:27 master200.yinzhengjie.org.cn dockerd[6182]: time="2020-02-04T11:30:27.858229542+08:00" level=info msg="Loading containers: done." Feb 04 11:30:27 master200.yinzhengjie.org.cn dockerd[6182]: time="2020-02-04T11:30:27.871357290+08:00" level=info msg="Docker daemon" commit=633a0ea graphdriver(s)=overlay2 version=19.03.5 Feb 04 11:30:27 master200.yinzhengjie.org.cn dockerd[6182]: time="2020-02-04T11:30:27.871433828+08:00" level=info msg="Daemon has completed initialization" Feb 04 11:30:27 master200.yinzhengjie.org.cn dockerd[6182]: time="2020-02-04T11:30:27.893618790+08:00" level=info msg="API listen on /var/run/docker.sock" Feb 04 11:30:27 master200.yinzhengjie.org.cn systemd[1]: Started Docker Application Container Engine. Hint: Some lines were ellipsized, use -l to show in full. [root@master200.yinzhengjie.org.cn ~]#
6>.查看docker的代理信息
[root@master200.yinzhengjie.org.cn ~]# docker info Client: Debug Mode: false Server: Containers: 0 Running: 0 Paused: 0 Stopped: 0 Images: 0 Server Version: 19.03.5 Storage Driver: overlay2 Backing Filesystem: xfs Supports d_type: true Native Overlay Diff: true Logging Driver: json-file Cgroup Driver: cgroupfs Plugins: Volume: local Network: bridge host ipvlan macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog Swarm: inactive Runtimes: runc Default Runtime: runc Init Binary: docker-init containerd version: b34a5c8af56e510852c35414db4c1f4fa6172339 runc version: 3e425f80a8c931f88e6d94a8c831b9d5aa481657 init version: fec3683 Security Options: seccomp Profile: default Kernel Version: 3.10.0-957.el7.x86_64 Operating System: CentOS Linux 7 (Core) OSType: linux Architecture: x86_64 CPUs: 2 Total Memory: 3.84GiB Name: master200.yinzhengjie.org.cn ID: CSHT:CVB3:JFYB:GQ77:FYDH:X3UJ:B2SH:5WEX:5QNE:NABW:MJZM:K2T2 Docker Root Dir: /var/lib/docker Debug Mode: false HTTPS Proxy: http://www.yinzhengjie.org.cn:10086 No Proxy: 127.0.0.0/8,172.200.0.0/21 Registry: https://index.docker.io/v1/ Labels: Experimental: false Insecure Registries: 127.0.0.0/8 Live Restore Enabled: false [root@master200.yinzhengjie.org.cn ~]#
7>.查看防火牆FORWARD鏈的默認策略
[root@master200.yinzhengjie.org.cn ~]# iptables -vnL Chain INPUT (policy ACCEPT 247 packets, 17304 bytes) pkts bytes target prot opt in out source destination Chain FORWARD (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination 0 0 DOCKER-USER all -- * * 0.0.0.0/0 0.0.0.0/0 0 0 DOCKER-ISOLATION-STAGE-1 all -- * * 0.0.0.0/0 0.0.0.0/0 0 0 ACCEPT all -- * docker0 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED 0 0 DOCKER all -- * docker0 0.0.0.0/0 0.0.0.0/0 0 0 ACCEPT all -- docker0 !docker0 0.0.0.0/0 0.0.0.0/0 0 0 ACCEPT all -- docker0 docker0 0.0.0.0/0 0.0.0.0/0 Chain OUTPUT (policy ACCEPT 147 packets, 18292 bytes) pkts bytes target prot opt in out source destination Chain DOCKER (1 references) pkts bytes target prot opt in out source destination Chain DOCKER-ISOLATION-STAGE-1 (1 references) pkts bytes target prot opt in out source destination 0 0 DOCKER-ISOLATION-STAGE-2 all -- docker0 !docker0 0.0.0.0/0 0.0.0.0/0 0 0 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0 Chain DOCKER-ISOLATION-STAGE-2 (1 references) pkts bytes target prot opt in out source destination 0 0 DROP all -- * docker0 0.0.0.0/0 0.0.0.0/0 0 0 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0 Chain DOCKER-USER (1 references) pkts bytes target prot opt in out source destination 0 0 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0 [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]#
8>.將master200的docker文件拷貝到其它節點啟動docker並設置為開機自啟動
[root@master200.yinzhengjie.org.cn ~]# scp /usr/lib/systemd/system/docker.service node201.yinzhengjie.org.cn:/usr/lib/systemd/system/docker.service root@node201.yinzhengjie.org.cn's password: docker.service 100% 1846 1.2MB/s 00:00 [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# scp /usr/lib/systemd/system/docker.service node202.yinzhengjie.org.cn:/usr/lib/systemd/system/docker.service root@node202.yinzhengjie.org.cn's password: docker.service 100% 1846 1.5MB/s 00:00 [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# scp /usr/lib/systemd/system/docker.service node203.yinzhengjie.org.cn:/usr/lib/systemd/system/docker.service root@node203.yinzhengjie.org.cn's password: docker.service 100% 1846 790.6KB/s 00:00 [root@master200.yinzhengjie.org.cn ~]#
9>.啟動docker后驗證的網絡相關的內核參數(了解一下即可，如果你遇到了類似問題安裝下面的方法解決即可)
[root@master200.yinzhengjie.org.cn ~]# sysctl -a | grep bridge net.bridge.bridge-nf-call-arptables = 1 net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1 net.bridge.bridge-nf-filter-pppoe-tagged = 0 net.bridge.bridge-nf-filter-vlan-tagged = 0 net.bridge.bridge-nf-pass-vlan-input-dev = 0 sysctl: reading key "net.ipv6.conf.all.stable_secret" sysctl: reading key "net.ipv6.conf.bond0.stable_secret" sysctl: reading key "net.ipv6.conf.bond1.stable_secret" sysctl: reading key "net.ipv6.conf.default.stable_secret" sysctl: reading key "net.ipv6.conf.docker0.stable_secret" sysctl: reading key "net.ipv6.conf.eth0.stable_secret" sysctl: reading key "net.ipv6.conf.eth1.stable_secret" sysctl: reading key "net.ipv6.conf.eth2.stable_secret" sysctl: reading key "net.ipv6.conf.eth3.stable_secret" sysctl: reading key "net.ipv6.conf.lo.stable_secret" [root@master200.yinzhengjie.org.cn ~]# [root@node203.yinzhengjie.org.cn ~]# cat /etc/sysctl.d/k8s.conf net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1 [root@node203.yinzhengjie.org.cn ~]# [root@node203.yinzhengjie.org.cn ~]# sysctl -p /etc/sysctl.d/k8s.conf net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1 [root@node203.yinzhengjie.org.cn ~]#
三.安裝kubernetes相關的軟件包並初始化集群
1>.編寫k8s的軟件源倉庫的配置文件
[root@master200.yinzhengjie.org.cn ~]# vim /etc/yum.repos.d/kubernetes.repo [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# cat /etc/yum.repos.d/kubernetes.repo [kubernetes] name=Yinzhengjie's Kubernetes Repository baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/ gpgcheck=1 gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg [root@master200.yinzhengjie.org.cn ~]#
[root@master200.yinzhengjie.org.cn ~]# yum repolist Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile * base: mirror.bit.edu.cn * extras: mirrors.tuna.tsinghua.edu.cn * updates: mirrors.tuna.tsinghua.edu.cn kubernetes | 1.4 kB 00:00:00 kubernetes/primary | 63 kB 00:00:00 kubernetes 460/460 repo id repo name status base/7/x86_64 CentOS-7 - Base 10,097 docker-ce-stable/x86_64 Docker CE Stable - x86_64 63 extras/7/x86_64 CentOS-7 - Extras 323 kubernetes Yinzhengjie's Kubernetes Repository 460 updates/7/x86_64 CentOS-7 - Updates 1,117 repolist: 12,060 [root@master200.yinzhengjie.org.cn ~]#
[root@master200.yinzhengjie.org.cn ~]# yum list all | grep "^kube" kubeadm.x86_64 1.17.2-0 kubernetes kubectl.x86_64 1.17.2-0 kubernetes kubelet.x86_64 1.17.2-0 kubernetes kubernetes.x86_64 1.5.2-0.7.git269f928.el7 extras kubernetes-client.x86_64 1.5.2-0.7.git269f928.el7 extras kubernetes-cni.x86_64 0.7.5-0 kubernetes kubernetes-master.x86_64 1.5.2-0.7.git269f928.el7 extras kubernetes-node.x86_64 1.5.2-0.7.git269f928.el7 extras [root@master200.yinzhengjie.org.cn ~]#
2>.安裝軟件包並查看安裝的版本
[root@master200.yinzhengjie.org.cn ~]# yum -y install kubeadm kubectl kubelet Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile * base: mirror.bit.edu.cn * extras: mirrors.tuna.tsinghua.edu.cn * updates: mirrors.tuna.tsinghua.edu.cn Resolving Dependencies --> Running transaction check ---> Package kubeadm.x86_64 0:1.17.2-0 will be installed --> Processing Dependency: kubernetes-cni >= 0.7.5 for package: kubeadm-1.17.2-0.x86_64 --> Processing Dependency: cri-tools >= 1.13.0 for package: kubeadm-1.17.2-0.x86_64 ---> Package kubectl.x86_64 0:1.17.2-0 will be installed ---> Package kubelet.x86_64 0:1.17.2-0 will be installed --> Processing Dependency: socat for package: kubelet-1.17.2-0.x86_64 --> Processing Dependency: conntrack for package: kubelet-1.17.2-0.x86_64 --> Running transaction check ---> Package conntrack-tools.x86_64 0:1.4.4-5.el7_7.2 will be installed --> Processing Dependency: libnetfilter_cttimeout.so.1(LIBNETFILTER_CTTIMEOUT_1.1)(64bit) for package: conntrack-tools-1.4.4-5.el7_7.2.x86_64 --> Processing Dependency: libnetfilter_cttimeout.so.1(LIBNETFILTER_CTTIMEOUT_1.0)(64bit) for package: conntrack-tools-1.4.4-5.el7_7.2.x86_64 --> Processing Dependency: libnetfilter_cthelper.so.0(LIBNETFILTER_CTHELPER_1.0)(64bit) for package: conntrack-tools-1.4.4-5.el7_7.2.x86_64 --> Processing Dependency: libnetfilter_queue.so.1()(64bit) for package: conntrack-tools-1.4.4-5.el7_7.2.x86_64 --> Processing Dependency: libnetfilter_cttimeout.so.1()(64bit) for package: conntrack-tools-1.4.4-5.el7_7.2.x86_64 --> Processing Dependency: libnetfilter_cthelper.so.0()(64bit) for package: conntrack-tools-1.4.4-5.el7_7.2.x86_64 ---> Package cri-tools.x86_64 0:1.13.0-0 will be installed ---> Package kubernetes-cni.x86_64 0:0.7.5-0 will be installed ---> Package socat.x86_64 0:1.7.3.2-2.el7 will be installed --> Running transaction check ---> Package libnetfilter_cthelper.x86_64 0:1.0.0-10.el7_7.1 will be installed ---> Package libnetfilter_cttimeout.x86_64 0:1.0.0-6.el7_7.1 will be installed ---> Package libnetfilter_queue.x86_64 0:1.0.2-2.el7_2 will be installed --> Finished Dependency Resolution Dependencies Resolved ========================================================================================================================================================================================================================================================== Package Arch Version Repository Size ========================================================================================================================================================================================================================================================== Installing: kubeadm x86_64 1.17.2-0 kubernetes 8.7 M kubectl x86_64 1.17.2-0 kubernetes 9.4 M kubelet x86_64 1.17.2-0 kubernetes 20 M Installing for dependencies: conntrack-tools x86_64 1.4.4-5.el7_7.2 updates 187 k cri-tools x86_64 1.13.0-0 kubernetes 5.1 M kubernetes-cni x86_64 0.7.5-0 kubernetes 10 M libnetfilter_cthelper x86_64 1.0.0-10.el7_7.1 updates 18 k libnetfilter_cttimeout x86_64 1.0.0-6.el7_7.1 updates 18 k libnetfilter_queue x86_64 1.0.2-2.el7_2 base 23 k socat x86_64 1.7.3.2-2.el7 base 290 k Transaction Summary ========================================================================================================================================================================================================================================================== Install 3 Packages (+7 Dependent packages) Total download size: 54 M Installed size: 243 M Downloading packages: (1/10): conntrack-tools-1.4.4-5.el7_7.2.x86_64.rpm | 187 kB 00:00:00 warning: /var/cache/yum/x86_64/7/kubernetes/packages/14bfe6e75a9efc8eca3f638eb22c7e2ce759c67f95b43b16fae4ebabde1549f3-cri-tools-1.13.0-0.x86_64.rpm: Header V4 RSA/SHA512 Signature, key ID 3e1ba8d5: NOKEY ] 2.0 MB/s | 1.7 MB 00:00:26 ETA Public key for 14bfe6e75a9efc8eca3f638eb22c7e2ce759c67f95b43b16fae4ebabde1549f3-cri-tools-1.13.0-0.x86_64.rpm is not installed (2/10): 14bfe6e75a9efc8eca3f638eb22c7e2ce759c67f95b43b16fae4ebabde1549f3-cri-tools-1.13.0-0.x86_64.rpm | 5.1 MB 00:00:00 (3/10): b44630896c69cd411db53be1d5cb5ae899a40aba7c0766317ea904390fcfc45b-kubectl-1.17.2-0.x86_64.rpm | 9.4 MB 00:00:01 (4/10): 105d89f0607c7baf91305ba352e78000bd20aad5cdf706bffff3b31cd546dbf3-kubeadm-1.17.2-0.x86_64.rpm | 8.7 MB 00:00:02 (5/10): libnetfilter_cthelper-1.0.0-10.el7_7.1.x86_64.rpm | 18 kB 00:00:00 (6/10): libnetfilter_queue-1.0.2-2.el7_2.x86_64.rpm | 23 kB 00:00:00 (7/10): libnetfilter_cttimeout-1.0.0-6.el7_7.1.x86_64.rpm | 18 kB 00:00:00 (8/10): socat-1.7.3.2-2.el7.x86_64.rpm | 290 kB 00:00:01 (9/10): 3ee7f2dff78e6fbb3ac3af8acb1a907f4bec1b1ef4cf627cbe02fa553707f2e9-kubelet-1.17.2-0.x86_64.rpm | 20 MB 00:00:02 (10/10): 548a0dcd865c16a50980420ddfa5fbccb8b59621179798e6dc905c9bf8af3b34-kubernetes-cni-0.7.5-0.x86_64.rpm | 10 MB 00:00:03 ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Total 9.1 MB/s | 54 MB 00:00:05 Retrieving key from https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg Importing GPG key 0x3E1BA8D5: Userid : "Google Cloud Packages RPM Signing Key <gc-team@google.com>" Fingerprint: 3749 e1ba 95a8 6ce0 5454 6ed2 f09c 394c 3e1b a8d5 From : https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg Retrieving key from https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg Importing GPG key 0xA7317B0F: Userid : "Google Cloud Packages Automatic Signing Key <gc-team@google.com>" Fingerprint: d0bc 747f d8ca f711 7500 d6fa 3746 c208 a731 7b0f From : https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg Running transaction check Running transaction test Transaction test succeeded Running transaction Installing : libnetfilter_cttimeout-1.0.0-6.el7_7.1.x86_64 1/10 Installing : socat-1.7.3.2-2.el7.x86_64 2/10 Installing : cri-tools-1.13.0-0.x86_64 3/10 Installing : kubectl-1.17.2-0.x86_64 4/10 Installing : libnetfilter_queue-1.0.2-2.el7_2.x86_64 5/10 Installing : libnetfilter_cthelper-1.0.0-10.el7_7.1.x86_64 6/10 Installing : conntrack-tools-1.4.4-5.el7_7.2.x86_64 7/10 Installing : kubernetes-cni-0.7.5-0.x86_64 8/10 Installing : kubelet-1.17.2-0.x86_64 9/10 Installing : kubeadm-1.17.2-0.x86_64 10/10 Verifying : libnetfilter_cthelper-1.0.0-10.el7_7.1.x86_64 1/10 Verifying : conntrack-tools-1.4.4-5.el7_7.2.x86_64 2/10 Verifying : libnetfilter_queue-1.0.2-2.el7_2.x86_64 3/10 Verifying : kubectl-1.17.2-0.x86_64 4/10 Verifying : cri-tools-1.13.0-0.x86_64 5/10 Verifying : kubeadm-1.17.2-0.x86_64 6/10 Verifying : kubernetes-cni-0.7.5-0.x86_64 7/10 Verifying : socat-1.7.3.2-2.el7.x86_64 8/10 Verifying : kubelet-1.17.2-0.x86_64 9/10 Verifying : libnetfilter_cttimeout-1.0.0-6.el7_7.1.x86_64 10/10 Installed: kubeadm.x86_64 0:1.17.2-0 kubectl.x86_64 0:1.17.2-0 kubelet.x86_64 0:1.17.2-0 Dependency Installed: conntrack-tools.x86_64 0:1.4.4-5.el7_7.2 cri-tools.x86_64 0:1.13.0-0 kubernetes-cni.x86_64 0:0.7.5-0 libnetfilter_cthelper.x86_64 0:1.0.0-10.el7_7.1 libnetfilter_cttimeout.x86_64 0:1.0.0-6.el7_7.1 libnetfilter_queue.x86_64 0:1.0.2-2.el7_2 socat.x86_64 0:1.7.3.2-2.el7 Complete! [root@master200.yinzhengjie.org.cn ~]#
[root@master200.yinzhengjie.org.cn ~]# rpm -ql kubelet /etc/kubernetes/manifests /etc/sysconfig/kubelet /usr/bin/kubelet /usr/lib/systemd/system/kubelet.service [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# rpm -ql kubeadm /usr/bin/kubeadm /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# rpm -ql kubectl /usr/bin/kubectl [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# rpm -q kubeadm kubeadm-1.17.2-0.x86_64 [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# rpm -q kubectl kubectl-1.17.2-0.x86_64 [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# rpm -q kubelet kubelet-1.17.2-0.x86_64 [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]#
[root@master200.yinzhengjie.org.cn ~]# systemctl list-unit-files | grep kubelet kubelet.service disabled [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# systemctl status kubelet ● kubelet.service - kubelet: The Kubernetes Node Agent Loaded: loaded (/usr/lib/systemd/system/kubelet.service; disabled; vendor preset: disabled) Drop-In: /usr/lib/systemd/system/kubelet.service.d └─10-kubeadm.conf Active: inactive (dead) Docs: https://kubernetes.io/docs/ [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# systemctl enable kubelet Created symlink from /etc/systemd/system/multi-user.target.wants/kubelet.service to /usr/lib/systemd/system/kubelet.service. [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# systemctl list-unit-files | grep kubelet kubelet.service enabled [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# systemctl start kubelet [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# systemctl status kubelet ● kubelet.service - kubelet: The Kubernetes Node Agent Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled) Drop-In: /usr/lib/systemd/system/kubelet.service.d └─10-kubeadm.conf Active: active (running) since Wed 2020-02-05 05:12:34 CST; 2s ago Docs: https://kubernetes.io/docs/ Main PID: 5675 (kubelet) Tasks: 11 Memory: 117.3M CGroup: /system.slice/kubelet.service └─5675 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --cgroup-driver=cgroupfs --network-plugin=cni --pod-infra-container-im... Feb 05 05:12:35 master200.yinzhengjie.org.cn kubelet[5675]: I0205 05:12:35.006504 5675 remote_image.go:50] parsed scheme: "" Feb 05 05:12:35 master200.yinzhengjie.org.cn kubelet[5675]: I0205 05:12:35.006511 5675 remote_image.go:50] scheme "" not registered, fallback to default scheme Feb 05 05:12:35 master200.yinzhengjie.org.cn kubelet[5675]: I0205 05:12:35.006519 5675 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{/var/run/dockershim.sock 0 <nil>}] <nil>} Feb 05 05:12:35 master200.yinzhengjie.org.cn kubelet[5675]: I0205 05:12:35.006523 5675 clientconn.go:577] ClientConn switching balancer to "pick_first" Feb 05 05:12:35 master200.yinzhengjie.org.cn kubelet[5675]: E0205 05:12:35.933510 5675 reflector.go:153] k8s.io/kubernetes/pkg/kubelet/kubelet.go:458: Failed to list *v1.Node: Get https://172.200.1.200:6443/api/v1/nodes?fieldSelector=metadata.n... Feb 05 05:12:35 master200.yinzhengjie.org.cn kubelet[5675]: E0205 05:12:35.934154 5675 reflector.go:153] k8s.io/kubernetes/pkg/kubelet/kubelet.go:449: Failed to list *v1.Service: Get https://172.200.1.200:6443/api/v1/services?li...nnection refused Feb 05 05:12:35 master200.yinzhengjie.org.cn kubelet[5675]: E0205 05:12:35.935345 5675 reflector.go:153] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: Failed to list *v1.Pod: Get https://172.200.1.200:6443/api/v1/pods?fieldSelector=spec... Feb 05 05:12:36 master200.yinzhengjie.org.cn kubelet[5675]: E0205 05:12:36.934349 5675 reflector.go:153] k8s.io/kubernetes/pkg/kubelet/kubelet.go:458: Failed to list *v1.Node: Get https://172.200.1.200:6443/api/v1/nodes?fieldSelector=metadata.n... Feb 05 05:12:36 master200.yinzhengjie.org.cn kubelet[5675]: E0205 05:12:36.935197 5675 reflector.go:153] k8s.io/kubernetes/pkg/kubelet/kubelet.go:449: Failed to list *v1.Service: Get https://172.200.1.200:6443/api/v1/services?li...nnection refused Feb 05 05:12:36 master200.yinzhengjie.org.cn kubelet[5675]: E0205 05:12:36.937286 5675 reflector.go:153] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: Failed to list *v1.Pod: Get https://172.200.1.200:6443/api/v1/pods?fieldSelector=spec... Hint: Some lines were ellipsized, use -l to show in full. [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]#
3>.設置忽略swap啟用的狀態錯誤(如果你未禁用swap設備則需要編輯kubelet的配置文件，我們在安裝操作系統時就壓根沒有分配swap分區,因此該步驟直接忽略即可)
[root@master200.yinzhengjie.org.cn ~]# vim /etc/sysconfig/kubelet [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# cat /etc/sysconfig/kubelet KUBELET_EXTRA_ARGS="--fail-swap-on=false" #該參數表示盡管swap分區是啟用的也不報錯，如果不設置默認情況下如果你的服務器啟用了swap分區那么初始化時會報錯的 [root@master200.yinzhengjie.org.cn ~]#
4>.先將鏡像文件下載到本地
[root@master200.yinzhengjie.org.cn ~]# kubeadm config print init-defaults W0204 12:49:25.072012 10194 validation.go:28] Cannot validate kubelet config - no validator is available W0204 12:49:25.072058 10194 validation.go:28] Cannot validate kube-proxy config - no validator is available apiVersion: kubeadm.k8s.io/v1beta2 bootstrapTokens: - groups: - system:bootstrappers:kubeadm:default-node-token token: abcdef.0123456789abcdef ttl: 24h0m0s usages: - signing - authentication kind: InitConfiguration localAPIEndpoint: advertiseAddress: 1.2.3.4 bindPort: 6443 nodeRegistration: criSocket: /var/run/dockershim.sock name: master200.yinzhengjie.org.cn taints: - effect: NoSchedule key: node-role.kubernetes.io/master --- apiServer: timeoutForControlPlane: 4m0s apiVersion: kubeadm.k8s.io/v1beta2 certificatesDir: /etc/kubernetes/pki clusterName: kubernetes controllerManager: {} dns: type: CoreDNS etcd: local: dataDir: /var/lib/etcd imageRepository: k8s.gcr.io kind: ClusterConfiguration kubernetesVersion: v1.17.0 networking: dnsDomain: cluster.local serviceSubnet: 10.96.0.0/12 scheduler: {} [root@master200.yinzhengjie.org.cn ~]#
如果你有對docker配置了代理(即指定了VPN服務器)那么就可以使用Google官方提供的鏡像倉庫進行下載,執行如下命令即可: [root@master200.yinzhengjie.org.cn ~]# kubeadm config images pull 如上圖所示,默認情況下是無法直接訪問Google的鏡像倉庫"k8s.gcr.io"，此時咱們有三種解決方案: (1)在互聯網上找一些免費的VPN進行FQ操作或者在網上購買收費的VPN，推薦使用收費版本的，因為免費版本的VPN並不太穩定而且下載速度也並不是很理想，付費的VPN相對來說穩定性比較強,但是這種情況存在一個風險，因為你的所有操作都會被提供VPN的服務商在后台記錄; (2)在互聯網上購買VPS服務器，自己搭建VPN，前提是你購買的VPS可以訪問國外的各種網站，直接購買國內的阿里雲服務器可以你搭建出來VPN也是白搭,這種方案也需要掏錢，但相比於上面的那種方式安全性較強，因為你訪問的記錄在你的VPS中有記錄，你想干啥就干啥; (3)我們的目的就是下載Google的鏡像而已,可以在其它國內站點下載即可，而且很多國人已經很熱心的提供了下載連接,如果你沒有經常FQ的需求其實沒有必要采取上面兩種方案(而且有很多公司也會為員工購買FQ的工具，自己搭建VPN服務器的一般就是一些運維或開發人員喜歡干這事)。 本篇博客采取第三種方案，直接在國內的其它網站(比如阿里雲的鏡像網站)下載需要的鏡像，下載后給鏡像重新更名即可，具體操作請參考我筆記: K8S鏡像下載報錯解決方案(使用阿里雲鏡像去下載kubeadm需要的鏡像文件):https://www.cnblogs.com/yinzhengjie/p/12258215.html 鏡像成功下載到本地后,可以直接使用docker命令查看本地鏡像，如下圖所示。
5>.初始化集群(初始化過程會自動去下載鏡像，但上一步驟我們已經把鏡像提前下載到本地啦，因此不會去官網下載鏡像啦,這一步驟也會快很多)
[root@master200.yinzhengjie.org.cn ~]# kubeadm init --kubernetes-version="v1.17.2" --pod-network-cidr="10.244.0.0/16" --dry-run W0204 19:37:26.784651 31963 validation.go:28] Cannot validate kube-proxy config - no validator is available W0204 19:37:26.784688 31963 validation.go:28] Cannot validate kubelet config - no validator is available [init] Using Kubernetes version: v1.17.2 [preflight] Running pre-flight checks [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/ [WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service' [preflight] Would pull the required images (like 'kubeadm config images pull') [kubelet-start] Writing kubelet environment file with flags to file "/etc/kubernetes/tmp/kubeadm-init-dryrun988930826/kubeadm-flags.env" [kubelet-start] Writing kubelet configuration to file "/etc/kubernetes/tmp/kubeadm-init-dryrun988930826/config.yaml" [certs] Using certificateDir folder "/etc/kubernetes/tmp/kubeadm-init-dryrun988930826" [certs] Generating "ca" certificate and key [certs] Generating "apiserver" certificate and key [certs] apiserver serving cert is signed for DNS names [master200.yinzhengjie.org.cn kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 172.200.1.200] [certs] Generating "apiserver-kubelet-client" certificate and key [certs] Generating "front-proxy-ca" certificate and key [certs] Generating "front-proxy-client" certificate and key [certs] Generating "etcd/ca" certificate and key [certs] Generating "etcd/server" certificate and key [certs] etcd/server serving cert is signed for DNS names [master200.yinzhengjie.org.cn localhost] and IPs [172.200.1.200 127.0.0.1 ::1] [certs] Generating "etcd/peer" certificate and key [certs] etcd/peer serving cert is signed for DNS names [master200.yinzhengjie.org.cn localhost] and IPs [172.200.1.200 127.0.0.1 ::1] [certs] Generating "etcd/healthcheck-client" certificate and key [certs] Generating "apiserver-etcd-client" certificate and key [certs] Generating "sa" key and public key [kubeconfig] Using kubeconfig folder "/etc/kubernetes/tmp/kubeadm-init-dryrun988930826" [kubeconfig] Writing "admin.conf" kubeconfig file [kubeconfig] Writing "kubelet.conf" kubeconfig file [kubeconfig] Writing "controller-manager.conf" kubeconfig file [kubeconfig] Writing "scheduler.conf" kubeconfig file [control-plane] Using manifest folder "/etc/kubernetes/tmp/kubeadm-init-dryrun988930826" [control-plane] Creating static Pod manifest for "kube-apiserver" [control-plane] Creating static Pod manifest for "kube-controller-manager" W0204 19:37:28.996203 31963 manifests.go:214] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC" [control-plane] Creating static Pod manifest for "kube-scheduler" W0204 19:37:28.997161 31963 manifests.go:214] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC" [dryrun] Would ensure that "/var/lib/etcd" directory is present [etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/tmp/kubeadm-init-dryrun988930826" [dryrun] Wrote certificates, kubeconfig files and control plane manifests to the "/etc/kubernetes/tmp/kubeadm-init-dryrun988930826" directory [dryrun] The certificates or kubeconfig files would not be printed due to their sensitive nature [dryrun] Please examine the "/etc/kubernetes/tmp/kubeadm-init-dryrun988930826" directory for details about what would be written [dryrun] Would write file "/etc/kubernetes/manifests/kube-apiserver.yaml" with content: apiVersion: v1 kind: Pod metadata: creationTimestamp: null labels: component: kube-apiserver tier: control-plane name: kube-apiserver namespace: kube-system spec: containers: - command: - kube-apiserver - --advertise-address=172.200.1.200 - --allow-privileged=true - --authorization-mode=Node,RBAC - --client-ca-file=/etc/kubernetes/pki/ca.crt - --enable-admission-plugins=NodeRestriction - --enable-bootstrap-token-auth=true - --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt - --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt - --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key - --etcd-servers=https://127.0.0.1:2379 - --insecure-port=0 - --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt - --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname - --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt - --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key - --requestheader-allowed-names=front-proxy-client - --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt - --requestheader-extra-headers-prefix=X-Remote-Extra- - --requestheader-group-headers=X-Remote-Group - --requestheader-username-headers=X-Remote-User - --secure-port=6443 - --service-account-key-file=/etc/kubernetes/pki/sa.pub - --service-cluster-ip-range=10.96.0.0/12 - --tls-cert-file=/etc/kubernetes/pki/apiserver.crt - --tls-private-key-file=/etc/kubernetes/pki/apiserver.key image: k8s.gcr.io/kube-apiserver:v1.17.2 imagePullPolicy: IfNotPresent livenessProbe: failureThreshold: 8 httpGet: host: 172.200.1.200 path: /healthz port: 6443 scheme: HTTPS initialDelaySeconds: 15 timeoutSeconds: 15 name: kube-apiserver resources: requests: cpu: 250m volumeMounts: - mountPath: /etc/ssl/certs name: ca-certs readOnly: true - mountPath: /etc/pki name: etc-pki readOnly: true - mountPath: /etc/kubernetes/pki name: k8s-certs readOnly: true hostNetwork: true priorityClassName: system-cluster-critical volumes: - hostPath: path: /etc/ssl/certs type: DirectoryOrCreate name: ca-certs - hostPath: path: /etc/pki type: DirectoryOrCreate name: etc-pki - hostPath: path: /etc/kubernetes/pki type: DirectoryOrCreate name: k8s-certs status: {} [dryrun] Would write file "/etc/kubernetes/manifests/kube-controller-manager.yaml" with content: apiVersion: v1 kind: Pod metadata: creationTimestamp: null labels: component: kube-controller-manager tier: control-plane name: kube-controller-manager namespace: kube-system spec: containers: - command: - kube-controller-manager - --allocate-node-cidrs=true - --authentication-kubeconfig=/etc/kubernetes/controller-manager.conf - --authorization-kubeconfig=/etc/kubernetes/controller-manager.conf - --bind-address=127.0.0.1 - --client-ca-file=/etc/kubernetes/pki/ca.crt - --cluster-cidr=10.244.0.0/16 - --cluster-signing-cert-file=/etc/kubernetes/pki/ca.crt - --cluster-signing-key-file=/etc/kubernetes/pki/ca.key - --controllers=*,bootstrapsigner,tokencleaner - --kubeconfig=/etc/kubernetes/controller-manager.conf - --leader-elect=true - --node-cidr-mask-size=24 - --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt - --root-ca-file=/etc/kubernetes/pki/ca.crt - --service-account-private-key-file=/etc/kubernetes/pki/sa.key - --service-cluster-ip-range=10.96.0.0/12 - --use-service-account-credentials=true image: k8s.gcr.io/kube-controller-manager:v1.17.2 imagePullPolicy: IfNotPresent livenessProbe: failureThreshold: 8 httpGet: host: 127.0.0.1 path: /healthz port: 10257 scheme: HTTPS initialDelaySeconds: 15 timeoutSeconds: 15 name: kube-controller-manager resources: requests: cpu: 200m volumeMounts: - mountPath: /etc/ssl/certs name: ca-certs readOnly: true - mountPath: /etc/pki name: etc-pki readOnly: true - mountPath: /usr/libexec/kubernetes/kubelet-plugins/volume/exec name: flexvolume-dir - mountPath: /etc/kubernetes/pki name: k8s-certs readOnly: true - mountPath: /etc/kubernetes/controller-manager.conf name: kubeconfig readOnly: true hostNetwork: true priorityClassName: system-cluster-critical volumes: - hostPath: path: /etc/ssl/certs type: DirectoryOrCreate name: ca-certs - hostPath: path: /etc/pki type: DirectoryOrCreate name: etc-pki - hostPath: path: /usr/libexec/kubernetes/kubelet-plugins/volume/exec type: DirectoryOrCreate name: flexvolume-dir - hostPath: path: /etc/kubernetes/pki type: DirectoryOrCreate name: k8s-certs - hostPath: path: /etc/kubernetes/controller-manager.conf type: FileOrCreate name: kubeconfig status: {} [dryrun] Would write file "/etc/kubernetes/manifests/kube-scheduler.yaml" with content: apiVersion: v1 kind: Pod metadata: creationTimestamp: null labels: component: kube-scheduler tier: control-plane name: kube-scheduler namespace: kube-system spec: containers: - command: - kube-scheduler - --authentication-kubeconfig=/etc/kubernetes/scheduler.conf - --authorization-kubeconfig=/etc/kubernetes/scheduler.conf - --bind-address=127.0.0.1 - --kubeconfig=/etc/kubernetes/scheduler.conf - --leader-elect=true image: k8s.gcr.io/kube-scheduler:v1.17.2 imagePullPolicy: IfNotPresent livenessProbe: failureThreshold: 8 httpGet: host: 127.0.0.1 path: /healthz port: 10259 scheme: HTTPS initialDelaySeconds: 15 timeoutSeconds: 15 name: kube-scheduler resources: requests: cpu: 100m volumeMounts: - mountPath: /etc/kubernetes/scheduler.conf name: kubeconfig readOnly: true hostNetwork: true priorityClassName: system-cluster-critical volumes: - hostPath: path: /etc/kubernetes/scheduler.conf type: FileOrCreate name: kubeconfig status: {} [dryrun] Would write file "/var/lib/kubelet/config.yaml" with content: apiVersion: kubelet.config.k8s.io/v1beta1 authentication: anonymous: enabled: false webhook: cacheTTL: 0s enabled: true x509: clientCAFile: /etc/kubernetes/pki/ca.crt authorization: mode: Webhook webhook: cacheAuthorizedTTL: 0s cacheUnauthorizedTTL: 0s clusterDNS: - 10.96.0.10 clusterDomain: cluster.local cpuManagerReconcilePeriod: 0s evictionPressureTransitionPeriod: 0s fileCheckFrequency: 0s healthzBindAddress: 127.0.0.1 healthzPort: 10248 httpCheckFrequency: 0s imageMinimumGCAge: 0s kind: KubeletConfiguration nodeStatusReportFrequency: 0s nodeStatusUpdateFrequency: 0s rotateCertificates: true runtimeRequestTimeout: 0s staticPodPath: /etc/kubernetes/manifests streamingConnectionIdleTimeout: 0s syncFrequency: 0s volumeStatsAggPeriod: 0s [dryrun] Would write file "/var/lib/kubelet/kubeadm-flags.env" with content: KUBELET_KUBEADM_ARGS="--cgroup-driver=cgroupfs --network-plugin=cni --pod-infra-container-image=k8s.gcr.io/pause:3.1" [wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/tmp/kubeadm-init-dryrun988930826". This can take up to 4m0s [upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace [dryrun] Would perform action CREATE on resource "configmaps" in API group "core/v1" [dryrun] Attached object: apiVersion: v1 data: ClusterConfiguration: | apiServer: extraArgs: authorization-mode: Node,RBAC timeoutForControlPlane: 4m0s apiVersion: kubeadm.k8s.io/v1beta2 certificatesDir: /etc/kubernetes/pki clusterName: kubernetes controllerManager: {} dns: type: CoreDNS etcd: local: dataDir: /var/lib/etcd imageRepository: k8s.gcr.io kind: ClusterConfiguration kubernetesVersion: v1.17.2 networking: dnsDomain: cluster.local podSubnet: 10.244.0.0/16 serviceSubnet: 10.96.0.0/12 scheduler: {} ClusterStatus: | apiEndpoints: master200.yinzhengjie.org.cn: advertiseAddress: 172.200.1.200 bindPort: 6443 apiVersion: kubeadm.k8s.io/v1beta2 kind: ClusterStatus kind: ConfigMap metadata: creationTimestamp: null name: kubeadm-config namespace: kube-system [dryrun] Would perform action CREATE on resource "roles" in API group "rbac.authorization.k8s.io/v1" [dryrun] Attached object: apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: creationTimestamp: null name: kubeadm:nodes-kubeadm-config namespace: kube-system rules: - apiGroups: - "" resourceNames: - kubeadm-config resources: - configmaps verbs: - get [dryrun] Would perform action CREATE on resource "rolebindings" in API group "rbac.authorization.k8s.io/v1" [dryrun] Attached object: apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: creationTimestamp: null name: kubeadm:nodes-kubeadm-config namespace: kube-system roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: kubeadm:nodes-kubeadm-config subjects: - kind: Group name: system:bootstrappers:kubeadm:default-node-token - kind: Group name: system:nodes [kubelet] Creating a ConfigMap "kubelet-config-1.17" in namespace kube-system with the configuration for the kubelets in the cluster [dryrun] Would perform action CREATE on resource "configmaps" in API group "core/v1" [dryrun] Attached object: apiVersion: v1 data: kubelet: | apiVersion: kubelet.config.k8s.io/v1beta1 authentication: anonymous: enabled: false webhook: cacheTTL: 0s enabled: true x509: clientCAFile: /etc/kubernetes/pki/ca.crt authorization: mode: Webhook webhook: cacheAuthorizedTTL: 0s cacheUnauthorizedTTL: 0s clusterDNS: - 10.96.0.10 clusterDomain: cluster.local cpuManagerReconcilePeriod: 0s evictionPressureTransitionPeriod: 0s fileCheckFrequency: 0s healthzBindAddress: 127.0.0.1 healthzPort: 10248 httpCheckFrequency: 0s imageMinimumGCAge: 0s kind: KubeletConfiguration nodeStatusReportFrequency: 0s nodeStatusUpdateFrequency: 0s rotateCertificates: true runtimeRequestTimeout: 0s staticPodPath: /etc/kubernetes/manifests streamingConnectionIdleTimeout: 0s syncFrequency: 0s volumeStatsAggPeriod: 0s kind: ConfigMap metadata: creationTimestamp: null name: kubelet-config-1.17 namespace: kube-system [dryrun] Would perform action CREATE on resource "roles" in API group "rbac.authorization.k8s.io/v1" [dryrun] Attached object: apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: creationTimestamp: null name: kubeadm:kubelet-config-1.17 namespace: kube-system rules: - apiGroups: - "" resourceNames: - kubelet-config-1.17 resources: - configmaps verbs: - get [dryrun] Would perform action CREATE on resource "rolebindings" in API group "rbac.authorization.k8s.io/v1" [dryrun] Attached object: apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: creationTimestamp: null name: kubeadm:kubelet-config-1.17 namespace: kube-system roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: kubeadm:kubelet-config-1.17 subjects: - kind: Group name: system:nodes - kind: Group name: system:bootstrappers:kubeadm:default-node-token [dryrun] Would perform action GET on resource "nodes" in API group "core/v1" [dryrun] Resource name: "master200.yinzhengjie.org.cn" [dryrun] Would perform action PATCH on resource "nodes" in API group "core/v1" [dryrun] Resource name: "master200.yinzhengjie.org.cn" [dryrun] Attached patch: {"metadata":{"annotations":{"kubeadm.alpha.kubernetes.io/cri-socket":"/var/run/dockershim.sock"}}} [upload-certs] Skipping phase. Please see --upload-certs [mark-control-plane] Marking the node master200.yinzhengjie.org.cn as control-plane by adding the label "node-role.kubernetes.io/master=''" [mark-control-plane] Marking the node master200.yinzhengjie.org.cn as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule] [dryrun] Would perform action GET on resource "nodes" in API group "core/v1" [dryrun] Resource name: "master200.yinzhengjie.org.cn" [dryrun] Would perform action PATCH on resource "nodes" in API group "core/v1" [dryrun] Resource name: "master200.yinzhengjie.org.cn" [dryrun] Attached patch: {"metadata":{"labels":{"node-role.kubernetes.io/master":""}},"spec":{"taints":[{"effect":"NoSchedule","key":"node-role.kubernetes.io/master"}]}} [bootstrap-token] Using token: dp9kc3.s7ceexx1xblkqklh [bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles [dryrun] Would perform action GET on resource "secrets" in API group "core/v1" [dryrun] Resource name: "bootstrap-token-dp9kc3" [dryrun] Would perform action CREATE on resource "secrets" in API group "core/v1" [dryrun] Attached object: apiVersion: v1 data: auth-extra-groups: c3lzdGVtOmJvb3RzdHJhcHBlcnM6a3ViZWFkbTpkZWZhdWx0LW5vZGUtdG9rZW4= description: VGhlIGRlZmF1bHQgYm9vdHN0cmFwIHRva2VuIGdlbmVyYXRlZCBieSAna3ViZWFkbSBpbml0Jy4= expiration: MjAyMC0wMi0wNVQxOTozNzozMCswODowMA== token-id: ZHA5a2Mz token-secret: czdjZWV4eDF4YmxrcWtsaA== usage-bootstrap-authentication: dHJ1ZQ== usage-bootstrap-signing: dHJ1ZQ== kind: Secret metadata: creationTimestamp: null name: bootstrap-token-dp9kc3 namespace: kube-system type: bootstrap.kubernetes.io/token [bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials [dryrun] Would perform action CREATE on resource "clusterrolebindings" in API group "rbac.authorization.k8s.io/v1" [dryrun] Attached object: apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: creationTimestamp: null name: kubeadm:kubelet-bootstrap roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: system:node-bootstrapper subjects: - kind: Group name: system:bootstrappers:kubeadm:default-node-token [bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token [dryrun] Would perform action CREATE on resource "clusterrolebindings" in API group "rbac.authorization.k8s.io/v1" [dryrun] Attached object: apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: creationTimestamp: null name: kubeadm:node-autoapprove-bootstrap roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: system:certificates.k8s.io:certificatesigningrequests:nodeclient subjects: - kind: Group name: system:bootstrappers:kubeadm:default-node-token [bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster [dryrun] Would perform action CREATE on resource "clusterrolebindings" in API group "rbac.authorization.k8s.io/v1" [dryrun] Attached object: apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: creationTimestamp: null name: kubeadm:node-autoapprove-certificate-rotation roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: system:certificates.k8s.io:certificatesigningrequests:selfnodeclient subjects: - kind: Group name: system:nodes [bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace [dryrun] Would perform action CREATE on resource "configmaps" in API group "core/v1" [dryrun] Attached object: apiVersion: v1 data: kubeconfig: | apiVersion: v1 clusters: - cluster: certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUN5RENDQWJDZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQjRYRFRJd01ESXdOREV4TXpjeU4xb1hEVE13TURJd01URXhNemN5TjFvd0ZURVRNQkVHQTFVRQpBeE1LYTNWaVpYSn VaWFJsY3pDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBSjgrCmhvL1ptZkNhSTJITDJEam5wZk9kT1l2QnNxQTF0YmtNT1NGKzdNOTFHQ1FWZE9STG5IbUFRZTZQRExIbjQ5TUEKbmVyemNLTnp6S203T0liNmdDRlFyRERjYUwvaUV0VWlFQWhPQXd3ZTIwTTAzQlFWblptY3VlcVFqbE1BSUVrdApNamRRT3lhYnluUkt5bHUwSXZPRHJaeUE5Mjc2dEFFSWNnZlFEVEUrU2hwSmU1aUIveDNXOTMwZGN1T3F1cVgvClZvRzkxcHVDdjNGSlF5aEdBSE91emhQMmtodmVuemdkZXEyc2RoUlVReEU4OXJVU294Q3VBS2Z3TjVONGVaR0QKbWRVc0dHS0pGblRLOXA5YkpFNncxWGh1d1VqTmltZmdhc2RvbVBOMmxVaU84MFFCcDdWM3ZlNjNhbVVYRVhVWQpVYk1oUFVlL21mZ1hyQnJZME1FQ0F3RUFBYU1qTUNFd0RnWURWUjBQQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCCi93UUZNQU1CQWY4d0RRWUpLb1pJaHZjTkFRRUxCUUFEZ2dFQkFCMXdtWW9tYzBXN210aEVlK0ErK0lkTGVrOEoKRUwyNnkyVXJFR3g5MVRkeTRvZFpCcVVpRzltNERmdTNlZlU2T0g3V0gyTG1RNGNXbWtLcitDdmJrRThRK3FHMwo3TGRvNTdzbHhwN2NyYU5tQ3dmb1ZIOCtLbVl4UUlGOWpxMGJ4RjlqRlNMSnJHa0N6cDBwMlhaQi9Tc1VFb3M4ClNsU0hvWEhuemQ2ekZSTmJLKzYyMFkvV1ZrU2h1V3VuRW5rV1kva0hHdTB6eHJqOUJiQ25yM0FETzVqZ0tyM08KN24wdmhnZzlJaE1Rb25lYldJTEdNVXV1b21VNlBvWDN3eHprOVJkSjV6T05samN6bXhVc2IvUFJEcnQ3bkFGUQpjYmdGclcrOUo1VmkyMFNMLzZmK1ZXZnV3aHNpVXFaU1VLWE10dE9OQitjYkRlVFVqYjN3MHErSjNPbz0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo= server: https://172.200.1.200:6443 name: "" contexts: null current-context: "" kind: Config preferences: {} users: null kind: ConfigMap metadata: creationTimestamp: null name: cluster-info namespace: kube-public [dryrun] Would perform action CREATE on resource "roles" in API group "rbac.authorization.k8s.io/v1" [dryrun] Attached object: apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: creationTimestamp: null name: kubeadm:bootstrap-signer-clusterinfo namespace: kube-public rules: - apiGroups: - "" resourceNames: - cluster-info resources: - configmaps verbs: - get [dryrun] Would perform action CREATE on resource "rolebindings" in API group "rbac.authorization.k8s.io/v1" [dryrun] Attached object: apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: creationTimestamp: null name: kubeadm:bootstrap-signer-clusterinfo namespace: kube-public roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: kubeadm:bootstrap-signer-clusterinfo subjects: - kind: User name: system:anonymous [dryrun] Would perform action GET on resource "configmaps" in API group "core/v1" [dryrun] Resource name: "kube-dns" [dryrun] Would perform action GET on resource "configmaps" in API group "core/v1" [dryrun] Resource name: "coredns" [dryrun] Would perform action CREATE on resource "configmaps" in API group "core/v1" [dryrun] Attached object: apiVersion: v1 data: Corefile: | .:53 { errors health { lameduck 5s } ready kubernetes cluster.local in-addr.arpa ip6.arpa { pods insecure fallthrough in-addr.arpa ip6.arpa ttl 30 } prometheus :9153 forward . /etc/resolv.conf cache 30 loop reload loadbalance } kind: ConfigMap metadata: creationTimestamp: null name: coredns namespace: kube-system [dryrun] Would perform action CREATE on resource "clusterroles" in API group "rbac.authorization.k8s.io/v1" [dryrun] Attached object: apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: creationTimestamp: null name: system:coredns rules: - apiGroups: - "" resources: - endpoints - services - pods - namespaces verbs: - list - watch - apiGroups: - "" resources: - nodes verbs: - get [dryrun] Would perform action CREATE on resource "clusterrolebindings" in API group "rbac.authorization.k8s.io/v1" [dryrun] Attached object: apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: creationTimestamp: null name: system:coredns roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: system:coredns subjects: - kind: ServiceAccount name: coredns namespace: kube-system [dryrun] Would perform action CREATE on resource "serviceaccounts" in API group "core/v1" [dryrun] Attached object: apiVersion: v1 kind: ServiceAccount metadata: creationTimestamp: null name: coredns namespace: kube-system [dryrun] Would perform action CREATE on resource "deployments" in API group "apps/v1" [dryrun] Attached object: apiVersion: apps/v1 kind: Deployment metadata: creationTimestamp: null labels: k8s-app: kube-dns name: coredns namespace: kube-system spec: replicas: 2 selector: matchLabels: k8s-app: kube-dns strategy: rollingUpdate: maxUnavailable: 1 type: RollingUpdate template: metadata: creationTimestamp: null labels: k8s-app: kube-dns spec: containers: - args: - -conf - /etc/coredns/Corefile image: k8s.gcr.io/coredns:1.6.5 imagePullPolicy: IfNotPresent livenessProbe: failureThreshold: 5 httpGet: path: /health port: 8080 scheme: HTTP initialDelaySeconds: 60 successThreshold: 1 timeoutSeconds: 5 name: coredns ports: - containerPort: 53 name: dns protocol: UDP - containerPort: 53 name: dns-tcp protocol: TCP - containerPort: 9153 name: metrics protocol: TCP readinessProbe: httpGet: path: /ready port: 8181 scheme: HTTP resources: limits: memory: 170Mi requests: cpu: 100m memory: 70Mi securityContext: allowPrivilegeEscalation: false capabilities: add: - NET_BIND_SERVICE drop: - all readOnlyRootFilesystem: true volumeMounts: - mountPath: /etc/coredns name: config-volume readOnly: true dnsPolicy: Default nodeSelector: beta.kubernetes.io/os: linux priorityClassName: system-cluster-critical serviceAccountName: coredns tolerations: - key: CriticalAddonsOnly operator: Exists - effect: NoSchedule key: node-role.kubernetes.io/master volumes: - configMap: items: - key: Corefile path: Corefile name: coredns name: config-volume status: {} [dryrun] Would perform action CREATE on resource "services" in API group "core/v1" [dryrun] Attached object: apiVersion: v1 kind: Service metadata: annotations: prometheus.io/port: "9153" prometheus.io/scrape: "true" creationTimestamp: null labels: k8s-app: kube-dns kubernetes.io/cluster-service: "true" kubernetes.io/name: KubeDNS name: kube-dns namespace: kube-system resourceVersion: "0" spec: clusterIP: 10.96.0.10 ports: - name: dns port: 53 protocol: UDP targetPort: 53 - name: dns-tcp port: 53 protocol: TCP targetPort: 53 - name: metrics port: 9153 protocol: TCP targetPort: 9153 selector: k8s-app: kube-dns status: loadBalancer: {} [addons] Applied essential addon: CoreDNS [dryrun] Would perform action CREATE on resource "serviceaccounts" in API group "core/v1" [dryrun] Attached object: apiVersion: v1 kind: ServiceAccount metadata: creationTimestamp: null name: kube-proxy namespace: kube-system [dryrun] Would perform action CREATE on resource "configmaps" in API group "core/v1" [dryrun] Attached object: apiVersion: v1 data: config.conf: |- apiVersion: kubeproxy.config.k8s.io/v1alpha1 bindAddress: 0.0.0.0 clientConnection: acceptContentTypes: "" burst: 0 contentType: "" kubeconfig: /var/lib/kube-proxy/kubeconfig.conf qps: 0 clusterCIDR: 10.244.0.0/16 configSyncPeriod: 0s conntrack: maxPerCore: null min: null tcpCloseWaitTimeout: null tcpEstablishedTimeout: null enableProfiling: false healthzBindAddress: "" hostnameOverride: "" iptables: masqueradeAll: false masqueradeBit: null minSyncPeriod: 0s syncPeriod: 0s ipvs: excludeCIDRs: null minSyncPeriod: 0s scheduler: "" strictARP: false syncPeriod: 0s kind: KubeProxyConfiguration metricsBindAddress: "" mode: "" nodePortAddresses: null oomScoreAdj: null portRange: "" udpIdleTimeout: 0s winkernel: enableDSR: false networkName: "" sourceVip: "" kubeconfig.conf: |- apiVersion: v1 kind: Config clusters: - cluster: certificate-authority: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt server: https://172.200.1.200:6443 name: default contexts: - context: cluster: default namespace: default user: default name: default current-context: default users: - name: default user: tokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token kind: ConfigMap metadata: creationTimestamp: null labels: app: kube-proxy name: kube-proxy namespace: kube-system [dryrun] Would perform action CREATE on resource "daemonsets" in API group "apps/v1" [dryrun] Attached object: apiVersion: apps/v1 kind: DaemonSet metadata: creationTimestamp: null labels: k8s-app: kube-proxy name: kube-proxy namespace: kube-system spec: selector: matchLabels: k8s-app: kube-proxy template: metadata: creationTimestamp: null labels: k8s-app: kube-proxy spec: containers: - command: - /usr/local/bin/kube-proxy - --config=/var/lib/kube-proxy/config.conf - --hostname-override=$(NODE_NAME) env: - name: NODE_NAME valueFrom: fieldRef: fieldPath: spec.nodeName image: k8s.gcr.io/kube-proxy:v1.17.2 imagePullPolicy: IfNotPresent name: kube-proxy resources: {} securityContext: privileged: true volumeMounts: - mountPath: /var/lib/kube-proxy name: kube-proxy - mountPath: /run/xtables.lock name: xtables-lock - mountPath: /lib/modules name: lib-modules readOnly: true hostNetwork: true nodeSelector: beta.kubernetes.io/os: linux priorityClassName: system-node-critical serviceAccountName: kube-proxy tolerations: - key: CriticalAddonsOnly operator: Exists - operator: Exists volumes: - configMap: name: kube-proxy name: kube-proxy - hostPath: path: /run/xtables.lock type: FileOrCreate name: xtables-lock - hostPath: path: /lib/modules name: lib-modules updateStrategy: type: RollingUpdate status: currentNumberScheduled: 0 desiredNumberScheduled: 0 numberMisscheduled: 0 numberReady: 0 [dryrun] Would perform action CREATE on resource "clusterrolebindings" in API group "rbac.authorization.k8s.io/v1" [dryrun] Attached object: apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: creationTimestamp: null name: kubeadm:node-proxier roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: system:node-proxier subjects: - kind: ServiceAccount name: kube-proxy namespace: kube-system [dryrun] Would perform action CREATE on resource "roles" in API group "rbac.authorization.k8s.io/v1" [dryrun] Attached object: apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: creationTimestamp: null name: kube-proxy namespace: kube-system rules: - apiGroups: - "" resourceNames: - kube-proxy resources: - configmaps verbs: - get [dryrun] Would perform action CREATE on resource "rolebindings" in API group "rbac.authorization.k8s.io/v1" [dryrun] Attached object: apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: creationTimestamp: null name: kube-proxy namespace: kube-system roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: kube-proxy subjects: - kind: Group name: system:bootstrappers:kubeadm:default-node-token [addons] Applied essential addon: kube-proxy Your Kubernetes control-plane has initialized successfully! To start using your cluster, you need to run the following as a regular user: mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/tmp/kubeadm-init-dryrun988930826/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config You should now deploy a pod network to the cluster. Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: https://kubernetes.io/docs/concepts/cluster-administration/addons/ Then you can join any number of worker nodes by running the following on each as root: kubeadm join 172.200.1.200:6443 --token dp9kc3.s7ceexx1xblkqklh \ --discovery-token-ca-cert-hash sha256:c87a7b40890a61c0b9c23d71418f1b63987a74d943839b018eb836d1cdded4eb [root@master200.yinzhengjie.org.cn ~]#
[root@master200.yinzhengjie.org.cn ~]# kubeadm init --kubernetes-version="v1.17.2" --pod-network-cidr="10.244.0.0/16" W0204 19:39:12.076223 32177 validation.go:28] Cannot validate kube-proxy config - no validator is available W0204 19:39:12.076260 32177 validation.go:28] Cannot validate kubelet config - no validator is available [init] Using Kubernetes version: v1.17.2 [preflight] Running pre-flight checks [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/ [WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service' [preflight] Pulling images required for setting up a Kubernetes cluster [preflight] This might take a minute or two, depending on the speed of your internet connection [preflight] You can also perform this action in beforehand using 'kubeadm config images pull' [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [kubelet-start] Starting the kubelet [certs] Using certificateDir folder "/etc/kubernetes/pki" [certs] Generating "ca" certificate and key [certs] Generating "apiserver" certificate and key [certs] apiserver serving cert is signed for DNS names [master200.yinzhengjie.org.cn kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 172.200.1.200] [certs] Generating "apiserver-kubelet-client" certificate and key [certs] Generating "front-proxy-ca" certificate and key [certs] Generating "front-proxy-client" certificate and key [certs] Generating "etcd/ca" certificate and key [certs] Generating "etcd/server" certificate and key [certs] etcd/server serving cert is signed for DNS names [master200.yinzhengjie.org.cn localhost] and IPs [172.200.1.200 127.0.0.1 ::1] [certs] Generating "etcd/peer" certificate and key [certs] etcd/peer serving cert is signed for DNS names [master200.yinzhengjie.org.cn localhost] and IPs [172.200.1.200 127.0.0.1 ::1] [certs] Generating "etcd/healthcheck-client" certificate and key [certs] Generating "apiserver-etcd-client" certificate and key [certs] Generating "sa" key and public key [kubeconfig] Using kubeconfig folder "/etc/kubernetes" [kubeconfig] Writing "admin.conf" kubeconfig file [kubeconfig] Writing "kubelet.conf" kubeconfig file [kubeconfig] Writing "controller-manager.conf" kubeconfig file [kubeconfig] Writing "scheduler.conf" kubeconfig file [control-plane] Using manifest folder "/etc/kubernetes/manifests" [control-plane] Creating static Pod manifest for "kube-apiserver" [control-plane] Creating static Pod manifest for "kube-controller-manager" W0204 19:39:14.609238 32177 manifests.go:214] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC" [control-plane] Creating static Pod manifest for "kube-scheduler" W0204 19:39:14.609827 32177 manifests.go:214] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC" [etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests" [wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s [apiclient] All control plane components are healthy after 18.505316 seconds [upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace [kubelet] Creating a ConfigMap "kubelet-config-1.17" in namespace kube-system with the configuration for the kubelets in the cluster [upload-certs] Skipping phase. Please see --upload-certs [mark-control-plane] Marking the node master200.yinzhengjie.org.cn as control-plane by adding the label "node-role.kubernetes.io/master=''" [mark-control-plane] Marking the node master200.yinzhengjie.org.cn as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule] [bootstrap-token] Using token: gu02ed.9k1rpyl4mtkhvdpk [bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles [bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials [bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token [bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster [bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace [kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key [addons] Applied essential addon: CoreDNS [addons] Applied essential addon: kube-proxy Your Kubernetes control-plane has initialized successfully! To start using your cluster, you need to run the following as a regular user: mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config You should now deploy a pod network to the cluster. Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: https://kubernetes.io/docs/concepts/cluster-administration/addons/ Then you can join any number of worker nodes by running the following on each as root: kubeadm join 172.200.1.200:6443 --token gu02ed.9k1rpyl4mtkhvdpk \ --discovery-token-ca-cert-hash sha256:755ff7d07588b75aec769e566e4257d832cf6d1ec79bcdbd655dad80f30a6794 [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# echo $? 0 [root@master200.yinzhengjie.org.cn ~]#
6>.部署網絡插件
[root@master200.yinzhengjie.org.cn ~]# mkdir .kube [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# cp /etc/kubernetes/admin.conf .kube/config [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# kubectl get nodes #kubectl是k8s的apiserver的客戶端工具，我們可以通過該命令查看節點的狀態，很明顯是"NotReady"狀態，我們需要執行嚇一跳命令下載網絡插件 NAME STATUS ROLES AGE VERSION master200.yinzhengjie.org.cn NotReady master 6m21s v1.17.2 [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml #下載flannel網絡插件 podsecuritypolicy.policy/psp.flannel.unprivileged created clusterrole.rbac.authorization.k8s.io/flannel created clusterrolebinding.rbac.authorization.k8s.io/flannel created serviceaccount/flannel created configmap/kube-flannel-cfg created daemonset.apps/kube-flannel-ds-amd64 created daemonset.apps/kube-flannel-ds-arm64 created daemonset.apps/kube-flannel-ds-arm created daemonset.apps/kube-flannel-ds-ppc64le created daemonset.apps/kube-flannel-ds-s390x created [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# kubectl get pods -n kube-system #使用"-n"查看名稱"kube-system"的名稱空間,如下圖所示，所有組件都是"Running"狀態，因此我們的master節點應該也是"Ready"狀態啦~ NAME READY STATUS RESTARTS AGE coredns-6955765f44-455fh 0/1 Running 0 8m17s coredns-6955765f44-q6zqj 0/1 Running 0 8m17s etcd-master200.yinzhengjie.org.cn 1/1 Running 0 8m31s kube-apiserver-master200.yinzhengjie.org.cn 1/1 Running 0 8m31s kube-controller-manager-master200.yinzhengjie.org.cn 1/1 Running 0 8m31s kube-flannel-ds-amd64-hnnhb 1/1 Running 0 33s kube-proxy-6r9dx 1/1 Running 0 8m17s kube-scheduler-master200.yinzhengjie.org.cn 1/1 Running 0 8m31s [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# kubectl get nodes #安裝flannel(由CoreOS公司研發)網絡插件后,它會自動運行,之后咱們的k8s master節點的狀態成功變為"Ready"。 NAME STATUS ROLES AGE VERSION master200.yinzhengjie.org.cn Ready master 9m40s v1.17.2 [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]#
7>.將k8s master節點的鏡像打包並發送到k8s node節點
[root@master200.yinzhengjie.org.cn ~]# docker image ls REPOSITORY TAG IMAGE ID CREATED SIZE k8s.gcr.io/kube-proxy v1.17.2 cba2a99699bd 2 weeks ago 116MB k8s.gcr.io/kube-apiserver v1.17.2 41ef50a5f06a 2 weeks ago 171MB k8s.gcr.io/kube-controller-manager v1.17.2 da5fd66c4068 2 weeks ago 161MB k8s.gcr.io/kube-scheduler v1.17.2 f52d4c527ef2 2 weeks ago 94.4MB k8s.gcr.io/coredns 1.6.5 70f311871ae1 3 months ago 41.6MB k8s.gcr.io/etcd 3.4.3-0 303ce5db0e90 3 months ago 288MB quay.io/coreos/flannel v0.11.0-amd64 ff281650a721 12 months ago 52.6MB k8s.gcr.io/pause 3.1 da86e6ba6ca1 2 years ago 742kB [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# ll total 0 [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# docker image save k8s.gcr.io/kube-proxy:v1.17.2 k8s.gcr.io/pause:3.1 quay.io/coreos/flannel:v0.11.0-amd64 -o k8s-node-v1.17.2.tar [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# ll total 170028 -rw------- 1 root root 174108672 Feb 4 20:18 k8s-node-v1.17.2.tar [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# ll -h total 167M -rw------- 1 root root 167M Feb 4 20:18 k8s-node-v1.17.2.tar [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# scp k8s-node-v1.17.2.tar node201.yinzhengjie.org.cn:~ root@node201.yinzhengjie.org.cn's password: k8s-node-v1.17.2.tar 100% 166MB 119.7MB/s 00:01 [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# scp k8s-node-v1.17.2.tar node202.yinzhengjie.org.cn:~ root@node202.yinzhengjie.org.cn's password: k8s-node-v1.17.2.tar 100% 166MB 118.3MB/s 00:01 [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# scp k8s-node-v1.17.2.tar node203.yinzhengjie.org.cn:~ root@node203.yinzhengjie.org.cn's password: k8s-node-v1.17.2.tar 100% 166MB 122.9MB/s 00:01 [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]#
8>.各個k8s node節點導入鏡像
[root@node201.yinzhengjie.org.cn ~]# ll total 170028 -rw------- 1 root root 174108672 Feb 4 20:19 k8s-node-v1.17.2.tar [root@node201.yinzhengjie.org.cn ~]# [root@node201.yinzhengjie.org.cn ~]# docker image load -i k8s-node-v1.17.2.tar 7bff100f35cb: Loading layer [==================================================>] 4.672MB/4.672MB 5d3f68f6da8f: Loading layer [==================================================>] 9.526MB/9.526MB 9b48060f404d: Loading layer [==================================================>] 5.912MB/5.912MB 3f3a4ce2b719: Loading layer [==================================================>] 35.25MB/35.25MB 9ce0bb155166: Loading layer [==================================================>] 5.12kB/5.12kB Loaded image: quay.io/coreos/flannel:v0.11.0-amd64 fc4976bd934b: Loading layer [==================================================>] 53.88MB/53.88MB 682fbb19de80: Loading layer [==================================================>] 21.06MB/21.06MB 2dc2f2423ad1: Loading layer [==================================================>] 5.168MB/5.168MB ad9fb2411669: Loading layer [==================================================>] 4.608kB/4.608kB 597151d24476: Loading layer [==================================================>] 8.192kB/8.192kB 0d8d54147a3a: Loading layer [==================================================>] 8.704kB/8.704kB ca7fe3329548: Loading layer [==================================================>] 37.81MB/37.81MB Loaded image: k8s.gcr.io/kube-proxy:v1.17.2 e17133b79956: Loading layer [==================================================>] 744.4kB/744.4kB Loaded image: k8s.gcr.io/pause:3.1 [root@node201.yinzhengjie.org.cn ~]# [root@node201.yinzhengjie.org.cn ~]# docker image ls REPOSITORY TAG IMAGE ID CREATED SIZE k8s.gcr.io/kube-proxy v1.17.2 cba2a99699bd 2 weeks ago 116MB quay.io/coreos/flannel v0.11.0-amd64 ff281650a721 12 months ago 52.6MB k8s.gcr.io/pause 3.1 da86e6ba6ca1 2 years ago 742kB [root@node201.yinzhengjie.org.cn ~]# [root@node201.yinzhengjie.org.cn ~]#
[root@node202.yinzhengjie.org.cn ~]# ll total 170028 -rw------- 1 root root 174108672 Feb 4 20:20 k8s-node-v1.17.2.tar [root@node202.yinzhengjie.org.cn ~]# [root@node202.yinzhengjie.org.cn ~]# docker image ls REPOSITORY TAG IMAGE ID CREATED SIZE [root@node202.yinzhengjie.org.cn ~]# [root@node202.yinzhengjie.org.cn ~]# docker image load -i k8s-node-v1.17.2.tar 7bff100f35cb: Loading layer [==================================================>] 4.672MB/4.672MB 5d3f68f6da8f: Loading layer [==================================================>] 9.526MB/9.526MB 9b48060f404d: Loading layer [==================================================>] 5.912MB/5.912MB 3f3a4ce2b719: Loading layer [==================================================>] 35.25MB/35.25MB 9ce0bb155166: Loading layer [==================================================>] 5.12kB/5.12kB Loaded image: quay.io/coreos/flannel:v0.11.0-amd64 fc4976bd934b: Loading layer [==================================================>] 53.88MB/53.88MB 682fbb19de80: Loading layer [==================================================>] 21.06MB/21.06MB 2dc2f2423ad1: Loading layer [==================================================>] 5.168MB/5.168MB ad9fb2411669: Loading layer [==================================================>] 4.608kB/4.608kB 597151d24476: Loading layer [==================================================>] 8.192kB/8.192kB 0d8d54147a3a: Loading layer [==================================================>] 8.704kB/8.704kB ca7fe3329548: Loading layer [==================================================>] 37.81MB/37.81MB Loaded image: k8s.gcr.io/kube-proxy:v1.17.2 e17133b79956: Loading layer [==================================================>] 744.4kB/744.4kB Loaded image: k8s.gcr.io/pause:3.1 [root@node202.yinzhengjie.org.cn ~]# [root@node202.yinzhengjie.org.cn ~]# [root@node202.yinzhengjie.org.cn ~]# ll total 170028 -rw------- 1 root root 174108672 Feb 4 20:20 k8s-node-v1.17.2.tar [root@node202.yinzhengjie.org.cn ~]# [root@node202.yinzhengjie.org.cn ~]# docker image ls REPOSITORY TAG IMAGE ID CREATED SIZE k8s.gcr.io/kube-proxy v1.17.2 cba2a99699bd 2 weeks ago 116MB quay.io/coreos/flannel v0.11.0-amd64 ff281650a721 12 months ago 52.6MB k8s.gcr.io/pause 3.1 da86e6ba6ca1 2 years ago 742kB [root@node202.yinzhengjie.org.cn ~]# [root@node202.yinzhengjie.org.cn ~]#
[root@node203.yinzhengjie.org.cn ~]# docker image ls REPOSITORY TAG IMAGE ID CREATED SIZE [root@node203.yinzhengjie.org.cn ~]# [root@node203.yinzhengjie.org.cn ~]# ll total 170028 -rw------- 1 root root 174108672 Feb 4 20:20 k8s-node-v1.17.2.tar [root@node203.yinzhengjie.org.cn ~]# [root@node203.yinzhengjie.org.cn ~]# docker image load -i k8s-node-v1.17.2.tar 7bff100f35cb: Loading layer [==================================================>] 4.672MB/4.672MB 5d3f68f6da8f: Loading layer [==================================================>] 9.526MB/9.526MB 9b48060f404d: Loading layer [==================================================>] 5.912MB/5.912MB 3f3a4ce2b719: Loading layer [==================================================>] 35.25MB/35.25MB 9ce0bb155166: Loading layer [==================================================>] 5.12kB/5.12kB Loaded image: quay.io/coreos/flannel:v0.11.0-amd64 fc4976bd934b: Loading layer [==================================================>] 53.88MB/53.88MB 682fbb19de80: Loading layer [==================================================>] 21.06MB/21.06MB 2dc2f2423ad1: Loading layer [==================================================>] 5.168MB/5.168MB ad9fb2411669: Loading layer [==================================================>] 4.608kB/4.608kB 597151d24476: Loading layer [==================================================>] 8.192kB/8.192kB 0d8d54147a3a: Loading layer [==================================================>] 8.704kB/8.704kB ca7fe3329548: Loading layer [==================================================>] 37.81MB/37.81MB Loaded image: k8s.gcr.io/kube-proxy:v1.17.2 e17133b79956: Loading layer [==================================================>] 744.4kB/744.4kB Loaded image: k8s.gcr.io/pause:3.1 [root@node203.yinzhengjie.org.cn ~]# [root@node203.yinzhengjie.org.cn ~]# docker image ls REPOSITORY TAG IMAGE ID CREATED SIZE k8s.gcr.io/kube-proxy v1.17.2 cba2a99699bd 2 weeks ago 116MB quay.io/coreos/flannel v0.11.0-amd64 ff281650a721 12 months ago 52.6MB k8s.gcr.io/pause 3.1 da86e6ba6ca1 2 years ago 742kB [root@node203.yinzhengjie.org.cn ~]# [root@node203.yinzhengjie.org.cn ~]#
四.將k8s node節點加入到上一步初始化的k8s集群
1>.將k8s master節點的k8s軟件源拷貝到其它3個k8s node節點
[root@master200.yinzhengjie.org.cn ~]# scp /etc/yum.repos.d/kubernetes.repo node201.yinzhengjie.org.cn:/etc/yum.repos.d/kubernetes.repo root@node201.yinzhengjie.org.cn's password: kubernetes.repo 100% 275 125.6KB/s 00:00 [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# scp /etc/yum.repos.d/kubernetes.repo node202.yinzhengjie.org.cn:/etc/yum.repos.d/kubernetes.repo root@node202.yinzhengjie.org.cn's password: kubernetes.repo 100% 275 199.6KB/s 00:00 [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# scp /etc/yum.repos.d/kubernetes.repo node203.yinzhengjie.org.cn:/etc/yum.repos.d/kubernetes.repo root@node203.yinzhengjie.org.cn's password: kubernetes.repo 100% 275 63.0KB/s 00:00 [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]#
2>.三個節點均安裝k8s相關軟件包
[root@node201.yinzhengjie.org.cn ~]# yum -y install kubelet kubeadm Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile * base: mirrors.aliyun.com * extras: mirrors.aliyun.com * updates: mirrors.aliyun.com docker-ce-stable | 3.5 kB 00:00:00 kubernetes | 1.4 kB 00:00:00 kubernetes/primary | 63 kB 00:00:00 kubernetes 460/460 Resolving Dependencies --> Running transaction check ---> Package kubeadm.x86_64 0:1.17.2-0 will be installed --> Processing Dependency: kubernetes-cni >= 0.7.5 for package: kubeadm-1.17.2-0.x86_64 --> Processing Dependency: kubectl >= 1.13.0 for package: kubeadm-1.17.2-0.x86_64 --> Processing Dependency: cri-tools >= 1.13.0 for package: kubeadm-1.17.2-0.x86_64 ---> Package kubelet.x86_64 0:1.17.2-0 will be installed --> Processing Dependency: socat for package: kubelet-1.17.2-0.x86_64 --> Processing Dependency: conntrack for package: kubelet-1.17.2-0.x86_64 --> Running transaction check ---> Package conntrack-tools.x86_64 0:1.4.4-5.el7_7.2 will be installed --> Processing Dependency: libnetfilter_cttimeout.so.1(LIBNETFILTER_CTTIMEOUT_1.1)(64bit) for package: conntrack-tools-1.4.4-5.el7_7.2.x86_64 --> Processing Dependency: libnetfilter_cttimeout.so.1(LIBNETFILTER_CTTIMEOUT_1.0)(64bit) for package: conntrack-tools-1.4.4-5.el7_7.2.x86_64 --> Processing Dependency: libnetfilter_cthelper.so.0(LIBNETFILTER_CTHELPER_1.0)(64bit) for package: conntrack-tools-1.4.4-5.el7_7.2.x86_64 --> Processing Dependency: libnetfilter_queue.so.1()(64bit) for package: conntrack-tools-1.4.4-5.el7_7.2.x86_64 --> Processing Dependency: libnetfilter_cttimeout.so.1()(64bit) for package: conntrack-tools-1.4.4-5.el7_7.2.x86_64 --> Processing Dependency: libnetfilter_cthelper.so.0()(64bit) for package: conntrack-tools-1.4.4-5.el7_7.2.x86_64 ---> Package cri-tools.x86_64 0:1.13.0-0 will be installed ---> Package kubectl.x86_64 0:1.17.2-0 will be installed ---> Package kubernetes-cni.x86_64 0:0.7.5-0 will be installed ---> Package socat.x86_64 0:1.7.3.2-2.el7 will be installed --> Running transaction check ---> Package libnetfilter_cthelper.x86_64 0:1.0.0-10.el7_7.1 will be installed ---> Package libnetfilter_cttimeout.x86_64 0:1.0.0-6.el7_7.1 will be installed ---> Package libnetfilter_queue.x86_64 0:1.0.2-2.el7_2 will be installed --> Finished Dependency Resolution Dependencies Resolved ============================================================================================================================================================================================================================================================================== Package Arch Version Repository Size ============================================================================================================================================================================================================================================================================== Installing: kubeadm x86_64 1.17.2-0 kubernetes 8.7 M kubelet x86_64 1.17.2-0 kubernetes 20 M Installing for dependencies: conntrack-tools x86_64 1.4.4-5.el7_7.2 updates 187 k cri-tools x86_64 1.13.0-0 kubernetes 5.1 M kubectl x86_64 1.17.2-0 kubernetes 9.4 M kubernetes-cni x86_64 0.7.5-0 kubernetes 10 M libnetfilter_cthelper x86_64 1.0.0-10.el7_7.1 updates 18 k libnetfilter_cttimeout x86_64 1.0.0-6.el7_7.1 updates 18 k libnetfilter_queue x86_64 1.0.2-2.el7_2 base 23 k socat x86_64 1.7.3.2-2.el7 base 290 k Transaction Summary ============================================================================================================================================================================================================================================================================== Install 2 Packages (+8 Dependent packages) Total download size: 54 M Installed size: 243 M Downloading packages: (1/10): conntrack-tools-1.4.4-5.el7_7.2.x86_64.rpm | 187 kB 00:00:00 warning: /var/cache/yum/x86_64/7/kubernetes/packages/14bfe6e75a9efc8eca3f638eb22c7e2ce759c67f95b43b16fae4ebabde1549f3-cri-tools-1.13.0-0.x86_64.rpm: Header V4 RSA/SHA512 Signature, key ID 3e1ba8d5: NOKEY ] 3.5 MB/s | 6.9 MB 00:00:13 ETA Public key for 14bfe6e75a9efc8eca3f638eb22c7e2ce759c67f95b43b16fae4ebabde1549f3-cri-tools-1.13.0-0.x86_64.rpm is not installed (2/10): 14bfe6e75a9efc8eca3f638eb22c7e2ce759c67f95b43b16fae4ebabde1549f3-cri-tools-1.13.0-0.x86_64.rpm | 5.1 MB 00:00:01 (3/10): 105d89f0607c7baf91305ba352e78000bd20aad5cdf706bffff3b31cd546dbf3-kubeadm-1.17.2-0.x86_64.rpm | 8.7 MB 00:00:02 (4/10): b44630896c69cd411db53be1d5cb5ae899a40aba7c0766317ea904390fcfc45b-kubectl-1.17.2-0.x86_64.rpm | 9.4 MB 00:00:01 (5/10): libnetfilter_cttimeout-1.0.0-6.el7_7.1.x86_64.rpm | 18 kB 00:00:00 (6/10): libnetfilter_queue-1.0.2-2.el7_2.x86_64.rpm | 23 kB 00:00:00 (7/10): 548a0dcd865c16a50980420ddfa5fbccb8b59621179798e6dc905c9bf8af3b34-kubernetes-cni-0.7.5-0.x86_64.rpm | 10 MB 00:00:01 (8/10): socat-1.7.3.2-2.el7.x86_64.rpm | 290 kB 00:00:01 (9/10): libnetfilter_cthelper-1.0.0-10.el7_7.1.x86_64.rpm | 18 kB 00:00:02 (10/10): 3ee7f2dff78e6fbb3ac3af8acb1a907f4bec1b1ef4cf627cbe02fa553707f2e9-kubelet-1.17.2-0.x86_64.rpm | 20 MB 00:00:04 ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Total 8.6 MB/s | 54 MB 00:00:06 Retrieving key from https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg Importing GPG key 0x3E1BA8D5: Userid : "Google Cloud Packages RPM Signing Key <gc-team@google.com>" Fingerprint: 3749 e1ba 95a8 6ce0 5454 6ed2 f09c 394c 3e1b a8d5 From : https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg Retrieving key from https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg Importing GPG key 0xA7317B0F: Userid : "Google Cloud Packages Automatic Signing Key <gc-team@google.com>" Fingerprint: d0bc 747f d8ca f711 7500 d6fa 3746 c208 a731 7b0f From : https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg Running transaction check Running transaction test Transaction test succeeded Running transaction Installing : libnetfilter_cttimeout-1.0.0-6.el7_7.1.x86_64 1/10 Installing : socat-1.7.3.2-2.el7.x86_64 2/10 Installing : cri-tools-1.13.0-0.x86_64 3/10 Installing : kubectl-1.17.2-0.x86_64 4/10 Installing : libnetfilter_queue-1.0.2-2.el7_2.x86_64 5/10 Installing : libnetfilter_cthelper-1.0.0-10.el7_7.1.x86_64 6/10 Installing : conntrack-tools-1.4.4-5.el7_7.2.x86_64 7/10 Installing : kubernetes-cni-0.7.5-0.x86_64 8/10 Installing : kubelet-1.17.2-0.x86_64 9/10 Installing : kubeadm-1.17.2-0.x86_64 10/10 Verifying : libnetfilter_cthelper-1.0.0-10.el7_7.1.x86_64 1/10 Verifying : conntrack-tools-1.4.4-5.el7_7.2.x86_64 2/10 Verifying : libnetfilter_queue-1.0.2-2.el7_2.x86_64 3/10 Verifying : kubectl-1.17.2-0.x86_64 4/10 Verifying : cri-tools-1.13.0-0.x86_64 5/10 Verifying : kubeadm-1.17.2-0.x86_64 6/10 Verifying : kubernetes-cni-0.7.5-0.x86_64 7/10 Verifying : socat-1.7.3.2-2.el7.x86_64 8/10 Verifying : kubelet-1.17.2-0.x86_64 9/10 Verifying : libnetfilter_cttimeout-1.0.0-6.el7_7.1.x86_64 10/10 Installed: kubeadm.x86_64 0:1.17.2-0 kubelet.x86_64 0:1.17.2-0 Dependency Installed: conntrack-tools.x86_64 0:1.4.4-5.el7_7.2 cri-tools.x86_64 0:1.13.0-0 kubectl.x86_64 0:1.17.2-0 kubernetes-cni.x86_64 0:0.7.5-0 libnetfilter_cthelper.x86_64 0:1.0.0-10.el7_7.1 libnetfilter_cttimeout.x86_64 0:1.0.0-6.el7_7.1 libnetfilter_queue.x86_64 0:1.0.2-2.el7_2 socat.x86_64 0:1.7.3.2-2.el7 Complete! [root@node201.yinzhengjie.org.cn ~]#
[root@node202.yinzhengjie.org.cn ~]# yum -y install kubelet kubeadm Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile * base: mirror.bit.edu.cn * extras: mirrors.aliyun.com * updates: mirrors.aliyun.com docker-ce-stable | 3.5 kB 00:00:00 kubernetes | 1.4 kB 00:00:00 kubernetes/primary | 63 kB 00:00:00 kubernetes 460/460 Resolving Dependencies --> Running transaction check ---> Package kubeadm.x86_64 0:1.17.2-0 will be installed --> Processing Dependency: kubernetes-cni >= 0.7.5 for package: kubeadm-1.17.2-0.x86_64 --> Processing Dependency: kubectl >= 1.13.0 for package: kubeadm-1.17.2-0.x86_64 --> Processing Dependency: cri-tools >= 1.13.0 for package: kubeadm-1.17.2-0.x86_64 ---> Package kubelet.x86_64 0:1.17.2-0 will be installed --> Processing Dependency: socat for package: kubelet-1.17.2-0.x86_64 --> Processing Dependency: conntrack for package: kubelet-1.17.2-0.x86_64 --> Running transaction check ---> Package conntrack-tools.x86_64 0:1.4.4-5.el7_7.2 will be installed --> Processing Dependency: libnetfilter_cttimeout.so.1(LIBNETFILTER_CTTIMEOUT_1.1)(64bit) for package: conntrack-tools-1.4.4-5.el7_7.2.x86_64 --> Processing Dependency: libnetfilter_cttimeout.so.1(LIBNETFILTER_CTTIMEOUT_1.0)(64bit) for package: conntrack-tools-1.4.4-5.el7_7.2.x86_64 --> Processing Dependency: libnetfilter_cthelper.so.0(LIBNETFILTER_CTHELPER_1.0)(64bit) for package: conntrack-tools-1.4.4-5.el7_7.2.x86_64 --> Processing Dependency: libnetfilter_queue.so.1()(64bit) for package: conntrack-tools-1.4.4-5.el7_7.2.x86_64 --> Processing Dependency: libnetfilter_cttimeout.so.1()(64bit) for package: conntrack-tools-1.4.4-5.el7_7.2.x86_64 --> Processing Dependency: libnetfilter_cthelper.so.0()(64bit) for package: conntrack-tools-1.4.4-5.el7_7.2.x86_64 ---> Package cri-tools.x86_64 0:1.13.0-0 will be installed ---> Package kubectl.x86_64 0:1.17.2-0 will be installed ---> Package kubernetes-cni.x86_64 0:0.7.5-0 will be installed ---> Package socat.x86_64 0:1.7.3.2-2.el7 will be installed --> Running transaction check ---> Package libnetfilter_cthelper.x86_64 0:1.0.0-10.el7_7.1 will be installed ---> Package libnetfilter_cttimeout.x86_64 0:1.0.0-6.el7_7.1 will be installed ---> Package libnetfilter_queue.x86_64 0:1.0.2-2.el7_2 will be installed --> Finished Dependency Resolution Dependencies Resolved ============================================================================================================================================================================================================================================================================== Package Arch Version Repository Size ============================================================================================================================================================================================================================================================================== Installing: kubeadm x86_64 1.17.2-0 kubernetes 8.7 M kubelet x86_64 1.17.2-0 kubernetes 20 M Installing for dependencies: conntrack-tools x86_64 1.4.4-5.el7_7.2 updates 187 k cri-tools x86_64 1.13.0-0 kubernetes 5.1 M kubectl x86_64 1.17.2-0 kubernetes 9.4 M kubernetes-cni x86_64 0.7.5-0 kubernetes 10 M libnetfilter_cthelper x86_64 1.0.0-10.el7_7.1 updates 18 k libnetfilter_cttimeout x86_64 1.0.0-6.el7_7.1 updates 18 k libnetfilter_queue x86_64 1.0.2-2.el7_2 base 23 k socat x86_64 1.7.3.2-2.el7 base 290 k Transaction Summary ============================================================================================================================================================================================================================================================================== Install 2 Packages (+8 Dependent packages) Total download size: 54 M Installed size: 243 M Downloading packages: (1/10): conntrack-tools-1.4.4-5.el7_7.2.x86_64.rpm | 187 kB 00:00:00 warning: /var/cache/yum/x86_64/7/kubernetes/packages/14bfe6e75a9efc8eca3f638eb22c7e2ce759c67f95b43b16fae4ebabde1549f3-cri-tools-1.13.0-0.x86_64.rpm: Header V4 RSA/SHA512 Signature, key ID 3e1ba8d5: NOKEY ] 1.8 MB/s | 3.7 MB 00:00:28 ETA Public key for 14bfe6e75a9efc8eca3f638eb22c7e2ce759c67f95b43b16fae4ebabde1549f3-cri-tools-1.13.0-0.x86_64.rpm is not installed (2/10): 14bfe6e75a9efc8eca3f638eb22c7e2ce759c67f95b43b16fae4ebabde1549f3-cri-tools-1.13.0-0.x86_64.rpm | 5.1 MB 00:00:01 (3/10): b44630896c69cd411db53be1d5cb5ae899a40aba7c0766317ea904390fcfc45b-kubectl-1.17.2-0.x86_64.rpm | 9.4 MB 00:00:01 (4/10): 3ee7f2dff78e6fbb3ac3af8acb1a907f4bec1b1ef4cf627cbe02fa553707f2e9-kubelet-1.17.2-0.x86_64.rpm | 20 MB 00:00:02 (5/10): libnetfilter_cthelper-1.0.0-10.el7_7.1.x86_64.rpm | 18 kB 00:00:00 (6/10): libnetfilter_cttimeout-1.0.0-6.el7_7.1.x86_64.rpm | 18 kB 00:00:00 (7/10): libnetfilter_queue-1.0.2-2.el7_2.x86_64.rpm | 23 kB 00:00:00 (8/10): socat-1.7.3.2-2.el7.x86_64.rpm | 290 kB 00:00:00 (9/10): 548a0dcd865c16a50980420ddfa5fbccb8b59621179798e6dc905c9bf8af3b34-kubernetes-cni-0.7.5-0.x86_64.rpm | 10 MB 00:00:01 (10/10): 105d89f0607c7baf91305ba352e78000bd20aad5cdf706bffff3b31cd546dbf3-kubeadm-1.17.2-0.x86_64.rpm | 8.7 MB 00:00:18 ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Total 2.9 MB/s | 54 MB 00:00:18 Retrieving key from https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg Importing GPG key 0x3E1BA8D5: Userid : "Google Cloud Packages RPM Signing Key <gc-team@google.com>" Fingerprint: 3749 e1ba 95a8 6ce0 5454 6ed2 f09c 394c 3e1b a8d5 From : https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg Retrieving key from https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg Importing GPG key 0xA7317B0F: Userid : "Google Cloud Packages Automatic Signing Key <gc-team@google.com>" Fingerprint: d0bc 747f d8ca f711 7500 d6fa 3746 c208 a731 7b0f From : https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg Running transaction check Running transaction test Transaction test succeeded Running transaction Installing : libnetfilter_cttimeout-1.0.0-6.el7_7.1.x86_64 1/10 Installing : socat-1.7.3.2-2.el7.x86_64 2/10 Installing : cri-tools-1.13.0-0.x86_64 3/10 Installing : kubectl-1.17.2-0.x86_64 4/10 Installing : libnetfilter_queue-1.0.2-2.el7_2.x86_64 5/10 Installing : libnetfilter_cthelper-1.0.0-10.el7_7.1.x86_64 6/10 Installing : conntrack-tools-1.4.4-5.el7_7.2.x86_64 7/10 Installing : kubernetes-cni-0.7.5-0.x86_64 8/10 Installing : kubelet-1.17.2-0.x86_64 9/10 Installing : kubeadm-1.17.2-0.x86_64 10/10 Verifying : libnetfilter_cthelper-1.0.0-10.el7_7.1.x86_64 1/10 Verifying : conntrack-tools-1.4.4-5.el7_7.2.x86_64 2/10 Verifying : libnetfilter_queue-1.0.2-2.el7_2.x86_64 3/10 Verifying : kubectl-1.17.2-0.x86_64 4/10 Verifying : cri-tools-1.13.0-0.x86_64 5/10 Verifying : kubeadm-1.17.2-0.x86_64 6/10 Verifying : kubernetes-cni-0.7.5-0.x86_64 7/10 Verifying : socat-1.7.3.2-2.el7.x86_64 8/10 Verifying : kubelet-1.17.2-0.x86_64 9/10 Verifying : libnetfilter_cttimeout-1.0.0-6.el7_7.1.x86_64 10/10 Installed: kubeadm.x86_64 0:1.17.2-0 kubelet.x86_64 0:1.17.2-0 Dependency Installed: conntrack-tools.x86_64 0:1.4.4-5.el7_7.2 cri-tools.x86_64 0:1.13.0-0 kubectl.x86_64 0:1.17.2-0 kubernetes-cni.x86_64 0:0.7.5-0 libnetfilter_cthelper.x86_64 0:1.0.0-10.el7_7.1 libnetfilter_cttimeout.x86_64 0:1.0.0-6.el7_7.1 libnetfilter_queue.x86_64 0:1.0.2-2.el7_2 socat.x86_64 0:1.7.3.2-2.el7 Complete! [root@node202.yinzhengjie.org.cn ~]#
[root@node203.yinzhengjie.org.cn ~]# yum -y install kubelet kubeadm Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile * base: mirrors.aliyun.com * extras: mirrors.aliyun.com * updates: mirrors.aliyun.com docker-ce-stable | 3.5 kB 00:00:00 kubernetes | 1.4 kB 00:00:00 kubernetes/primary | 63 kB 00:00:01 kubernetes 460/460 Resolving Dependencies --> Running transaction check ---> Package kubeadm.x86_64 0:1.17.2-0 will be installed --> Processing Dependency: kubernetes-cni >= 0.7.5 for package: kubeadm-1.17.2-0.x86_64 --> Processing Dependency: kubectl >= 1.13.0 for package: kubeadm-1.17.2-0.x86_64 --> Processing Dependency: cri-tools >= 1.13.0 for package: kubeadm-1.17.2-0.x86_64 ---> Package kubelet.x86_64 0:1.17.2-0 will be installed --> Processing Dependency: socat for package: kubelet-1.17.2-0.x86_64 --> Processing Dependency: conntrack for package: kubelet-1.17.2-0.x86_64 --> Running transaction check ---> Package conntrack-tools.x86_64 0:1.4.4-5.el7_7.2 will be installed --> Processing Dependency: libnetfilter_cttimeout.so.1(LIBNETFILTER_CTTIMEOUT_1.1)(64bit) for package: conntrack-tools-1.4.4-5.el7_7.2.x86_64 --> Processing Dependency: libnetfilter_cttimeout.so.1(LIBNETFILTER_CTTIMEOUT_1.0)(64bit) for package: conntrack-tools-1.4.4-5.el7_7.2.x86_64 --> Processing Dependency: libnetfilter_cthelper.so.0(LIBNETFILTER_CTHELPER_1.0)(64bit) for package: conntrack-tools-1.4.4-5.el7_7.2.x86_64 --> Processing Dependency: libnetfilter_queue.so.1()(64bit) for package: conntrack-tools-1.4.4-5.el7_7.2.x86_64 --> Processing Dependency: libnetfilter_cttimeout.so.1()(64bit) for package: conntrack-tools-1.4.4-5.el7_7.2.x86_64 --> Processing Dependency: libnetfilter_cthelper.so.0()(64bit) for package: conntrack-tools-1.4.4-5.el7_7.2.x86_64 ---> Package cri-tools.x86_64 0:1.13.0-0 will be installed ---> Package kubectl.x86_64 0:1.17.2-0 will be installed ---> Package kubernetes-cni.x86_64 0:0.7.5-0 will be installed ---> Package socat.x86_64 0:1.7.3.2-2.el7 will be installed --> Running transaction check ---> Package libnetfilter_cthelper.x86_64 0:1.0.0-10.el7_7.1 will be installed ---> Package libnetfilter_cttimeout.x86_64 0:1.0.0-6.el7_7.1 will be installed ---> Package libnetfilter_queue.x86_64 0:1.0.2-2.el7_2 will be installed --> Finished Dependency Resolution Dependencies Resolved ============================================================================================================================================================================================================================================================================== Package Arch Version Repository Size ============================================================================================================================================================================================================================================================================== Installing: kubeadm x86_64 1.17.2-0 kubernetes 8.7 M kubelet x86_64 1.17.2-0 kubernetes 20 M Installing for dependencies: conntrack-tools x86_64 1.4.4-5.el7_7.2 updates 187 k cri-tools x86_64 1.13.0-0 kubernetes 5.1 M kubectl x86_64 1.17.2-0 kubernetes 9.4 M kubernetes-cni x86_64 0.7.5-0 kubernetes 10 M libnetfilter_cthelper x86_64 1.0.0-10.el7_7.1 updates 18 k libnetfilter_cttimeout x86_64 1.0.0-6.el7_7.1 updates 18 k libnetfilter_queue x86_64 1.0.2-2.el7_2 base 23 k socat x86_64 1.7.3.2-2.el7 base 290 k Transaction Summary ============================================================================================================================================================================================================================================================================== Install 2 Packages (+8 Dependent packages) Total download size: 54 M Installed size: 243 M Downloading packages: (1/10): conntrack-tools-1.4.4-5.el7_7.2.x86_64.rpm | 187 kB 00:00:00 warning: /var/cache/yum/x86_64/7/kubernetes/packages/14bfe6e75a9efc8eca3f638eb22c7e2ce759c67f95b43b16fae4ebabde1549f3-cri-tools-1.13.0-0.x86_64.rpm: Header V4 RSA/SHA512 Signature, key ID 3e1ba8d5: NOKEY ] 1.4 MB/s | 7.3 MB 00:00:34 ETA Public key for 14bfe6e75a9efc8eca3f638eb22c7e2ce759c67f95b43b16fae4ebabde1549f3-cri-tools-1.13.0-0.x86_64.rpm is not installed (2/10): 14bfe6e75a9efc8eca3f638eb22c7e2ce759c67f95b43b16fae4ebabde1549f3-cri-tools-1.13.0-0.x86_64.rpm | 5.1 MB 00:00:03 (3/10): 105d89f0607c7baf91305ba352e78000bd20aad5cdf706bffff3b31cd546dbf3-kubeadm-1.17.2-0.x86_64.rpm | 8.7 MB 00:00:03 (4/10): b44630896c69cd411db53be1d5cb5ae899a40aba7c0766317ea904390fcfc45b-kubectl-1.17.2-0.x86_64.rpm | 9.4 MB 00:00:02 (5/10): libnetfilter_cthelper-1.0.0-10.el7_7.1.x86_64.rpm | 18 kB 00:00:00 (6/10): libnetfilter_cttimeout-1.0.0-6.el7_7.1.x86_64.rpm | 18 kB 00:00:00 (7/10): libnetfilter_queue-1.0.2-2.el7_2.x86_64.rpm | 23 kB 00:00:00 (8/10): socat-1.7.3.2-2.el7.x86_64.rpm | 290 kB 00:00:00 (9/10): 548a0dcd865c16a50980420ddfa5fbccb8b59621179798e6dc905c9bf8af3b34-kubernetes-cni-0.7.5-0.x86_64.rpm | 10 MB 00:00:03 (10/10): 3ee7f2dff78e6fbb3ac3af8acb1a907f4bec1b1ef4cf627cbe02fa553707f2e9-kubelet-1.17.2-0.x86_64.rpm | 20 MB 00:00:06 ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Total 5.3 MB/s | 54 MB 00:00:10 Retrieving key from https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg Importing GPG key 0x3E1BA8D5: Userid : "Google Cloud Packages RPM Signing Key <gc-team@google.com>" Fingerprint: 3749 e1ba 95a8 6ce0 5454 6ed2 f09c 394c 3e1b a8d5 From : https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg Retrieving key from https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg Importing GPG key 0xA7317B0F: Userid : "Google Cloud Packages Automatic Signing Key <gc-team@google.com>" Fingerprint: d0bc 747f d8ca f711 7500 d6fa 3746 c208 a731 7b0f From : https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg Running transaction check Running transaction test Transaction test succeeded Running transaction Installing : libnetfilter_cttimeout-1.0.0-6.el7_7.1.x86_64 1/10 Installing : socat-1.7.3.2-2.el7.x86_64 2/10 Installing : cri-tools-1.13.0-0.x86_64 3/10 Installing : kubectl-1.17.2-0.x86_64 4/10 Installing : libnetfilter_queue-1.0.2-2.el7_2.x86_64 5/10 Installing : libnetfilter_cthelper-1.0.0-10.el7_7.1.x86_64 6/10 Installing : conntrack-tools-1.4.4-5.el7_7.2.x86_64 7/10 Installing : kubernetes-cni-0.7.5-0.x86_64 8/10 Installing : kubelet-1.17.2-0.x86_64 9/10 Installing : kubeadm-1.17.2-0.x86_64 10/10 Verifying : libnetfilter_cthelper-1.0.0-10.el7_7.1.x86_64 1/10 Verifying : conntrack-tools-1.4.4-5.el7_7.2.x86_64 2/10 Verifying : libnetfilter_queue-1.0.2-2.el7_2.x86_64 3/10 Verifying : kubectl-1.17.2-0.x86_64 4/10 Verifying : cri-tools-1.13.0-0.x86_64 5/10 Verifying : kubeadm-1.17.2-0.x86_64 6/10 Verifying : kubernetes-cni-0.7.5-0.x86_64 7/10 Verifying : socat-1.7.3.2-2.el7.x86_64 8/10 Verifying : kubelet-1.17.2-0.x86_64 9/10 Verifying : libnetfilter_cttimeout-1.0.0-6.el7_7.1.x86_64 10/10 Installed: kubeadm.x86_64 0:1.17.2-0 kubelet.x86_64 0:1.17.2-0 Dependency Installed: conntrack-tools.x86_64 0:1.4.4-5.el7_7.2 cri-tools.x86_64 0:1.13.0-0 kubectl.x86_64 0:1.17.2-0 kubernetes-cni.x86_64 0:0.7.5-0 libnetfilter_cthelper.x86_64 0:1.0.0-10.el7_7.1 libnetfilter_cttimeout.x86_64 0:1.0.0-6.el7_7.1 libnetfilter_queue.x86_64 0:1.0.2-2.el7_2 socat.x86_64 0:1.7.3.2-2.el7 Complete! [root@node203.yinzhengjie.org.cn ~]#
[root@node201.yinzhengjie.org.cn ~]# systemctl list-unit-files | grep kubelet kubelet.service disabled [root@node201.yinzhengjie.org.cn ~]# [root@node201.yinzhengjie.org.cn ~]# systemctl enable kubelet.service Created symlink from /etc/systemd/system/multi-user.target.wants/kubelet.service to /usr/lib/systemd/system/kubelet.service. [root@node201.yinzhengjie.org.cn ~]# [root@node201.yinzhengjie.org.cn ~]# systemctl list-unit-files | grep kubelet kubelet.service enabled [root@node201.yinzhengjie.org.cn ~]# [root@node201.yinzhengjie.org.cn ~]#
[root@node202.yinzhengjie.org.cn ~]# systemctl list-unit-files | grep kubelet kubelet.service disabled [root@node202.yinzhengjie.org.cn ~]# [root@node202.yinzhengjie.org.cn ~]# systemctl enable kubelet.service Created symlink from /etc/systemd/system/multi-user.target.wants/kubelet.service to /usr/lib/systemd/system/kubelet.service. [root@node202.yinzhengjie.org.cn ~]# [root@node202.yinzhengjie.org.cn ~]# systemctl list-unit-files | grep kubelet kubelet.service enabled [root@node202.yinzhengjie.org.cn ~]# [root@node202.yinzhengjie.org.cn ~]#
[root@node203.yinzhengjie.org.cn ~]# systemctl list-unit-files | grep kubelet kubelet.service disabled [root@node203.yinzhengjie.org.cn ~]# [root@node203.yinzhengjie.org.cn ~]# systemctl enable kubelet.service Created symlink from /etc/systemd/system/multi-user.target.wants/kubelet.service to /usr/lib/systemd/system/kubelet.service. [root@node203.yinzhengjie.org.cn ~]# [root@node203.yinzhengjie.org.cn ~]# systemctl list-unit-files | grep kubelet kubelet.service enabled [root@node203.yinzhengjie.org.cn ~]# [root@node203.yinzhengjie.org.cn ~]#
3>.將k8s master節點的"/etc/sysconfig/kubelet"拷貝到其它三個k8s node節點(如果你的節點已經禁用了swap分區，此步驟可直接忽略)
[root@master200.yinzhengjie.org.cn ~]# cat /etc/sysconfig/kubelet KUBELET_EXTRA_ARGS="--fail-swap-on=false" [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# scp /etc/sysconfig/kubelet node201.yinzhengjie.org.cn:/etc/sysconfig/kubelet root@node201.yinzhengjie.org.cn's password: kubelet 100% 42 23.3KB/s 00:00 [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# scp /etc/sysconfig/kubelet node202.yinzhengjie.org.cn:/etc/sysconfig/kubelet root@node202.yinzhengjie.org.cn's password: kubelet 100% 42 32.2KB/s 00:00 [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# scp /etc/sysconfig/kubelet node203.yinzhengjie.org.cn:/etc/sysconfig/kubelet root@node203.yinzhengjie.org.cn's password: kubelet 100% 42 21.7KB/s 00:00 [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]#
4>.各節點加入到k8s集群
[root@node201.yinzhengjie.org.cn ~]# kubeadm join 172.200.1.200:6443 --token gu02ed.9k1rpyl4mtkhvdpk \ > --discovery-token-ca-cert-hash sha256:755ff7d07588b75aec769e566e4257d832cf6d1ec79bcdbd655dad80f30a6794 W0204 20:11:11.564558 32762 join.go:346] [preflight] WARNING: JoinControlPane.controlPlane settings will be ignored when control-plane flag is not set. [preflight] Running pre-flight checks [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/ [WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service' [preflight] Reading configuration from the cluster... [preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml' [kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.17" ConfigMap in the kube-system namespace [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" [kubelet-start] Starting the kubelet [kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap... This node has joined the cluster: * Certificate signing request was sent to apiserver and a response was received. * The Kubelet was informed of the new secure connection details. Run 'kubectl get nodes' on the control-plane to see this node join the cluster. [root@node201.yinzhengjie.org.cn ~]# [root@node201.yinzhengjie.org.cn ~]# echo $? 0 [root@node201.yinzhengjie.org.cn ~]#
[root@node202.yinzhengjie.org.cn ~]# kubeadm join 172.200.1.200:6443 --token gu02ed.9k1rpyl4mtkhvdpk --discovery-token-ca-cert-hash sha256:755ff7d07588b75aec769e566e4257d832cf6d1ec79bcdbd655dad80f30a6794 W0204 20:26:07.049488 31891 join.go:346] [preflight] WARNING: JoinControlPane.controlPlane settings will be ignored when control-plane flag is not set. [preflight] Running pre-flight checks [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/ [WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service' [preflight] Reading configuration from the cluster... [preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml' [kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.17" ConfigMap in the kube-system namespace [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" [kubelet-start] Starting the kubelet [kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap... This node has joined the cluster: * Certificate signing request was sent to apiserver and a response was received. * The Kubelet was informed of the new secure connection details. Run 'kubectl get nodes' on the control-plane to see this node join the cluster. [root@node202.yinzhengjie.org.cn ~]# [root@node202.yinzhengjie.org.cn ~]# echo $? 0 [root@node202.yinzhengjie.org.cn ~]#
[root@node203.yinzhengjie.org.cn ~]# kubeadm join 172.200.1.200:6443 --token gu02ed.9k1rpyl4mtkhvdpk --discovery-token-ca-cert-hash sha256:755ff7d07588b75aec769e566e4257d832cf6d1ec79bcdbd655dad80f30a6794 W0204 20:26:19.457993 32429 join.go:346] [preflight] WARNING: JoinControlPane.controlPlane settings will be ignored when control-plane flag is not set. [preflight] Running pre-flight checks [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/ [WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service' [preflight] Reading configuration from the cluster... [preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml' [kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.17" ConfigMap in the kube-system namespace [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" [kubelet-start] Starting the kubelet [kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap... This node has joined the cluster: * Certificate signing request was sent to apiserver and a response was received. * The Kubelet was informed of the new secure connection details. Run 'kubectl get nodes' on the control-plane to see this node join the cluster. [root@node203.yinzhengjie.org.cn ~]#
5>.將節點加入成功后,再來k8s master節點查看節點是否加入成功
[root@master200.yinzhengjie.org.cn ~]# kubectl get nodes #如果k8s master和k8s node狀態的狀態為"Ready"說明k8s集群搭建完畢啦~ NAME STATUS ROLES AGE VERSION master200.yinzhengjie.org.cn Ready master 47m v1.17.2 node201.yinzhengjie.org.cn Ready <none> 16m v1.17.2 node202.yinzhengjie.org.cn Ready <none> 78s v1.17.2 node203.yinzhengjie.org.cn Ready <none> 66s v1.17.2 [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]#
五.配置k8s node節點也可以管理api server服務器
1>.在各個k8s node節點創建"~/.kubu"目錄
[root@node201.yinzhengjie.org.cn ~]# mkdir ~/.kube [root@node201.yinzhengjie.org.cn ~]# [root@node201.yinzhengjie.org.cn ~]# ll .kube/ total 0 [root@node201.yinzhengjie.org.cn ~]# [root@node201.yinzhengjie.org.cn ~]#
[root@node202.yinzhengjie.org.cn ~]# mkdir .kube [root@node202.yinzhengjie.org.cn ~]# [root@node202.yinzhengjie.org.cn ~]# ll .kube/ total 0 [root@node202.yinzhengjie.org.cn ~]# [root@node202.yinzhengjie.org.cn ~]#
[root@node203.yinzhengjie.org.cn ~]# mkdir ~/.kube [root@node203.yinzhengjie.org.cn ~]# [root@node203.yinzhengjie.org.cn ~]# ll .kube/ total 0 [root@node203.yinzhengjie.org.cn ~]# [root@node203.yinzhengjie.org.cn ~]#
2>.將k8s master節點的"/etc/kubernetes/admin.conf"配置文件拷貝到各個k8s node節點的"~/.kube/"目錄並命名為"config"
[root@master200.yinzhengjie.org.cn ~]# scp /etc/kubernetes/admin.conf node201.yinzhengjie.org.cn:~/.kube/config root@node201.yinzhengjie.org.cn's password: admin.conf 100% 5453 2.2MB/s 00:00 [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]#
[root@master200.yinzhengjie.org.cn ~]# scp /etc/kubernetes/admin.conf node202.yinzhengjie.org.cn:~/.kube/config root@node202.yinzhengjie.org.cn's password: admin.conf 100% 5453 3.6MB/s 00:00 [root@master200.yinzhengjie.org.cn ~]#
[root@master200.yinzhengjie.org.cn ~]# scp /etc/kubernetes/admin.conf node203.yinzhengjie.org.cn:~/.kube/config root@node203.yinzhengjie.org.cn's password: admin.conf 100% 5453 2.9MB/s 00:00 [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]#
3>.在k8s node節點查看node信息
[root@node201.yinzhengjie.org.cn ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION master200.yinzhengjie.org.cn Ready master 61m v1.17.2 node201.yinzhengjie.org.cn Ready <none> 29m v1.17.2 node202.yinzhengjie.org.cn Ready <none> 15m v1.17.2 node203.yinzhengjie.org.cn Ready <none> 14m v1.17.2 [root@node201.yinzhengjie.org.cn ~]# [root@node201.yinzhengjie.org.cn ~]# [root@node201.yinzhengjie.org.cn ~]# kubectl config view apiVersion: v1 clusters: - cluster: certificate-authority-data: DATA+OMITTED server: https://172.200.1.200:6443 name: kubernetes contexts: - context: cluster: kubernetes user: kubernetes-admin name: kubernetes-admin@kubernetes current-context: kubernetes-admin@kubernetes kind: Config preferences: {} users: - name: kubernetes-admin user: client-certificate-data: REDACTED client-key-data: REDACTED [root@node201.yinzhengjie.org.cn ~]# [root@node201.yinzhengjie.org.cn ~]#
4>.API Server的證書說明
早期版本的K8S的API Server支持2個套接字，分別監聽兩個不同的端口，一個是8080(http協議),另一個是6443(https),后來官方考慮http並不安全於是采用了只保留6443端口。
API Server是需要雙向認證的，即除了API Server有自己的證書文件還要求客戶端也提供證書文件，這就是為什么上面會將"/etc/kubernetes/admin.conf"拷貝到"~/.kube/config"的原因。
我們使用kubeadmin部署K8S集群時，API Server組件需要的證書會自動幫咱們生成，默認存放目錄在"/etc/kubernetes/pki/",如下圖所示。
5>.kubernetes快速入門
博主推薦閱讀: https://www.cnblogs.com/yinzhengjie/p/12261980.html