最近kubernetes很火，加上我又在尋找適合快速搭建測試環境的方法，kubernetes的理念很適合用於測試環境的搭建。
因此在學習的過程中寫下此教程（記錄）以供回顧。
0x00 環境准備
0x00 0x00 環境要求
0x00 0x01 預備內容
0x00 0x02 安裝虛擬環境Ubuntu
0x00 0x03 404 Not Found
0x01 准備kubernetes虛擬鏡像
0x01 0x00 安裝CRI Docker
0x01 0x01 安裝kubeadm
0x01 0x02 其他要點
0x01 0x03 告一段落
0x02 master節點與Calico網絡
0x02 0x00 選擇網絡
0x02 0x01 master節點創建完成
0x03 創建普通節點並加入集群
0x03 0x00 創建普通節點
0x03 0x01 加入集群
0x00 環境准備
我的目標是：搭建擁有3節點的kubernetes集群。
0x00 0x00 環境要求
我的PC機器配置如下：
個人PC機配置
驅動精靈驅動管理硬件檢測報告 ================================================================ 電腦型號 映泰Hi-Fi H170Z3台式機 處理器 Intel(R) Core(TM) i5-6500 CPU @ 3.20GHz 內存容量 8.0GB 顯卡 NVIDIA GeForce GTX 1060 6GB 硬盤 1、Samsung SSD 750 EVO 250GB (250GB) 2、WDC WD5000AADS-00M2B0 (500GB) 3、ST1000DM003-1SB102 (1.0TB) 主板 Hi-Fi H170Z3 網卡 1、Intel(R) Ethernet Connection (2) I219-V 2、TP-LINK Wireless USB Adapter 聲卡 1、Realtek High Definition Audio 2、NVIDIA High Definition Audio 顯示器 2217 分辨率:1680x1050 當前操作系統 Windows 10 64位
具體要求沒找到，但如果使用kubeadm的構建工具倒是對每台節點ji’q有要求(2CPU和2GB內存需要划重點)：
Figure 2. kubeadm環境要求
|
特別注意
宿主機僅有8GB內存，但3節點需要 6G 的內存，加上宿主機操作系統自身需要占用的內存，我們的物理機8G內存完全不夠使用。 好在VMware上進行額外內存配置，盡量將虛擬機的2G內存映射為物理機的虛擬內存，具體步驟下面會說。 |
0x00 0x01 預備內容
下載和安裝VMware Workstation Pro 15、Ubuntu Server 18.04.2 LTS。
為了可以創建2G x3的虛擬機，需要在VMware Workstation中，“編輯→首選項→內存”中做如下額外的內存配置：
Figure 3. 額外內存配置
0x00 0x04 安裝虛擬環境Ubuntu
注意，這里要注意一下，內存要求是2GB，CPS要求是2 Core。
緊接着安裝系統咯。
配置空代理
鏡像地址用清華大學的鏡像
Mirror Address
https://mirrors.tuna.tsinghua.edu.cn/ubuntu/
重啟后安裝完成，查看IP地址后可以用SSH登陸。
0x00 0x03 404 Not Found
一般來說，我們是沒法訪問google的，在虛擬機中尤其如此。 那么可以試着用下面的方式達到目的：
修改DNS
使用google的DNS：
vim /etc/systemd/resolved.conf追加內容：
[Resolve] DNS=8.8.8.8 FallbackDNS=8.8.4.4
重新加載網卡：
systemctl restart systemd-resolved.service
配置普通代理
ubuntu中，大部分命令行的代理可以通過此配置實現。
點開我們的小飛機（最好是擁有自己的小飛機服務器），“允許來自局域網的連接”上打勾，並且觀察到代理端口為1080。
再使用ipconfig觀察虛擬機網絡中宿主機IP：
那么，可以通過如下命令啟用代理：
export http_proxy="192.168.232.1:1080" export https_proxy=${http_proxy} export ftp_proxy=${http_proxy}如果不需要代理，則退出重新登陸即可。
|
注意
配置代理后apt-get update會報錯。因此使用apt-get時需要取消代理（用APT單獨代理模式即可）。 E: The repository 'https://mirrors.tuna.tsinghua.edu.cn/ubuntu bionic Release' no longer has a Release file. N: Updating from such a repository can't be done securely, and is therefore disabled by default. |
配置APT代理
vim /etc/apt/apt.conf.d/proxy.conf增加如下內容
Acquire::http::Proxy "http://192.168.232.1:1080/"; Acquire::https::Proxy "http://192.168.232.1:1080/"; Acquire::ftp::Proxy "http://192.168.232.1:8080"; Acquire::socks5::Proxy "socks://192.168.232.1:8080/";配置Docker代理
創建代理文件：
mkdir -p /etc/systemd/system/docker.service.d/
vim /etc/systemd/system/docker.service.d/http-proxy.conf加入如下內容並保存。
[Service]
Environment="HTTP_PROXY=http://192.168.232.1:1080" "HTTPS_PROXY=http://192.168.232.1:1080" "NO_PROXY=localhost,127.0.0.1,192.168.*,10.*"重啟docker
systemctl daemon-reload
systemctl restart docker0x01 准備kubernetes虛擬鏡像
本章都參考自：https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/
0x01 0x00 安裝CRI Docker
先切換到root用戶
sudo su -
然后使用如下命令行進行Docker安裝：
# Install Docker CE ## Set up the repository: ### Install packages to allow apt to use a repository over HTTPS apt-get update && apt-get install apt-transport-https ca-certificates curl software-properties-common ### Add Docker’s official GPG key curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add - ### Add Docker apt repository. add-apt-repository \ "deb [arch=amd64] https://download.docker.com/linux/ubuntu \ $(lsb_release -cs) \ stable" ## Install Docker CE. apt-get update && apt-get install docker-ce=18.06.2~ce~3-0~ubuntu # Setup daemon. # 這里還增加了國內鏡像 cat > /etc/docker/daemon.json <<EOF { "exec-opts": ["native.cgroupdriver=systemd"], "log-driver": "json-file", "log-opts": { "max-size": "100m" }, "storage-driver": "overlay2", "registry-mirrors": ["https://registry.docker-cn.com"] } EOF mkdir -p /etc/systemd/system/docker.service.d # Restart docker. systemctl daemon-reload systemctl restart docker當然你也可以選擇 離線安裝。
為了使普通用戶也能夠使用docker命令，還需要做如下配置：
export someone=XXXXX sudo groupadd docker sudo gpasswd -a ${someone} docker sudo systemctl restart docker0x01 0x01 安裝kubeadm
同樣先切換到root用戶
sudo su -
再使用如下命令進行安裝
# 下面這幾行命令先啟用APT代理才能執行 apt-get update && apt-get install -y apt-transport-https curl curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add - cat <<EOF >/etc/apt/sources.list.d/kubernetes.list deb https://packages.cloud.google.com/apt/ kubernetes-xenial main EOF apt-get update apt-get install -y kubelet kubeadm kubectl apt-mark hold kubelet kubeadm kubectl0x01 0x02 其他要點
不要忘記第一章的要求：
Example 1. 官方文檔
-
每個節點的唯一主機名、MAC地址和product_uuid。 詳情點擊這里
-
禁用swap。為了讓kubelet正常工作，您必須禁用swap
下面命令依然是root用戶執行：
允許修改hostname
# 允許手工修改hostname cat /etc/cloud/cloud.cfg | grep preserve_hostname sed -i 's/preserve_hostname: false/preserve_hostname: true/g' /etc/cloud/cloud.cfg cat /etc/cloud/cloud.cfg | grep preserve_hostname # hostnamectl set-hostname k8s
禁用swap
sudo swapoff -a sudo vim /etc/fstab注釋掉swap並保存：
禁用swap
UUID=4e4e12d8-9f83-4863-95a7-703245fea25a / ext4 defaults 0 0 # /swap.img none swap sw 0 00x01 0x03 告一段落
通過上面的步驟，我們已經把docker、kubeadm、kubelet、kubectl都安裝好啦。
我們還在系統上做了一些准備，比如禁用swap，允許修改hostname，並配置好了各式代理。
將該機器關閉后留做鏡像，后續我們可以通過該鏡像一鍵克隆出所需要的k8s節點機器。
Figure 4. 關閉客戶機
Figure 5. 克隆機器
一定要選擇完整克隆
Figure 6. 完整克隆
我們的原始鏡像暫時不要再做修改。
0x02 master節點與Calico網絡
通過《0x02 0x02》中的說明，克隆出一個新虛擬機用作master節點。
Figure 7. 完整克隆
先修改hostname，確保集群中節點的hostname唯一。
sudo hostnamectl set-hostname k8s-node0為防止DHCP自動分配IP地址，還需要在VMware DHCP中綁定虛擬機的MAC與IP。
man@k8s:~$ ifconfig
ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 192.168.232.135 netmask 255.255.255.0 broadcast 192.168.232.255 inet6 fe80::20c:29ff:fef5:3f39 prefixlen 64 scopeid 0x20<link> ether 00:0c:29:f5:3f:39 txqueuelen 1000 (Ethernet) RX packets 4901 bytes 6136383 (6.1 MB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 2924 bytes 198870 (198.8 KB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0在宿主機C:\ProgramData\VMware\vmnetdhcp.conf中追加下列內容並保存：
host k8s-node0 {
hardware ethernet 00:0c:29:f5:3f:39; fixed-address 192.168.232.135; }然后在宿主機(win10)服務中重啟VMware DHCP Service
Figure 8. VMware DHCP Service
最后重啟節點，使hostname和靜態dhcp生效。
sudo shutdown -r now0x02 0x00 選擇網絡
必須在任何應用程序之前部署網絡。此外，在網絡安裝之前，CoreDNS不會啟動。kubeadm只支持基於容器網絡接口(CNI)的網絡(不支持kubenet)。
這里我們選擇官方推薦的 Calico（其實就是官方文檔里面的第一個選項……）。
依然是root用戶。
先進行網絡網絡配置
mkdir -p /etc/NetworkManager/conf.d/
vim /etc/NetworkManager/conf.d/calico.conf輸入如下內容並保存：
[keyfile] unmanaged-devices=interface-name:cali*;interface-name:tunl*
然后通過如下命令初始化節點：
kubeadm init --pod-network-cidr=192.168.0.0/16看到如下輸出說明初始化成功
I0407 05:27:17.562508 2814 version.go:96] could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable-1.txt": Get https://dl.k8s.io/release/stable-1.txt: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) I0407 05:27:17.562579 2814 version.go:97] falling back to the local client version: v1.14.0 [init] Using Kubernetes version: v1.14.0 [preflight] Running pre-flight checks [preflight] Pulling images required for setting up a Kubernetes cluster [preflight] This might take a minute or two, depending on the speed of your internet connection [preflight] You can also perform this action in beforehand using 'kubeadm config images pull' [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [kubelet-start] Activating the kubelet service [certs] Using certificateDir folder "/etc/kubernetes/pki" [certs] Generating "ca" certificate and key [certs] Generating "apiserver" certificate and key [certs] apiserver serving cert is signed for DNS names [k8s-node0 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.232.135] [certs] Generating "apiserver-kubelet-client" certificate and key [certs] Generating "etcd/ca" certificate and key [certs] Generating "apiserver-etcd-client" certificate and key [certs] Generating "etcd/server" certificate and key [certs] etcd/server serving cert is signed for DNS names [k8s-node0 localhost] and IPs [192.168.232.135 127.0.0.1 ::1] [certs] Generating "etcd/peer" certificate and key [certs] etcd/peer serving cert is signed for DNS names [k8s-node0 localhost] and IPs [192.168.232.135 127.0.0.1 ::1] [certs] Generating "etcd/healthcheck-client" certificate and key [certs] Generating "front-proxy-ca" certificate and key [certs] Generating "front-proxy-client" certificate and key [certs] Generating "sa" key and public key [kubeconfig] Using kubeconfig folder "/etc/kubernetes" [kubeconfig] Writing "admin.conf" kubeconfig file [kubeconfig] Writing "kubelet.conf" kubeconfig file [kubeconfig] Writing "controller-manager.conf" kubeconfig file [kubeconfig] Writing "scheduler.conf" kubeconfig file [control-plane] Using manifest folder "/etc/kubernetes/manifests" [control-plane] Creating static Pod manifest for "kube-apiserver" [control-plane] Creating static Pod manifest for "kube-controller-manager" [control-plane] Creating static Pod manifest for "kube-scheduler" [etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests" [wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s [apiclient] All control plane components are healthy after 25.023530 seconds [upload-config] storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace [kubelet] Creating a ConfigMap "kubelet-config-1.14" in namespace kube-system with the configuration for the kubelets in the cluster [upload-certs] Skipping phase. Please see --experimental-upload-certs [mark-control-plane] Marking the node k8s-node0 as control-plane by adding the label "node-role.kubernetes.io/master=''" [mark-control-plane] Marking the node k8s-node0 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule] [bootstrap-token] Using token: vjea4g.dewcrbcdmgsweysd [bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles [bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials [bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token [bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster [bootstrap-token] creating the "cluster-info" ConfigMap in the "kube-public" namespace [addons] Applied essential addon: CoreDNS [addons] Applied essential addon: kube-proxy Your Kubernetes control-plane has initialized successfully! To start using your cluster, you need to run the following as a regular user: mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config You should now deploy a pod network to the cluster. Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: https://kubernetes.io/docs/concepts/cluster-administration/addons/ Then you can join any number of worker nodes by running the following on each as root: kubeadm join 192.168.232.135:6443 --token vjea4g.dewcrbcdmgsweysd \ --discovery-token-ca-cert-hash sha256:4d545119d00c40aa66565108eb140e23a0c01ec43b6b954dc7e6e66f7bf30303然后，我們終於可以切換回普通用戶，並執行如下命令：
mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config chown $(id -u):$(id -g) $HOME/.kube/config| 如果沒有讀取/etc/kubernetes的權限，則root用戶來添加。 |
之后創建Calico網絡：
創建Calico網絡
kubectl apply -f https://docs.projectcalico.org/v3.3/getting-started/kubernetes/installation/hosted/rbac-kdd.yaml kubectl apply -f https://docs.projectcalico.org/v3.3/getting-started/kubernetes/installation/hosted/kubernetes-datastore/calico-networking/1.7/calico.yaml然后檢查master的pods服務狀態，直到所有pds都Running，說明master部署完成。
man@k8s-node0:~$ kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system calico-node-2k2wg 2/2 Running 0 113s kube-system coredns-fb8b8dccf-69lvf 1/1 Running 0 6m30s kube-system coredns-fb8b8dccf-v8k2g 1/1 Running 0 6m30s kube-system etcd-k8s-node0 1/1 Running 0 6m27s kube-system kube-apiserver-k8s-node0 1/1 Running 0 6m27s kube-system kube-controller-manager-k8s-node0 1/1 Running 0 6m27s kube-system kube-proxy-5llxx 1/1 Running 0 6m30s kube-system kube-scheduler-k8s-node0 1/1 Running 0 6m27s引用官方的一句話：
Example 2. 官方文檔
安裝了pod網絡之后，您可以通過檢查CoreDNS pod是否在kubectl get pods --all-namespaces的輸出中運行(所有名稱空間)來確認它是否正常工作。一旦CoreDNS pod啟動並運行，就可以通過連接節點繼續。
如果您的網絡不工作或核心網絡不處於運行狀態，請查看我們的 故障排除文檔。
0x02 0x01 master節點創建完成
其實執行完上面這步，單節點（master）的kubernetes已經ok。
man@k8s-node0:~$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s-node0 Ready master 8m57s v1.14.00x03 創建普通節點並加入集群
0x03 0x00 創建普通節點
同樣，從《0x02》的鏡像中克隆出新的節點機器並啟動。
Figure 9. 克隆出新的節點機器
同master節點一樣，先修改hostname，確保集群中節點的hostname唯一。
sudo hostnamectl set-hostname k8s-node1為防止DHCP自動分配IP地址，還需要在VMware DHCP中綁定虛擬機的MAC與IP。
man@k8s:~$ ifconfig
ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 192.168.232.134 netmask 255.255.255.0 broadcast 192.168.232.255 inet6 fe80::20c:29ff:fe9c:cf7b prefixlen 64 scopeid 0x20<link> ether 00:0c:29:9c:cf:7b txqueuelen 1000 (Ethernet) RX packets 34760 bytes 47115095 (47.1 MB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 14387 bytes 954662 (954.6 KB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0在宿主機C:\ProgramData\VMware\vmnetdhcp.conf中追加下列內容並保存（靜態ip改為136）：
host k8s-node1 {
hardware ethernet 00:0c:29:9c:cf:7b; fixed-address 192.168.232.136; }同master，重啟VMware DHCP Service。
最后重啟節點，使hostname和靜態dhcp生效。
sudo shutdown -r now0x03 0x01 加入集群
根據《0x02》的部署成功后打印出來的內容，我們可以在普通節點下輸入如下命令將其加入集群：
（依然要root權限）
man@k8s-node1:~$ sudo kubeadm join 192.168.232.135:6443 --token vjea4g.dewcrbcdmgsweysd --discovery-token-ca-cert-hash sha256:4d545119d00c40aa66565108eb140e23a0c01ec43b6b954dc7e6e66f7bf30303 [preflight] Running pre-flight checks [preflight] Reading configuration from the cluster... [preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml' [kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.14" ConfigMap in the kube-system namespace [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" [kubelet-start] Activating the kubelet service [kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap... This node has joined the cluster: * Certificate signing request was sent to apiserver and a response was received. * The Kubelet was informed of the new secure connection details. Run 'kubectl get nodes' on the control-plane to see this node join the cluster.如果你之前忘了記錄master的token，可以在master通過kubeadm token list命令來查看，輸出如下：
TOKEN TTL EXPIRES USAGES DESCRIPTION EXTRA GROUPS
u4qtxt.rz0v4frzay21mkvm 15h 2019-04-07T18:38:29Z authentication,signing The default bootstrap token generated by 'kubeadm init'. system:bootstrappers:kubeadm:default-node-token默認情況下，令牌在24小時后過期。如果在當前令牌過期后加入節點到集群，可以在主節點上運行kubeadm create命令創建一個新令牌，輸出類似如下:
5didvk.d09sbcov8ph2amjw如果沒有--discovery-token-ca-cert-hash值，可以在主節點上運行以下命令鏈獲得:
openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | \ openssl dgst -sha256 -hex | sed 's/^.* //'輸出會像下面這樣：
e83c9026395d7201f87a4b7cf063a6c74557f5c46951f3e8aa8ac24a6d21c04e|
注意
如果需要通過<master-ip>:<master-port>的方式指定IPv6地址，那么務必要用方括號擴起來，像這樣[fd00::101]:2073。 |
輸出如下：
[preflight] Running pre-flight checks
... (log output of join workflow) ... Node join complete: * Certificate signing request sent to master and response received. * Kubelet informed of new secure connection details. Run 'kubectl get nodes' on the master to see this machine join.之后你可以通過在master執行kubectl get nodes觀察節點是否成功加入。
man@k8s-node0:~$ kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system calico-node-2k2wg 2/2 Running 0 17m kube-system calico-node-wtkqc 2/2 Running 0 2m32s kube-system coredns-fb8b8dccf-69lvf 1/1 Running 0 21m kube-system coredns-fb8b8dccf-v8k2g 1/1 Running 0 21m kube-system etcd-k8s-node0 1/1 Running 0 21m kube-system kube-apiserver-k8s-node0 1/1 Running 0 21m kube-system kube-controller-manager-k8s-node0 1/1 Running 0 21m kube-system kube-proxy-5llxx 1/1 Running 0 21m kube-system kube-proxy-c658r 1/1 Running 0 2m32s kube-system kube-scheduler-k8s-node0 1/1 Running 0 21m man@k8s-node0:~$ kubectl get nodes NAME STATUS ROLES AGE VERSION k8s-node0 Ready master 22m v1.14.0 k8s-node1 Ready <none> 2m34s v1.14.0之后依法炮制第三個節點，最后觀察結果：
man@k8s-node0:~$ kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system calico-node-2k2wg 2/2 Running 0 40m kube-system calico-node-mwnpf 2/2 Running 0 2m12s kube-system calico-node-wtkqc 2/2 Running 0 26m kube-system coredns-fb8b8dccf-69lvf 1/1 Running 0 45m kube-system coredns-fb8b8dccf-v8k2g 1/1 Running 0 45m kube-system etcd-k8s-node0 1/1 Running 0 45m kube-system kube-apiserver-k8s-node0 1/1 Running 0 45m kube-system kube-controller-manager-k8s-node0 1/1 Running 0 45m kube-system kube-proxy-5llxx 1/1 Running 0 45m kube-system kube-proxy-c658r 1/1 Running 0 26m kube-system kube-proxy-h7lnd 1/1 Running 0 2m12s kube-system kube-scheduler-k8s-node0 1/1 Running 0 45m man@k8s-node0:~$ kubectl get nodes NAME STATUS ROLES AGE VERSION k8s-node0 Ready master 45m v1.14.0 k8s-node1 Ready <none> 26m v1.14.0 k8s-node2 Ready <none> 2m14s v1.14.0漂亮！我們終於做到了！！！
（在這過程中踩了很多坑，絕大部分是網絡的坑……學技術真不容易啊。）
最后，可憐一下我的小機器，內存已經hold不住了。
Figure 10. 內存占用
我准備去給它加內存條了！回見！
敬請期待后續監控搭建篇~