k8s(1.14.0)+etcd(3.3.10)+flanneld(0.10)

K8s(1.14)

幾張比較不錯的圖

1.kubernetes 組件圖

kubernetes 架構圖

2.kubernetes 網絡架構圖

• 數據從源容器中發出后，經由所在主機的docker0虛擬網卡轉發到flannel0虛擬網卡，這是個P2P的虛擬網卡，flanneld服務監聽在網卡的另外一端。
• Flannel通過Etcd服務維護了一張節點間的路由表，在稍后的配置部分我們會介紹其中的內容。
• 源主機的flanneld服務將原本的數據內容UDP封裝后根據自己的路由表投遞給目的節點的flanneld服務，數據到達以后被解包，然后直接進入目的節點的flannel0虛擬網卡，
  然后被轉發到目的主機的docker0虛擬網卡，最后就像本機容器通信一下的有docker0路由到達目標容器。

3.Kubernetes工作流程

集群功能各模塊功能描述：

Master節點：
Master節點上面主要由四個模塊組成，APIServer，schedule,controller-manager,etcd

• APIServer: APIServer負責對外提供RESTful的kubernetes API的服務，它是系統管理指令的統一接口，任何對資源的增刪該查都要交給APIServer處理后再交給etcd，如圖，kubectl(kubernetes提供的客戶端工具，該工具內部是對kubernetes API的調用）是直接和APIServer交互的。

• schedule: schedule負責調度Pod到合適的Node上，如果把scheduler看成一個黑匣子，那么它的輸入是pod和由多個Node組成的列表，輸出是Pod和一個Node的綁定。 kubernetes目前提供了調度算法，同樣也保留了接口。用戶根據自己的需求定義自己的調度算法。

• controller manager: 如果APIServer做的是前台的工作的話，那么controller manager就是負責后台的。每一個資源都對應一個控制器。而control manager就是負責管理這些控制器的，比如我們通過APIServer創建了一個Pod，當這個Pod創建成功后，APIServer的任務就算完成了。

• etcd：etcd是一個高可用的鍵值存儲系統，kubernetes使用它來存儲各個資源的狀態，從而實現了Restful的API。

Node節點：
每個Node節點主要由三個模板組成：kublet, kube-proxy

• kube-proxy: 該模塊實現了kubernetes中的服務發現和反向代理功能。kube-proxy支持TCP和UDP連接轉發，默認基Round Robin算法將客戶端流量轉發到與service對應的一組后端pod。服務發現方面，kube-proxy使用etcd的watch機制監控集群中service和endpoint對象數據的動態變化，並且維護一個service到endpoint的映射關系，從而保證了后端pod的IP變化不會對訪問者造成影響，另外，kube-proxy還支持session affinity。

• kublet：kublet是Master在每個Node節點上面的agent，是Node節點上面最重要的模塊，它負責維護和管理該Node上的所有容器，但是如果容器不是通過kubernetes創建的，它並不會管理。本質上，它負責使Pod的運行狀態與期望的狀態一致。

安裝和配置

前提

os：centos 7.5.1804

IP	hostname	服務
4.71	master	kube-apiservice,kube-scheduler,kube-controller-manager,etcd
4.72	node1	kubelet,kube-proxy,flanneld,docker
4.76	node2	kubelet,kube-proxy,flanneld,docker

1.初始化環境

1.1 關閉防火牆和SeLinux

systemctl stop firewalld
setenforce 0 (臨時關閉)
vi /etc/selinux/config
SELINUX=disabled

1.2 下載鏈接

Client Binaries
https://dl.k8s.io/v1.14.0/kubernetes-client-linux-amd64.tar.gz
Server Binaries
https://dl.k8s.io/v1.14.0/kubernetes-server-linux-amd64.tar.gz
Node Binaries
https://dl.k8s.io/v1.14.0/kubernetes-node-linux-amd64.tar.gz
etcd
https://github.com/etcd-io/etcd/releases/download/v3.3.10/etcd-v3.3.10-linux-amd64.tar.gz
flannel
https://github.com/coreos/flannel/releases/download/v0.10.0/flannel-v0.10.0-linux-amd64.tar.gz

2.Master部署

2.1 下載資源(wget不到的話，可以用kexue上網的機器直接download)

wget https://dl.k8s.io/v1.14.0/kubernetes-server-linux-amd64.tar.gz
wget https://dl.k8s.io/v1.14.0/kubernetes-client-linux-amd64.tar.gz
wget https://github.com/etcd-io/etcd/releases/download/v3.3.10/etcd-v3.3.10-linux-amd64.tar.gz
wget https://github.com/coreos/flannel/releases/download/v0.10.0/flannel-v0.10.0-linux-amd64.tar.gz

2.2 cfssl安裝(wget不到的話，可以用kexue上網的機器直接download)

wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64
chmod +x cfssl_linux-amd64 cfssljson_linux-amd64 cfssl-certinfo_linux-amd64
mv cfssl_linux-amd64 /usr/local/bin/cfssl
mv cfssljson_linux-amd64 /usr/local/bin/cfssljson
mv cfssl-certinfo_linux-amd64 /usr/bin/cfssl-certinfo

2.3 創建etcd證書

mkdir /k8s/etcd/{bin,cfg,ssl} -p
mkdir /k8s/kubernetes/{bin,cfg,ssl} -p
cd /k8s/etcd/ssl/

• 2.3.1 etcd ca配置

cat << EOF | tee ca-config.json
{
  "signing": {
    "default": {
      "expiry": "87600h"
    },
    "profiles": {
      "etcd": {
         "expiry": "87600h",
         "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ]
      }
    }
  }
}
EOF

• 2.3.2 etcd ca證書

cat << EOF | tee ca-csr.json
{
    "CN": "etcd CA",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "Beijing",
            "ST": "Beijing"
        }
    ]
}
EOF

• 2.3.3 etcd server證書

cat << EOF | tee server-csr.json
{
    "CN": "etcd",
    "hosts": [
    "192.168.4.71",
    "192.168.4.72",
    "192.168.4.76"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "Beijing",
            "ST": "Beijing"
        }
    ]
}
EOF

• 2.3.4 生成etcd ca證書和私鑰 初始化ca

cfssl gencert -initca ca-csr.json | cfssljson -bare ca

• 2.3.5 生成server證書

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=etcd server-csr.json | cfssljson -bare server

2.4 etcd安裝

• 2.4.1 解壓

tar -xvf etcd-v3.3.10-linux-amd64.tar.gz
cd etcd-v3.3.10-linux-amd64/
cp etcd etcdctl /k8s/etcd/bin/

• 2.4.2 配置etcd的配置文件

[root@t71 cfg]# vim /k8s/etcd/cfg/etcd.conf 

#[Member]
ETCD_NAME="etcd01"
ETCD_DATA_DIR="/data1/etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.4.71:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.4.71:2379"

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.4.71:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.4.71:2379"
ETCD_INITIAL_CLUSTER="etcd01=https://192.168.4.71:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"

#[Security]
ETCD_CERT_FILE="/k8s/etcd/ssl/server.pem"
ETCD_KEY_FILE="/k8s/etcd/ssl/server-key.pem"
ETCD_TRUSTED_CA_FILE="/k8s/etcd/ssl/ca.pem"
ETCD_CLIENT_CERT_AUTH="true"
ETCD_PEER_CERT_FILE="/k8s/etcd/ssl/server.pem"
ETCD_PEER_KEY_FILE="/k8s/etcd/ssl/server-key.pem"
ETCD_PEER_TRUSTED_CA_FILE="/k8s/etcd/ssl/ca.pem"
ETCD_PEER_CLIENT_CERT_AUTH="true"

• 2.4.3 配置etcd的啟動文件

mkdir /data1/etcd
[root@t71 cfg]# vim /usr/lib/systemd/system/etcd.service 

[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
WorkingDirectory=/data1/etcd/
EnvironmentFile=-/k8s/etcd/cfg/etcd.conf
# set GOMAXPROCS to number of processors
ExecStart=/bin/bash -c "GOMAXPROCS=$(nproc) /k8s/etcd/bin/etcd --name=\"${ETCD_NAME}\" --data-dir=\"${ETCD_DATA_DIR}\" --listen-client-urls=\"${ETCD_LISTEN_CLIENT_URLS}\" --listen-peer-urls=\"${ETCD_LISTEN_PEER_URLS}\" --advertise-client-urls=\"${ETCD_ADVERTISE_CLIENT_URLS}\" --initial-cluster-token=\"${ETCD_INITIAL_CLUSTER_TOKEN}\" --initial-cluster=\"${ETCD_INITIAL_CLUSTER}\" --initial-cluster-state=\"${ETCD_INITIAL_CLUSTER_STATE}\" --cert-file=\"${ETCD_CERT_FILE}\" --key-file=\"${ETCD_KEY_FILE}\" --trusted-ca-file=\"${ETCD_TRUSTED_CA_FILE}\" --client-cert-auth=\"${ETCD_CLIENT_CERT_AUTH}\" --peer-cert-file=\"${ETCD_PEER_CERT_FILE}\" --peer-key-file=\"${ETCD_PEER_KEY_FILE}\" --peer-trusted-ca-file=\"${ETCD_PEER_TRUSTED_CA_FILE}\" --peer-client-cert-auth=\"${ETCD_PEER_CLIENT_CERT_AUTH}\""
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.targe

• 2.4.4 啟動

systemctl daemon-reload
systemctl enabel etcd
systemctl start etcd

• 2.4.5 檢查服務

[root@t71 bin]# etcdctl --ca-file=/k8s/etcd/ssl/ca.pem --cert-file=/k8s/etcd/ssl/server.pem --key-file=/k8s/etcd/ssl/server-key.pem --endpoints="https://192.168.4.71:2379" cluster-health 
member ac829673d2b22824 is healthy: got healthy result from https://192.168.4.71:2379
cluster is healthy
[root@t71 bin]# 

2.5 生成kubernets證書與私鑰

• 2.5.1 制作kubernetes ca證書

cd /k8s/kubernetes/ssl
cat << EOF | tee ca-config.json
{
  "signing": {
    "default": {
      "expiry": "87600h"
    },
    "profiles": {
      "kubernetes": {
         "expiry": "87600h",
         "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ]
      }
    }
  }
}
EOF

cat << EOF | tee ca-csr.json
{
    "CN": "kubernetes",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "Beijing",
            "ST": "Beijing",
            "O": "k8s",
            "OU": "System"
        }
    ]
}
EOF

cfssl gencert -initca ca-csr.json | cfssljson -bare ca -

• 2.5.2 制作apiserver證書

cat << EOF | tee server-csr.json
{
    "CN": "kubernetes",
    "hosts": [
      "10.254.0.1",
      "127.0.0.1",
      "192.168.4.71",
	  "192.168.4.72",
	  "192.168.4.76",
      "kubernetes",
      "kubernetes.default",
      "kubernetes.default.svc",
      "kubernetes.default.svc.cluster",
      "kubernetes.default.svc.cluster.local"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "Beijing",
            "ST": "Beijing",
            "O": "k8s",
            "OU": "System"
        }
    ]
}
EOF

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes server-csr.json | cfssljson -bare server

• 2.5.3 制作kube-proxy證書

cat << EOF | tee kube-proxy-csr.json
{
  "CN": "system:kube-proxy",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "L": "Beijing",
      "ST": "Beijing",
      "O": "k8s",
      "OU": "System"
    }
  ]
}
EOF

 cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy

2.6 安裝k8s服務

2.6.1 解壓縮文件

tar -zxvf kubernetes-server-linux-amd64.tar.gz 
cd kubernetes/server/bin/
cp kube-scheduler kube-apiserver kube-controller-manager kubectl /k8s/kubernetes/bin/

2.6.2 安裝kube-apiserver

• 2.6.2.1 部署kube-apiserver組件 創建TLS Bootstrapping Token

[root@elasticsearch01 bin]# head -c 16 /dev/urandom | od -An -t x | tr -d ' '
663eb46fb81c4cf2a9bedb84bea03582
 
vim /k8s/kubernetes/cfg/token.csv
663eb46fb81c4cf2a9bedb84bea03582,kubelet-bootstrap,10001,"system:kubelet-bootstrap"

• 2.6.2.2 創建Apiserver配置文件

[root@t71 cfg]# vim /k8s/kubernetes/cfg/kube-apiserver 

KUBE_APISERVER_OPTS="--logtostderr=true \
--v=4 \
--etcd-servers=https://192.168.4.71:2379 \
--bind-address=192.168.4.71 \
--secure-port=6443 \
--advertise-address=192.168.4.71 \
--allow-privileged=true \
--service-cluster-ip-range=10.254.0.0/16 \
--enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota,NodeRestriction \
--authorization-mode=RBAC,Node \
--enable-bootstrap-token-auth \
--token-auth-file=/k8s/kubernetes/cfg/token.csv \
--service-node-port-range=30000-50000 \
--tls-cert-file=/k8s/kubernetes/ssl/server.pem  \
--tls-private-key-file=/k8s/kubernetes/ssl/server-key.pem \
--client-ca-file=/k8s/kubernetes/ssl/ca.pem \
--service-account-key-file=/k8s/kubernetes/ssl/ca-key.pem \
--etcd-cafile=/k8s/etcd/ssl/ca.pem \
--etcd-certfile=/k8s/etcd/ssl/server.pem \
--etcd-keyfile=/k8s/etcd/ssl/server-key.pem"

• 2.6.2.3 創建apiserver systemd文件

[root@t71 cfg]# vim /usr/lib/systemd/system/kube-apiserver.service 

[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes

[Service]
EnvironmentFile=-/k8s/kubernetes/cfg/kube-apiserver
ExecStart=/k8s/kubernetes/bin/kube-apiserver $KUBE_APISERVER_OPTS
Restart=on-failure

[Install]
WantedBy=multi-user.target

• 2.6.2.4 啟動服務

systemctl daemon-reload
systemctl enable kube-apiserver
systemctl start kube-apiserver

2.6.3 安裝kube-scheduler

• 2.6.3.1 創建kube-scheduler配置文件

[root@t71 cfg]# vim /k8s/kubernetes/cfg/kube-scheduler 
KUBE_SCHEDULER_OPTS="--logtostderr=true --v=4 --master=127.0.0.1:8080 

• 2.6.3.2 創建kube-scheduler systemd文件

[root@t71 cfg]# vim /usr/lib/systemd/system/kube-scheduler.service 

[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/kubernetes/kubernetes

[Service]
EnvironmentFile=-/k8s/kubernetes/cfg/kube-scheduler
ExecStart=/k8s/kubernetes/bin/kube-scheduler $KUBE_SCHEDULER_OPTS
Restart=on-failure

[Install]
WantedBy=multi-user.target

• 2.6.3.3 啟動服務

systemctl daemon-reload
systemctl start kube-scheduler
systemctl enable kube-scheduler

2.6.4 安裝kube-controller-manager

• 2.6.3.1 創建kube-controller-managerr配置文件

[root@t71 cfg]# vim /k8s/kubernetes/cfg/kube-controller-manager 

KUBE_CONTROLLER_MANAGER_OPTS="--logtostderr=true \
--v=4 \
--master=127.0.0.1:8080 \
--leader-elect=true \
--address=127.0.0.1 \
--service-cluster-ip-range=10.254.0.0/16 \
--cluster-name=kubernetes \
--cluster-signing-cert-file=/k8s/kubernetes/ssl/ca.pem \
--cluster-signing-key-file=/k8s/kubernetes/ssl/ca-key.pem  \
--root-ca-file=/k8s/kubernetes/ssl/ca.pem \
--service-account-private-key-file=/k8s/kubernetes/ssl/ca-key.pem"

• 2.6.3.2 創建kube-controller-manager systemd文件

[root@t71 cfg]# vim /usr/lib/systemd/system/kube-controller-manager.service 

[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/kubernetes/kubernetes

[Service]
EnvironmentFile=-/k8s/kubernetes/cfg/kube-controller-manager
ExecStart=/k8s/kubernetes/bin/kube-controller-manager $KUBE_CONTROLLER_MANAGER_OPTS
Restart=on-failure

[Install]
WantedBy=multi-user.target

• 2.6.3.3 啟動服務

systemctl daemon-reload
systemctl start kube-controller-manager
systemctl enable kube-controller-manager

2.7 驗證kube server

• 2.7.1 添加環境變量

vim /etc/profile
export PATH=/k8s/kubernetes/bin:$PATH
source /etc/profile

• 2.7.2 get

3.Node部署

3.1 docker 環境安裝

yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
yum list docker-ce --showduplicates | sort -r
yum install docker-ce -y
systemctl start docker && systemctl enable docker

3.2 部署kubelet插件

kublet 運行在每個 worker 節點上，接收 kube-apiserver 發送的請求，管理 Pod 容器，執行交互式命令，如exec、run、logs 等; kublet 啟動時自動向 kube-apiserver 注冊節點信息，內置的 cadvisor 統計和監控節點的資源使用情況; 為確保安全，只開啟接收 https 請求的安全端口，對請求進行認證和授權，拒絕未授權的訪問(如apiserver、heapster)

• 3.2.1 安裝二進制文件

wget https://dl.k8s.io/v1.13.1/kubernetes-node-linux-amd64.tar.gz
tar zxvf kubernetes-node-linux-amd64.tar.gz
cd kubernetes/node/bin/
cp kube-proxy kubelet kubectl /k8s/kubernetes/bin/

• 3.2.2 復制相關證書到node節點

scp *.pem 10.2.8.65:$PWD

• 3.2.3 創建kubelet bootstrap kubeconfig文件 通過腳本實現

[root@t72 cfg]# vim environment.sh 

#!/bin/bash
#創建kubelet bootstrapping kubeconfig 
BOOTSTRAP_TOKEN=663eb46fb81c4cf2a9bedb84bea03582
KUBE_APISERVER="https://192.168.4.71:6443"
#設置集群參數
kubectl config set-cluster kubernetes \
  --certificate-authority=/k8s/kubernetes/ssl/ca.pem \
  --embed-certs=true \
  --server=${KUBE_APISERVER} \
  --kubeconfig=bootstrap.kubeconfig

#設置客戶端認證參數
kubectl config set-credentials kubelet-bootstrap \
  --token=${BOOTSTRAP_TOKEN} \
  --kubeconfig=bootstrap.kubeconfig

# 設置上下文參數
kubectl config set-context default \
  --cluster=kubernetes \
  --user=kubelet-bootstrap \
  --kubeconfig=bootstrap.kubeconfig

# 設置默認上下文
kubectl config use-context default --kubeconfig=bootstrap.kubeconfig

#----------------------

# 創建kube-proxy kubeconfig文件

kubectl config set-cluster kubernetes \
  --certificate-authority=/k8s/kubernetes/ssl/ca.pem \
  --embed-certs=true \
  --server=${KUBE_APISERVER} \
  --kubeconfig=kube-proxy.kubeconfig

kubectl config set-credentials kube-proxy \
  --client-certificate=/k8s/kubernetes/ssl/kube-proxy.pem \
  --client-key=/k8s/kubernetes/ssl/kube-proxy-key.pem \
  --embed-certs=true \
  --kubeconfig=kube-proxy.kubeconfig

kubectl config set-context default \
  --cluster=kubernetes \
  --user=kube-proxy \
  --kubeconfig=kube-proxy.kubeconfig

kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig

• 3.2.4 執行腳本

sh environment.sh  

• 3.2.5 創建kubelet參數配置模板文件

[root@t72 cfg]# vim kubelet.config 

kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
address: 192.168.4.72
port: 10250
readOnlyPort: 10255
cgroupDriver: cgroupfs
clusterDNS: ["10.254.0.10"]
clusterDomain: cluster.local.
failSwapOn: false
#authentication:
#  anonymous:
#    enabled: true
authentication:
  anonymous:
    enabled: true # Defaults to false as of 1.10
  webhook:
    enabled: false # Deafults to true as of 1.10
authorization:
  mode: AlwaysAllow

• 3.2.6 創建kubelet配置文件

[root@t72 cfg]# vim kubelet

KUBELET_OPTS="--logtostderr=true \
--v=4 \
--hostname-override=192.168.4.72 \
--kubeconfig=/k8s/kubernetes/cfg/kubelet.kubeconfig \
--bootstrap-kubeconfig=/k8s/kubernetes/cfg/bootstrap.kubeconfig \
--config=/k8s/kubernetes/cfg/kubelet.config \
--cert-dir=/k8s/kubernetes/ssl \
--pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google-containers/pause-amd64:3.0"

• 3.2.7 創建kubelet systemd文件

[root@t72 cfg]# vim /usr/lib/systemd/system/kubelet.service 

[Unit]
Description=Kubernetes Kubelet
After=docker.service
Requires=docker.service

[Service]
EnvironmentFile=-/k8s/kubernetes/cfg/kubelet
ExecStart=/k8s/kubernetes/bin/kubelet $KUBELET_OPTS
Restart=on-failure
KillMode=process

[Install]
WantedBy=multi-user.target

• 3.2.8 在master節點上面操作，將kubelet-bootstrap用戶綁定到系統集群角色

kubectl create clusterrolebinding kubelet-bootstrap \
  --clusterrole=system:node-bootstrapper \
  --user=kubelet-bootstrap

• 3.2.9 啟動kubelet服務

systemctl daemon-reload
systemctl start kubelet 
systemctl enable kubelet

• 3.2.10 Master接受kubelet CSR請求 可以手動或自動 approve CSR 請求。推薦使用自動的方式，因為從 v1.8 版本開始，可以自動輪轉approve csr 后生成的證書，如下是手動 approve CSR請求操作方法 查看CSR列表

# kubectl get csr
NAME                                                   AGE    REQUESTOR           CONDITION
node-csr-ij3py9j-yi-eoa8sOHMDs7VeTQtMv0N3Efj3ByZLMdc   102s   kubelet-bootstrap   Pending

接受node

# kubectl certificate approve node-csr-ij3py9j-yi-eoa8sOHMDs7VeTQtMv0N3Efj3ByZLMdc
certificatesigningrequest.certificates.k8s.io/node-csr-ij3py9j-yi-eoa8sOHMDs7VeTQtMv0N3Efj3ByZLMdc approved

再看CRS

kubectl get csr
NAME                                                   AGE     REQUESTOR           CONDITION
node-csr-ij3py9j-yi-eoa8sOHMDs7VeTQtMv0N3Efj3ByZLMdc   5m13s   kubelet-bootstrap   Approved,Issued

3.3 安裝kube-proxy

• 3.3.1 kube-proxy的配置文件

[root@t72 cfg]# vim kube-proxy

KUBE_PROXY_OPTS="--logtostderr=true \
--v=4 \
--hostname-override=192.168.4.72 \
--cluster-cidr=10.254.0.0/16 \
--kubeconfig=/k8s/kubernetes/cfg/kube-proxy.kubeconfig"

• 3.3.2 kube-proxy的systemd 文件

[root@t72 cfg]# vim /usr/lib/systemd/system/kube-proxy.service 

[Unit]
Description=Kubernetes Proxy
After=network.target

[Service]
EnvironmentFile=-/k8s/kubernetes/cfg/kube-proxy
ExecStart=/k8s/kubernetes/bin/kube-proxy $KUBE_PROXY_OPTS
Restart=on-failure

[Install]
WantedBy=multi-user.target

• 3.3.3 啟動

systemctl daemon-reload
systemctl start kube-proxy
systemctl enable kube-proxy

3.4 部署另外一個node和認證csr

注意期間要是kubelet，kube-proxy配置錯誤，比如監聽IP或者hostname錯誤導致node not found，需要刪除kubelet-client證書，重啟kubelet服務，重啟認證csr即可

4. 網絡部署

4.1 etcd注冊網段（master節點，因為只有master安裝了etcd）

/k8s/etcd/bin/etcdctl --ca-file=/k8s/etcd/ssl/ca.pem --cert-file=/k8s/etcd/ssl/server.pem --key-file=/k8s/etcd/ssl/server-key.pem --endpoints="https://192.168.4.71:2379"  set /k8s/network/config  '{ "Network": "10.254.0.0/16", "Backend": {"Type": "vxlan"}}'

注：寫入的 Pod 網段 ${CLUSTER_CIDR} 必須是 /16 段地址，必須與 kube-controller-manager 的 –cluster-cidr 參數值一致

4.2 Flanneld部署

• 4.2.1 解壓

tar -xvf flannel-v0.10.0-linux-amd64.tar.gz
mv flanneld mk-docker-opts.sh /k8s/kubernetes/bin/

• 4.2.2 配置flanneld

[root@t72 cfg]# vim flanneld 

FLANNEL_OPTIONS="--etcd-endpoints=https://192.168.4.71:2379, -etcd-cafile=/k8s/etcd/ssl/ca.pem -etcd-certfile=/k8s/etcd/ssl/server.pem -etcd-keyfile=/k8s/etcd/ssl/server-key.pem -etcd-prefix=/k8s/network"

• 4.2.3 創建flanneld systemd文件

[root@t72 cfg]# vim /usr/lib/systemd/system/flanneld.service 

[Unit]
Description=Flanneld overlay address etcd agent
After=network-online.target network.target
Before=docker.service

[Service]
Type=notify
EnvironmentFile=/k8s/kubernetes/cfg/flanneld
ExecStart=/k8s/kubernetes/bin/flanneld --ip-masq $FLANNEL_OPTIONS
#ExecStart=/k8s/kubernetes/bin/flanneld $FLANNEL_OPTIONS
ExecStartPost=/k8s/kubernetes/bin/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/subnet.env
Restart=on-failure

[Install]
WantedBy=multi-user.target

注：mk-docker-opts.sh 腳本將分配給 flanneld 的 Pod 子網網段信息寫入 /run/flannel/docker 文件，后續 docker 啟動時 使用這個文件中的環境變量配置 docker0 網橋； flanneld 使用系統缺省路由所在的接口與其它節點通信，對於有多個網絡接口（如內網和公網）的節點，可以用 -iface 參數指定通信接口; flanneld 運行時需要 root 權限；

• 4.2.4 配置Docker啟動指定子網 修改EnvironmentFile=/run/flannel/subnet.env，ExecStart=/usr/bin/dockerd $DOCKER_NETWORK_OPTIONS即可

[root@t72 cfg]# cat /usr/lib/systemd/system/docker.service | grep -v "#" 
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
BindsTo=containerd.service
After=network-online.target firewalld.service containerd.service
Wants=network-online.target
Requires=docker.socket

[Service]
Type=notify
EnvironmentFile=/run/flannel/subnet.env
ExecStart=/usr/bin/dockerd $DOCKER_NETWORK_OPTIONS
ExecReload=/bin/kill -s HUP $MAINPID
TimeoutSec=0
RestartSec=2
Restart=always

StartLimitBurst=3

StartLimitInterval=60s

LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity

TasksMax=infinity

Delegate=yes

KillMode=process

[Install]
WantedBy=multi-user.target

• 4.2.5 啟動服務 注意啟動flannel前要關閉docker及相關的kubelet這樣flannel才會覆蓋docker0網橋(如果已經存在docker0，則先刪除docker0 ip link delete docker0)

systemctl daemon-reload
systemctl stop docker
systemctl start flanneld
systemctl enable flanneld
systemctl start docker
systemctl restart kubelet
systemctl restart kube-proxy

• 4.2.6 驗證服務

ip addr

kubectl get nodes -o wide