二進制搭建kubernetes多master集群【二、配置flannel網絡】

上一篇我們已經搭建etcd高可用集群，參考：二進制搭建kubernetes多master集群【一、使用TLS證書搭建etcd集群】

此文將搭建flannel網絡，目的使跨主機的docker能夠互相通信，也是保障kubernetes集群的網絡基礎和保障，下面正式開始配置。

 

一、生成Flannel網絡TLS證書

在所有集群節點都安裝Flannel，下面的操作在k8s-master1上進行，其他節點重復執行即可。（證書生成一次就行）

1、創建證書簽名請求

cat > flanneld-csr.json <<EOF
{
  "CN": "flanneld",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "BeiJing",
      "L": "BeiJing",
      "O": "k8s",
      "OU": "4Paradigm"
    }
  ]
}
EOF

• 該證書只會被 kubectl 當做 client 證書使用，所以 hosts 字段為空；

生成證書和私鑰：

cfssl gencert -ca=/etc/kubernetes/cert/ca.pem \
  -ca-key=/etc/kubernetes/cert/ca-key.pem \
  -config=/etc/kubernetes/cert/ca-config.json \
  -profile=kubernetes flanneld-csr.json | cfssljson -bare flanneld

2、將證書分發到所有集群節點/etc/kubernetes/cert/目錄下

scp flanneld*.pem k8s-master1:/etc/kubernetes/cert/
scp flanneld*.pem k8s-master2:/etc/kubernetes/cert/
scp flanneld*.pem k8s-master3:/etc/kubernetes/cert/
scp flanneld*.pem k8s-node1:/etc/kubernetes/cert/
scp flanneld*.pem k8s-node2:/etc/kubernetes/cert/
scp flanneld*.pem k8s-node3:/etc/kubernetes/cert/

 

二、部署 Flannel

1、下載安裝Flannel

wget https://github.com/coreos/flannel/releases/download/v0.10.0/flannel-v0.10.0-linux-amd64.tar.gz
tar -xzvf flannel-v0.10.0-linux-amd64.tar.gz
cp {flanneld,mk-docker-opts.sh} /usr/local/bin

2、向 etcd 寫入網段信息

下面2條命令在etcd集群中任意一台執行一次即可，也是是創建一個flannel網段供docker分配使用

[root@etcd1 cert]# etcdctl --ca-file=/etc/kubernetes/cert/ca.pem --cert-file=/etc/kubernetes/cert/etcd.pem --key-file=/etc/kubernetes/cert/etcd-key.pem mkdir /kubernetes/network
[root@etcd1 cert]# etcdctl --ca-file=/etc/kubernetes/cert/ca.pem --cert-file=/etc/kubernetes/cert/etcd.pem --key-file=/etc/kubernetes/cert/etcd-key.pem mk /kubernetes/network/config '{"Network":"172.30.0.0/16","SubnetLen":24,"Backend":{"Type":"vxlan"}}'

3、創建system unit文件

[root@k8s-master1 ssl]# cat > /etc/systemd/system/flanneld.service << EOF
[Unit]
Description=Flanneld overlay address etcd agent
After=network.target
After=network-online.target
Wants=network-online.target
After=etcd.service
Before=docker.service

[Service]
Type=notify
ExecStart=/usr/local/bin/flanneld \
  -etcd-cafile=/etc/kubernetes/cert/ca.pem \
  -etcd-certfile=/etc/kubernetes/cert/flanneld.pem \
  -etcd-keyfile=/etc/kubernetes/cert/flanneld-key.pem \
  -etcd-endpoints=https://192.168.80.4:2379,https://192.168.80.5:2379,https://192.168.80.6:2379 \
  -etcd-prefix=/kubernetes/network
ExecStartPost=/usr/local/bin/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/docker
Restart=on-failure

[Install]
WantedBy=multi-user.target
RequiredBy=docker.service
EOF

mk-docker-opts.sh 腳本將分配給 flanneld 的 Pod 子網網段信息寫入到 /run/flannel/docker 文件中，后續 docker 啟動時使用這個文件中參數值設置 docker0 網橋。

flanneld 使用系統缺省路由所在的接口和其它節點通信，對於有多個網絡接口的機器（如，內網和公網），可以用 -iface=enpxx 選項值指定通信接口。

4、啟動flannel並且設置開機自啟動

systemctl daemon-reload
systemctl enable flanneld
systemctl start flanneld

5、查看flannel分配的子網信息

[root@k8s-master1 ~]# cat /run/flannel/docker 
DOCKER_OPT_BIP="--bip=172.30.94.1/24" DOCKER_OPT_IPMASQ="--ip-masq=true" DOCKER_OPT_MTU="--mtu=1450" DOCKER_NETWORK_OPTIONS=" --bip=172.30.94.1/24 --ip-masq=true --mtu=1450" [root@k8s-master1 ~]# cat /run/flannel/subnet.env FLANNEL_NETWORK=172.30.0.0/16 FLANNEL_SUBNET=172.30.94.1/24 FLANNEL_MTU=1450 FLANNEL_IPMASQ=false

/run/flannel/docker是flannel分配給docker的子網信息，/run/flannel/subnet.env包含了flannel整個大網段以及在此節點上的子網段。

6、查看flannel網絡是否生效

[root@k8s-master1 ~]# ifconfig 
docker0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet 172.30.94.1  netmask 255.255.255.0  broadcast 172.30.94.255
        inet6 fe80::42:1aff:fed2:a4b4  prefixlen 64  scopeid 0x20<link>
        ether 02:42:1a:d2:a4:b4  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 8  bytes 648 (648.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ens192: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.20.210  netmask 255.255.254.0  broadcast 192.168.21.255
        inet6 fe80::ff2:187b:66fc:621b  prefixlen 64  scopeid 0x20<link>
        ether 00:0c:29:c3:dc:a5  txqueuelen 1000  (Ethernet)
        RX packets 3965867  bytes 619350597 (590.6 MiB)
        RX errors 0  dropped 583  overruns 0  frame 0
        TX packets 3159970  bytes 390102190 (372.0 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

flannel.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet 172.30.94.0  netmask 255.255.255.255  broadcast 0.0.0.0
        inet6 fe80::7827:fcff:fe4e:b5ff  prefixlen 64  scopeid 0x20<link>
        ether 7a:27:fc:4e:b5:ff  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 8 overruns 0  carrier 0  collisions 0

可以明顯看到flannel1.1的網絡信息，說明flannel網絡已經正常。

三、配置docker支持flannel網絡

1、所有node安裝docker

關於安裝docker，請參考：安裝指定版本的docker

2、配置docker支持flannel網絡，所有docker節點都操作

[root@k8s-master1 ~]# vi /etc/systemd/system/multi-user.target.wants/docker.service
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network-online.target firewalld.service
Wants=network-online.target

[Service]
Type=notify
# the default is not to use systemd for cgroups because the delegate issues still
# exists and systemd currently does not support the cgroup feature set required
# for containers run by docker
EnvironmentFile=/run/flannel/docker
ExecStart=/usr/bin/dockerd $DOCKER_NETWORK_OPTIONS
ExecReload=/bin/kill -s HUP $MAINPID
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
# Uncomment TasksMax if your systemd version supports it.
# Only systemd 226 and above support this version.
#TasksMax=infinity
TimeoutStartSec=0
# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes
# kill only the docker process, not all processes in the cgroup
KillMode=process
# restart the docker process if it exits prematurely
Restart=on-failure
StartLimitBurst=3
StartLimitInterval=60s

[Install]
WantedBy=multi-user.target

如上紅色即為新增加支持flannel網絡的配置

3、重啟docker，使配置生效

systemctl daemon-reload
systemctl restart docker

4、查看docker網絡是否生效

i、啟動一個容器（如有現有容器可以不run一個新的）
[root@k8s-master1 ~]# docker run -itd centos
d63ee9c72b5023fgfg36ld93j6723hd72jd1hsp2303kf7
ii、查看ip地址是否是flannel網絡分配的網段
[root@k8s-master1 ~]# docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' d63
172.30.94.2

5、查看所有集群主機的網絡情況

[root@etcd1 cert]# etcdctl --ca-file=/etc/kubernetes/cert/ca.pem --cert-file=/etc/kubernetes/cert/etcd.pem --key-file=/etc/kubernetes/cert/etcd-key.pem ls /kubernetes/network/subnets
/kubernetes/network/subnets/172.30.94.0-24
/kubernetes/network/subnets/172.30.51.0-24
/kubernetes/network/subnets/172.30.10.0-24
/kubernetes/network/subnets/172.30.92.0-24
/kubernetes/network/subnets/172.30.85.0-24
/kubernetes/network/subnets/172.30.89.0-24

發現容器使用了172.30.94.0/24網段。屬於flannel，配置完成。

下一篇將部署二進制搭建kubernetes多master集群【三、配置k8s master及高可用】