k8s集群之Docker安裝鏡像加速器配置與k8s容器網絡

安裝Docker

參考：https://www.cnblogs.com/rdchenxi/p/10381631.html

加速器配置

參考：https://www.cnblogs.com/rdchenxi/p/10399885.html

網絡介紹k8s（CNI網絡模型）

Flannel網絡

overlay

覆蓋網絡就是應用層網絡，它是面向應用層的，不考慮或很少考慮網絡層，物理層的問題。

詳細說來，覆蓋網絡是指建立在另一個網絡上的網絡。該網絡中的結點可以看作通過虛擬或邏輯鏈路而連接起來的。雖然在底層有很多條物理鏈路，但是這些虛擬或邏輯鏈路都與路徑一一對應。例如：許多P2P網絡就是覆蓋網絡，因為它運行在互連網的上層。覆蓋網絡允許對沒有IP地址標識的目的主機路由信息，例如：Freenet 和DHT（分布式哈希表）可以路由信息到一個存儲特定文件的結點，而這個結點的IP地址事先並不知道。

覆蓋網絡被認為是一條用來改善互連網路由的途徑，讓二層網絡在三層網絡中傳遞，既解決了二層的缺點，又解決了三層的不靈活！

FIannel

Flannel實質上是一種“覆蓋網絡(overlay network)”，也就是將TCP數據包裝在另一種網絡包里面進行路由轉發和通信，目前已經支持UDP、VxLAN、AWS VPC和GCE路由等數據轉發方式。

默認的節點間數據通信方式是UDP轉發。

安裝Flannel

分配子網段寫入edcd里

[root@mast-1 k8s]# /opt/etcd/bin/etcdctl --ca-file=/opt/etcd/ssl/ca.pem --cert-file=/opt/etcd/ssl/server.pem --key-file=/opt/etcd/ssl/server-key.pem --endpoints="https://192.168.10.11:2379,ht
tps://192.168.10.12:2379,https://192.168.10.13:2379" set /coreos.com/network/config '{ "Network": "172.17.0.0/16", "Backend": {"Type": "vxlan"}}'
{ "Network": "172.17.0.0/16", "Backend": {"Type": "vxlan"}}
[root@mast-1 k8s]# 

查看數據

[root@mast-1 k8s]# /opt/etcd/bin/etcdctl --ca-file=/opt/etcd/ssl/ca.pem --cert-file=/opt/etcd/ssl/server.pem --key-file=/opt/etcd/ssl/server-key.pem --endpoints="https://192.168.10.11:2379,ht
tps:////192.168.10.12:2379,https://192.168.10.13:2379" get /coreos.com/network/config { "Network": "172.17.0.0/16", "Backend": {"Type": "vxlan"}}

下載安裝Flannel

[root@node-1 ~]# wget https://github.com/coreos/flannel/releases/download/v0.10.0/flannel-v0.10.0-linux-amd64.tar.gz
--2019-04-20 09:38:45--  https://github.com/coreos/flannel/releases/download/v0.10.0/flannel-v0.10.0-linux-amd64.tar.gz
正在解析主機 github.com (github.com)... 13.250.177.223, 52.74.223.119, 13.229.188.59
正在連接 github.com (github.com)|13.250.177.223|:443... 已連接。
已發出 HTTP 請求，正在等待回應... 302 Found
位置：https://github-production-release-asset-2e65be.s3.amazonaws.com/21704134/596e76e2-002c-11e8-9359-36689058e7af?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20
190420%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20190420T013853Z&X-Amz-Expires=300&X-Amz-Signature=9c7a12bd05f366c722480fd53b3968d2a3b6ed6f690baab3a24ef7b1955e2d11&X-Amz-SignedHeaders=host&actor_id=0&response-content-disposition=attachment%3B%20filename%3Dflannel-v0.10.0-linux-amd64.tar.gz&response-content-type=application%2Foctet-stream [跟隨至新的 URL]--2019-04-20 09:38:53--  https://github-production-release-asset-2e65be.s3.amazonaws.com/21704134/596e76e2-002c-11e8-9359-36689058e7af?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIW
NJYAX4CSVEH53A%2F20190420%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20190420T013853Z&X-Amz-Expires=300&X-Amz-Signature=9c7a12bd05f366c722480fd53b3968d2a3b6ed6f690baab3a24ef7b1955e2d11&X-Amz-SignedHeaders=host&actor_id=0&response-content-disposition=attachment%3B%20filename%3Dflannel-v0.10.0-linux-amd64.tar.gz&response-content-type=application%2Foctet-stream正在解析主機 github-production-release-asset-2e65be.s3.amazonaws.com (github-production-release-asset-2e65be.s3.amazonaws.com)... 52.216.139.211
正在連接 github-production-release-asset-2e65be.s3.amazonaws.com (github-production-release-asset-2e65be.s3.amazonaws.com)|52.216.139.211|:443... 已連接。
已發出 HTTP 請求，正在等待回應... 200 OK
長度：9706487 (9.3M) [application/octet-stream]
正在保存至: “flannel-v0.10.0-linux-amd64.tar.gz”

100%[=====================================================================================================================================================>] 9,706,487   15.6KB/s 用時 7m 23s  

2019-04-20 09:46:19 (21.4 KB/s) - 已保存 “flannel-v0.10.0-linux-amd64.tar.gz” [9706487/9706487])

　　node-1安裝

 

[root@node-1 ~]# mkdir /opt/kubernetes/{bin,cfg} -pv
mkdir: 已創建目錄 "/opt/kubernetes"
mkdir: 已創建目錄 "/opt/kubernetes/bin"
mkdir: 已創建目錄 "/opt/kubernetes/cfg"
[root@node-1 ~]# tar xf flannel-v0.10.0-linux-amd64.tar.gz -C /opt/kubernetes/bin/
[root@node-1 ~]# cat flannel.sh 
#!/bin/bash

ETCD_ENDPOINTS=${1:-"http://127.0.0.1:2379"}

cat <<EOF >/opt/kubernetes/cfg/flanneld

FLANNEL_OPTIONS="--etcd-endpoints=${ETCD_ENDPOINTS} \
-etcd-cafile=/opt/etcd/ssl/ca.pem \
-etcd-certfile=/opt/etcd/ssl/server.pem \
-etcd-keyfile=/opt/etcd/ssl/server-key.pem"

EOF

cat <<EOF >/usr/lib/systemd/system/flanneld.service
[Unit]
Description=Flanneld overlay address etcd agent
After=network-online.target network.target
Before=docker.service

[Service]
Type=notify
EnvironmentFile=/opt/kubernetes/cfg/flanneld
ExecStart=/opt/kubernetes/bin/flanneld --ip-masq \$FLANNEL_OPTIONS
ExecStartPost=/opt/kubernetes/bin/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/subnet.env
Restart=on-failure

[Install]
WantedBy=multi-user.target

EOF

cat <<EOF >/usr/lib/systemd/system/docker.service

[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network-online.target firewalld.service
Wants=network-online.target

[Service]
Type=notify
EnvironmentFile=/run/flannel/subnet.env    讀取生成的子網
ExecStart=/usr/bin/dockerd \$DOCKER_NETWORK_OPTIONS
ExecReload=/bin/kill -s HUP \$MAINPID
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
TimeoutStartSec=0
Delegate=yes
KillMode=process
Restart=on-failure
StartLimitBurst=3
StartLimitInterval=60s

[Install]
WantedBy=multi-user.target

EOF

systemctl daemon-reload
systemctl enable flanneld
systemctl restart flanneld
systemctl restart docker
[root@node-1 ~]# bash flannel.sh "https://192.168.10.11:2379,https://192.168.10.12:2379,https://192.168.10.13:2379"
[root@node-1 ~]# cat /opt/kubernetes/cfg/flanneld 

FLANNEL_OPTIONS="--etcd-endpoints=https://192.168.10.11:2379,https://192.168.10.12:2379,https://192.168.10.13:2379 -etcd-cafile=/opt/etcd/ssl/ca.pem -etcd-certfile=/opt/etcd/ssl/server.pem -e
tcd-keyfile=/opt/etcd/ssl/server-key.pem"
[root@node-1 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:0c:29:f7:91:47 brd ff:ff:ff:ff:ff:ff
    inet 192.168.10.13/24 brd 192.168.10.255 scope global noprefixroute ens33
       valid_lft forever preferred_lft forever
    inet6 fe80::6017:43d:a11c:2a9f/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 02:42:19:5d:ee:63 brd ff:ff:ff:ff:ff:ff
    inet 172.17.8.1/24 brd 172.17.8.255 scope global docker0
       valid_lft forever preferred_lft forever
4: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default 
    link/ether 56:2f:96:00:5c:05 brd ff:ff:ff:ff:ff:ff
    inet 172.17.8.0/32 scope global flannel.1
       valid_lft forever preferred_lft forever
    inet6 fe80::542f:96ff:fe00:5c05/64 scope link 
       valid_lft forever preferred_lft forever

　　node-2安裝

[root@node-1 ~]# scp -r /usr/lib/systemd/system/docker.service 192.168.10.14:/usr/lib/systemd/system
root@192.168.10.14's password: 
docker.service                                                                                                                                               100%  526   236.7KB/s   00:00    
[root@node-1 ~]# scp -r /usr/lib/systemd/system/flanneld.service 192.168.10.14:/usr/lib/systemd/system
root@192.168.10.14's password: 
flanneld.service                                                                                                                                             100%  417   178.3KB/s   00:00 
[root@node-1 ~]# scp -r /opt/kubernetes 192.168.10.14:/opt/
root@192.168.10.14's password: 
Permission denied, please try again.
root@192.168.10.14's password: 
flanneld                                                                                                                                                     100%   35MB  11.5MB/s   00:03    
mk-docker-opts.sh                                                                                                                                            100% 2139    40.6KB/s   00:00    
README.md                                                                                                                                                    100% 4298   109.4KB/s   00:00    
flanneld                                                                                                                                                     100%  235    55.1KB/s   00:00  
[root@node-2 ~]# mkdir /opt/etcd   node-2創建目錄

[root@node-1 ~]# scp -r /opt/etcd/ssl 192.168.10.14:/opt/etcd/ 
root@192.168.10.14's password: 
ca.pem                                                                                                                                                       100% 1265    70.7KB/s   00:00    
server-key.pem                                                                                                                                               100% 1675    79.2KB/s   00:00    
server.pem    
node-2啟動                                                                                                                                               100% 1338    39.5KB/s   00:00 
[root@node-2 ~]# systemctl daemon-reload
[root@node-2 ~]# systemctl restart flanneld
[root@node-2 ~]# systemctl restart docker
[root@node-2 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:0c:29:e9:c2:41 brd ff:ff:ff:ff:ff:ff
    inet 192.168.10.14/24 brd 192.168.10.255 scope global noprefixroute ens33
       valid_lft forever preferred_lft forever
    inet6 fe80::85fd:b3b3:c97:eca3/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 02:42:28:a8:bb:18 brd ff:ff:ff:ff:ff:ff
    inet 172.17.82.1/24 scope global docker0
       valid_lft forever preferred_lft forever
4: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default 
    link/ether 42:02:5f:e8:9d:d8 brd ff:ff:ff:ff:ff:ff
    inet 172.17.82.0/32 scope global flannel.1
       valid_lft forever preferred_lft forever
    inet6 fe80::4002:5fff:fee8:9dd8/64 scope link 
       valid_lft forever preferred_lft forever

　　添加路由，容器互通；注意正常應該是Flannel自己添加路由的，可能因為我沒裝route工具原因吧

[root@node-1 ~]# route add -net 172.17.82.0/24 gw 192.168.10.14   node-1添加的路由
[root@node-2 ~]# route add -net 172.17.8.0/24 gw 192.168.10.13   node-2 路由
[root@node-1 ~]# docker run -it busybox sh
/ # ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
5: eth0@if6: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1450 qdisc noqueue 
    link/ether 02:42:ac:11:08:02 brd ff:ff:ff:ff:ff:ff
    inet 172.17.8.2/24 brd 172.17.8.255 scope global eth0
       valid_lft forever preferred_lft forever
node-2容器
/ # ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
15: eth0@if16: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue 
    link/ether 02:42:ac:11:52:02 brd ff:ff:ff:ff:ff:ff
    inet 172.17.82.2/24 brd 172.17.82.255 scope global eth0
       valid/ # ping 172.17.8.2
PING 172.17.8.2 (172.17.8.2): 56 data bytes
64 bytes from 172.17.8.2: seq=3283 ttl=62 time=0.944 ms
64 bytes from 172.17.8.2: seq=3284 ttl=62 time=0.950 ms
64 bytes from 172.17.8.2: seq=3285 ttl=62 time=0.712 ms

　　查看生產網絡配置

[root@node-1 ~]# /opt/etcd/bin/etcdctl --ca-file=/opt/etcd/ssl/ca.pem --cert-file=/opt/etcd/ssl/server.pem --key-file=/opt/etcd/ssl/server-key.pem --endpoints="https://192.168.10.11:2379,https://192.168.10.12:2379,https://192.168.10.13:2379" ls /coreos.com/network
/coreos.com/network/config
/coreos.com/network/subnets
[root@node-1 ~]# /opt/etcd/bin/etcdctl --ca-file=/opt/etcd/ssl/ca.pem --cert-file=/opt/etcd/ssl/server.pem --key-file=/opt/etcd/ssl/server-key.pem --endpoints="https://192.168.10.11:2379,https://192.168.10.12:2379,https://192.168.10.13:2379" ls /coreos.com/network/subnets
/coreos.com/network/subnets/172.17.8.0-24
/coreos.com/network/subnets/172.17.82.0-24

　　查看etcd里網絡設置

[root@node-1 ~]# /opt/etcd/bin/etcdctl --ca-file=/opt/etcd/ssl/ca.pem --cert-file=/opt/etcd/ssl/server.pem --key-file=/opt/etcd/ssl/server-key.pem --endpoints="https://192.168.10.11:2379,https://192.168.10.12:2379,https://192.168.10.13:2379" get /coreos.com/network/subnets/172.17.8.0-24
 

{"PublicIP":"192.168.10.13","BackendType":"vxlan","BackendData"： {"VtepMAC":"56:2f:96:00:5c:05"}}
[root@node-1 ~]# /opt/etcd/bin/etcdctl --ca-file=/opt/etcd/ssl/ca.pem --cert-file=/opt/etcd/ssl/server.pem --key-file=/opt/etcd/ssl/server-key.pem --endpoints="https://192.168.10.11:2379,https://192.168.10.12:2379,https://192.168.10.13:2379" get /coreos.com/network/subnets/172.17.82.0-24


{"PublicIP":"192.168.10.14","BackendType":"vxlan","BackendData":{"VtepMAC":"42:02:5f:e8:9d:d8"}}