docker使用 Flannel（etcd+flannel）網絡

一、Flannel網絡簡介

Flannel是一種基於overlay網絡的跨主機容器網絡解決方案，也就是將TCP數據包封裝在另一種網絡包里面進行路由轉發和通信，Flannel是CoreOS開發,專門用於docker多機互聯的一個工具,讓集群中的不同節點主機創建的容器都具有全集群唯一的虛擬ip地址，Flannel使用go語言編寫

二、Flannel實現原理

2.1、原理說明

1、Flannel為每個host分配一個subnet，容器從這個subnet中分配IP，這些IP可以在host間路由，容器間無需使用nat和端口映射即可實現跨主機通信

2、每個subnet都是從一個更大的IP池中划分的，flannel會在每個主機上運行一個叫flanneld的agent，其職責就是從池子中分配subnet

3、Flannel使用etcd存放網絡配置、已分配 的subnet、host的IP等信息

4、Flannel數據包在主機間轉發是由backend實現的，目前已經支持UDP、VxLAN、host-gw、AWS VPC和GCE路由等多種backend

2.2、數據轉發流程

1、容器直接使用目標容器的ip訪問，默認通過容器內部的eth0發送出去。

2、報文通過veth pair被發送到vethXXX。 3、ethXXX是直接連接到虛擬交換機docker0的，報文通過虛擬bridge docker0發送出去。 4、查找路由表，外部容器ip的報文都會轉發到flannel0虛擬網卡，這是一個P2P的虛擬網卡，然后報文就被轉發到監聽在另一端的flanneld。 5、flanneld通過etcd維護了各個節點之間的路由表，把原來的報文UDP封裝一層，通過配置的iface發送出去。 6、報文通過主機之間的網絡找到目標主機。 7、報文繼續往上，到傳輸層，交給監聽在8285端口的flanneld程序處理。 8、數據被解包，然后發送給flannel0虛擬網卡。 9、查找路由表，發現對應容器的報文要交給docker0。 10、docker0找到連到自己的容器，把報文發送過去。

三、部署etcd集群

3.1、環境准備

節點名稱	IP地址	安裝軟件
node1	192.168.0.115	etcd
node2	192.168.0.116	etcd
node3	192.168.0.117	etcd

3.2、安裝etcd

# yum -y install etcd

3.3、配置etcd

# cp /etc/etcd/etcd.conf{,_bak}

【注釋：每個ETCD_NAME必須不同，綠色部分的ip為當前宿主機的ip】
# grep -v '^#' /etc/etcd/etcd.conf
ETCD_NAME="node1"
ETCD_DATA_DIR="/var/lib/etcd/node1.etcd"
ETCD_LISTEN_PEER_URLS="http://192.168.0.115:2380"
ETCD_LISTEN_CLIENT_URLS="http://192.168.0.115:2379,http://127.0.0.1:2379"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.0.115:2380"
ETCD_ADVERTISE_CLIENT_URLS="http://192.168.0.115:2379"
ETCD_INITIAL_CLUSTER="node1=http://192.168.0.115:2380,node2=http://192.168.0.116:2380,node3=http://192.168.0.117:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"

3.4、修改etcd啟動文件

# cp /usr/lib/systemd/system/etcd.service{,_bak}

# cat /usr/lib/systemd/system/etcd.service
[Service]
Type=notify
WorkingDirectory=/var/lib/etcd/
EnvironmentFile=-/etc/etcd/etcd.conf
User=etcd
# set GOMAXPROCS to number of processors
ExecStart=/bin/bash -c "GOMAXPROCS=$(nproc) /usr/bin/etcd \
--name=\"${ETCD_NAME}\" \
--data-dir=\"${ETCD_DATA_DIR}\" \
--listen-peer-urls=\"${ETCD_LISTEN_PEER_URLS}\" \
--listen-client-urls=\"${ETCD_LISTEN_CLIENT_URLS}\" \
--initial-advertise-peer-urls=\"${ETCD_INITIAL_ADVERTISE_PEER_URLS}\" \
--advertise-client-urls=\"${ETCD_ADVERTISE_CLIENT_URLS}\" \
--initial-cluster=\"${ETCD_INITIAL_CLUSTER}\"  \
--initial-cluster-token=\"${ETCD_INITIAL_CLUSTER_TOKEN}\" \
--initial-cluster-state=\"${ETCD_INITIAL_CLUSTER_STATE}\""
Restart=on-failure
LimitNOFILE=65536

3.5、啟動etcd服務

【注釋：另外兩台服務器，操作也如上】
# systemctl start etcd.service

3.6、檢測etcd集群狀態，至此etcd安裝完成

# 查看cluster狀態
# etcdctl cluster-health
member 3e398d43ae9c8720 is healthy: got healthy result from http://192.168.0.116:2379
member 65368524050cc2e8 is healthy: got healthy result from http://192.168.0.115:2379
member d8ff06c8c9b413da is healthy: got healthy result from http://192.168.0.117:2379
cluster is healthy

# 列出etcd服務狀態，從列出信息可以看出，目前是node2為主節點。
# etcdctl member list
3e398d43ae9c8720: name=node2 peerURLs=http://192.168.0.116:2380 clientURLs=http://192.168.0.116:2379 isLeader=true
65368524050cc2e8: name=node1 peerURLs=http://192.168.0.115:2380 clientURLs=http://192.168.0.115:2379 isLeader=false
d8ff06c8c9b413da: name=node3 peerURLs=http://192.168.0.117:2380 clientURLs=http://192.168.0.117:2379 isLeader=false

3.7、添加flannel網絡配置信息到etcd

【注釋： 此（flannel_use）目錄自己可以定義，但是此處設置的目錄必須與flannel配置文件中FLANNEL_ETCD_PREFIX="/flannel_use/network"配置保持一致，flannel啟動程序只認帶“config”的key，否則會報錯Not a directory (/flannel_use/network)】 

# 固定配置方式
# etcdctl set /flannel_use/network/config '{"Network":"10.10.0.0/16"}'

四、部署flannel

4.1、安裝flannel

# yum install -y flannel

4.2、修改flannel配置文件

# cp /etc/sysconfig/flanneld{,_bak}

# cat /etc/sysconfig/flanneld
# Flanneld configuration options  
# etcd url location.  Point this to the server where etcd runs
FLANNEL_ETCD_ENDPOINTS="http://192.168.0.115:2379,http://192.168.0.116:2379,http://192.168.0.117:2379"
# etcd config key.  This is the configuration key that flannel queries
# For address range assignment
FLANNEL_ETCD_PREFIX="/flannel_use/network"
# Any additional options that you want to pass
#FLANNEL_OPTIONS=""

4.3、啟動flannel

# systemctl start flanneld


# systemctl status flanneld
● flanneld.service - Flanneld overlay address etcd agent
   Loaded: loaded (/usr/lib/systemd/system/flanneld.service; disabled; vendor preset: disabled)
   Active: active (running) since Mon 2019-12-23 15:01:07 CST; 4s ago
  Process: 1828 ExecStartPost=/usr/libexec/flannel/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/docker (code=exited, status=0/SUCCESS)
 Main PID: 1817 (flanneld)
   Memory: 18.8M
   CGroup: /system.slice/flanneld.service
           └─1817 /usr/bin/flanneld -etcd-endpoints=http://192.168.0.115:2379,http://192.168.0.116:2379,http://192.168.0.117:2379 -etcd-prefix=/flannel_use/network

Dec 23 15:01:07 nanan-product-yanpan-bigdate01 flanneld-start[1817]: I1223 15:01:07.365994    1817 main.go:132] Installing signal handlers
Dec 23 15:01:07 nanan-product-yanpan-bigdate01 flanneld-start[1817]: I1223 15:01:07.366705    1817 manager.go:136] Determining IP address of default interface
Dec 23 15:01:07 nanan-product-yanpan-bigdate01 flanneld-start[1817]: I1223 15:01:07.366916    1817 manager.go:149] Using interface with name eth0 and address 192.168.0.109
Dec 23 15:01:07 nanan-product-yanpan-bigdate01 flanneld-start[1817]: I1223 15:01:07.366933    1817 manager.go:166] Defaulting external address to interface address (192.168.0.109)
Dec 23 15:01:07 nanan-product-yanpan-bigdate01 flanneld-start[1817]: I1223 15:01:07.375600    1817 local_manager.go:179] Picking subnet in range 10.10.1.0 ... 10.10.255.0
Dec 23 15:01:07 nanan-product-yanpan-bigdate01 flanneld-start[1817]: I1223 15:01:07.383110    1817 manager.go:250] Lease acquired: 10.10.88.0/24
Dec 23 15:01:07 nanan-product-yanpan-bigdate01 flanneld-start[1817]: I1223 15:01:07.383333    1817 network.go:98] Watching for new subnet leases
Dec 23 15:01:07 nanan-product-yanpan-bigdate01 flanneld-start[1817]: I1223 15:01:07.388324    1817 network.go:191] Subnet added: 10.10.65.0/24
Dec 23 15:01:07 nanan-product-yanpan-bigdate01 flanneld-start[1817]: I1223 15:01:07.388344    1817 network.go:191] Subnet added: 10.10.50.0/24
Dec 23 15:01:07 nanan-product-yanpan-bigdate01 systemd[1]: Started Flanneld overlay address etcd agent.

# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 00:16:3e:2c:36:be brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.109/24 brd 192.168.0.255 scope global dynamic eth0
       valid_lft 314756444sec preferred_lft 314756444sec
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN 
    link/ether 02:42:30:1b:b8:fd brd ff:ff:ff:ff:ff:ff
    inet 10.10.0.1/24 brd 10.10.0.255 scope global docker0
       valid_lft forever preferred_lft forever
4: flannel0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1472 qdisc pfifo_fast state UNKNOWN qlen 500
    link/none 
    inet 10.10.88.0/16 scope global flannel0
       valid_lft forever preferred_lft forever

4.4、注釋

啟動Flannel后，一定要記得重啟docker，這樣Flannel配置分配的ip才能生效，即docker0虛擬網卡的ip會變成上面flannel設定的ip段

4.5、修改docker啟動/配置文件使用flannel網絡

# cat /usr/lib/systemd/system/docker.service 
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
BindsTo=containerd.service
After=network-online.target firewalld.service containerd.service
Wants=network-online.target
Requires=docker.socket
[Service]
Type=notify
# the default is not to use systemd for cgroups because the delegate issues still
# exists and systemd currently does not support the cgroup feature set required
# for containers run by docker
ExecStart=/usr/bin/dockerd --insecure-registry=172.17.29.74 -H fd:// --containerd=/run/containerd/containerd.sock $DOCKER_NETWORK_OPTIONS
ExecReload=/bin/kill -s HUP $MAINPID
TimeoutSec=0
RestartSec=2
Restart=always
# Note that StartLimit* options were moved from "Service" to "Unit" in systemd 229.
# Both the old, and new location are accepted by systemd 229 and up, so using the old location
# to make them work for either version of systemd.
StartLimitBurst=3
# Note that StartLimitInterval was renamed to StartLimitIntervalSec in systemd 230.
# Both the old, and new name are accepted by systemd 230 and up, so using the old name to make
# this option work for either version of systemd.
StartLimitInterval=60s
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
# Comment TasksMax if your systemd version does not support it.
# Only systemd 226 and above support this option.
TasksMax=infinity
# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes
# kill only the docker process, not all processes in the cgroup
KillMode=process
[Install]
WantedBy=multi-user.target


# cat /etc/docker/daemon.json 
{
  "registry-mirrors": ["https://registry.docker-cn.com"]
}

4.6、重啟docker

# systemctl daemon-reload

# systemctl restart docker

4.7、查看docker是否使用flannel網絡

# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 00:16:3e:2c:36:be brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.109/24 brd 192.168.0.255 scope global dynamic eth0
       valid_lft 314756133sec preferred_lft 314756133sec
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN 
    link/ether 02:42:30:1b:b8:fd brd ff:ff:ff:ff:ff:ff
    inet 10.10.88.1/24 brd 10.10.88.255 scope global docker0
       valid_lft forever preferred_lft forever
4: flannel0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1472 qdisc pfifo_fast state UNKNOWN qlen 500
    link/none 
    inet 10.10.88.0/16 scope global flannel0
       valid_lft forever preferred_lft forever

4.8、如果容器無法聯通，是由於flannel.0網卡和docker0網卡通過iptables的forward轉發，所以需確保如下設置

1、核中的forward功能開啟(立即生效，重啟后效果不再)
# echo "1" > /proc/sys/net/ipv4/ip_forward
　　
2、包不會被iptables的forward規則攔截
# iptables -P FORWARD ACCEPT