碼上快樂

相關內容    簡體    繁體

Docker - 用Flannel跨主機

本文轉載自   查看原文   2017-01-17 18:20   4484    Tools and Tips/ docker

試了下比較流行的幾種SDN，感覺flannel還是比較好用，這里簡單記錄一下。

 

用的是virtualbox，3個機器，分別為:

  • genesis : inet 192.168.99.103/24 brd 192.168.99.255 scope global dynamic enp0s3
  • exodus : inet 192.168.99.105/24 brd 192.168.99.255 scope global dynamic enp0s3
  • leviticus : inet 192.168.99.106/24 brd 192.168.99.255 scope global dynamic enp0s3


虛機信息如下

[root@localhost yum.repos.d]# uname -mars  
Linux localhost.localdomain 3.10.0-229.el7.x86_64 #1 SMP Fri Mar 6 11:36:42 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

 

[root@localhost yum.repos.d]# cat /etc/*-release
CentOS Linux release 7.3.1611 (Core)
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

CentOS Linux release 7.3.1611 (Core)
CentOS Linux release 7.3.1611 (Core)

[root@localhost yum.repos.d]# docker version
Client:
 Version:      1.12.5
 API version:  1.24
 Go version:   go1.6.4
 Git commit:   7392c3b
 Built:        Fri Dec 16 02:23:59 2016
 OS/Arch:      linux/amd64


隨便選擇兩台機器run一下，在容器中ifconfig:

[root@localhost ~]# docker run -it busybox
/ # ifconfig
eth0      Link encap:Ethernet  HWaddr 02:42:AC:11:00:02
          inet addr:172.17.0.2  Bcast:0.0.0.0  Mask:255.255.0.0
          inet6 addr: fe80::42:acff:fe11:2/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:12 errors:0 dropped:0 overruns:0 frame:0
          TX packets:6 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:1016 (1016.0 B)  TX bytes:508 (508.0 B)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

 

發現參數完全相同，單純bridge模式下並沒有跨主機互通，而host模式是並不建議使用的。

 

install

先yum install -y etcd flannel，如果沒問題是再好不過了。

etcd 3.x支持--config-file參數，需要的話可以從源代碼install(需要golang 1.6+)。

 

先從etcd開始，簡單說就是"distributed key value store"。

 

etcd集群的3種方式:

  • static
  • etcd discovery
  • DNS discovery

 

DNS discovery主要是用srv record，這里先不搞DNS服務，下面對static和etcd discovery兩種方式簡單說明一下。

 

static

參數可以在啟動時置頂，或者寫到配置文件中，默認配置文件為/etc/etcd/etcd.conf。

 

genesis的配置如下:

ETCD_NAME=genesis
ETCD_DATA_DIR="/var/lib/etcd/genesis"
ETCD_LISTEN_PEER_URLS="http://192.168.99.103:2380"
ETCD_LISTEN_CLIENT_URLS="http://192.168.99.103:2379,http://127.0.0.1:2379"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.99.103:2380"
ETCD_ADVERTISE_CLIENT_URLS="http://192.168.99.103:2379"
ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_INITIAL_CLUSTER_TOKEN="etct-fantasy"
ETCD_INITIAL_CLUSTER="exodus=http://192.168.99.105:2380,genesis=http://192.168.99.103:2380"

 

exodus的配置如下:

ETCD_NAME=exodus
ETCD_DATA_DIR="/var/lib/etcd/exodus"
ETCD_LISTEN_PEER_URLS="http://192.168.99.105:2380"
ETCD_LISTEN_CLIENT_URLS="http://192.168.99.105:2379,http://127.0.0.1:2379"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.99.105:2380"
ETCD_ADVERTISE_CLIENT_URLS="http://192.168.99.105:2379"
ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_INITIAL_CLUSTER_TOKEN="etct-fantasy"
ETCD_INITIAL_CLUSTER="exodus=http://192.168.99.105:2380,genesis=http://192.168.99.103:2380"

 

啟動方式看自己喜好，如果打算用systemctl啟動的話，注意/usr/lib/systemd/system/etcd.service中的內容可能不會如你所願。

 

啟動后，檢查一下集群狀態，順便也看看有哪些成員:

[root@localhost etcd]# etcdctl cluster-health
member 7a4f27f78a05e755 is healthy: got healthy result from http://192.168.99.103:2379
failed to check the health of member 8e8718b335c6c9a2 on http://192.168.99.105:2379: Get http://192.168.99.105:2379/health: dial tcp 192.168.99.105:2379: i/o timeout
member 8e8718b335c6c9a2 is unreachable: [http://192.168.99.105:2379] are all unreachable
cluster is healthy

 

提示"member unreachable"，看來是被exodus的防火牆攔住了，我們先粗暴一點。

[root@localhost etcd]# systemctl stop firewalld
[root@localhost etcd]# etcdctl cluster-health
member 7a4f27f78a05e755 is healthy: got healthy result from http://192.168.99.103:2379
member 8e8718b335c6c9a2 is healthy: got healthy result from http://192.168.99.105:2379
cluster is healthy

 

etcd discovery

當然，這樣配置的前提是已經知道各個節點的信息。

但實際場景中可能無法預知各個member，所以我們需要讓etcd自己去發現(discovery)。

 

首先，etcd提供了一個public discovery service - discovery.etcd.io，我們用它來生成一個discovery token，並在genesis創建目錄:

[root@localhost etcd]# curl https://discovery.etcd.io/new?size=3
https://discovery.etcd.io/6321c0706046c91f2b2598206ffa3272
[root@localhost etcd]# etcdctl set /discovery/6321c0706046c91f2b2598206ffa3272/_config/size 3

 

修改exodus的配置，用discovery代替之前的cluster:

ETCD_NAME=exodus
ETCD_DATA_DIR="/var/lib/etcd/exodus"
ETCD_LISTEN_PEER_URLS="http://192.168.99.105:2380"
ETCD_LISTEN_CLIENT_URLS="http://192.168.99.105:2379,http://127.0.0.1:2379"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.99.105:2380"
ETCD_ADVERTISE_CLIENT_URLS="http://192.168.99.105:2379"
ETCD_DISCOVERY=http://192.168.99.103:2379/v2/keys/discovery/98a976dac265a218f1a1959eb8dde57f

 

如果啟動后一直顯示如下錯誤(參考: raft election):

rafthttp: the clock difference against peer ?????? is too high [??????s > 1s]

 

簡單的解決方法是通過ntp:

[root@localhost etcd]yum install ntp -y
[root@localhost etcd]# systemctl enable ntpd
Created symlink from /etc/systemd/system/multi-user.target.wants/ntpd.service to /usr/lib/systemd/system/ntpd.service.
[root@localhost etcd]# systemctl start ntpd

 

flannel

set一個路徑給flannel用:

etcdctl set /coreos.com/network/config '{ "Network": "10.1.0.0/16" }'

 

以systemctl start flanneld方式啟動時，如果出現以下錯誤

network.go:53] Failed to retrieve network config: 100: Key not found (/coreos.net) [9]

 

注意/etc/sysconfig/flanneld中的內容，FLANNEL_ETCD_PREFIX很可能是/atomic.io/network，將其改為/coreos.com/network。
或者也可以通過-etcd-prefix指定。

 

啟動成功后，查看subnet:

[root@localhost etcd]# etcdctl ls /coreos.com/network/subnets
/coreos.com/network/subnets/10.1.90.0-24
/coreos.com/network/subnets/10.1.30.0-24
/coreos.com/network/subnets/10.1.18.0-24

 

flannel啟動成功后會生成/run/flannel/docker，內容如下:

DOCKER_OPT_BIP="--bip=10.1.30.1/24"
DOCKER_OPT_IPMASQ="--ip-masq=true"
DOCKER_OPT_MTU="--mtu=1450"
DOCKER_NETWORK_OPTIONS=" --bip=10.1.30.1/24 --ip-masq=true --mtu=1450 "

 

用以下方式啟動docker:

[root@localhost etcd]# source /run/flannel/docker
[root@localhost etcd]# docker daemon ${DOCKER_NETWORK_OPTIONS} >> /dev/null 2>&1 &

 

／run/flannel/docker是怎么來的？

參考flanneld的兩個啟動參數，-subnet-dir和-subnet-file。

 

在genesis進入容器看看效果:

[root@localhost etcd]# docker run -it busybox
/ # ifconfig
eth0      Link encap:Ethernet  HWaddr 02:42:0A:01:5A:02
          inet addr:10.1.90.2  Bcast:0.0.0.0  Mask:255.255.255.0
          inet6 addr: fe80::42:aff:fe01:5a02/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1450  Metric:1
          RX packets:6 errors:0 dropped:0 overruns:0 frame:0
          TX packets:6 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:508 (508.0 B)  TX bytes:508 (508.0 B)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

 

在exodus也做類似操作，在exodus中ping一下10.1.90.2，發現是通的。
並在各自容器中也ping一下，檢查是否跨主機互通。

 


免責聲明！

本站轉載的文章為個人學習借鑒使用，本站對版權不負任何法律責任。如果侵犯了您的隱私權益，請聯系本站郵箱yoyou2525@163.com刪除。



猜您在找 安裝docker跨主機網絡flannel 部署Flannel，實現跨主機Docker容器通信 docker跨主機互聯 Docker的跨主機連接： K8S Flannel容器跨主機通信淺析 Docker跨主機網絡——overlay docker-跨主機存儲 Docker跨主機網絡實踐 CentOS安裝etcd和flannel實現Docker跨物理機通信 docker 實踐十一：docker 跨主機通訊
 
粵ICP備18138465號   © 2018-2025 CODEPRJ.COM