一、建立Consul Cluster环境
利用Consul提供的服务实现服务的注册与发现,需要建立Consul Cluster。在Consul方案中,每个提供服务的节点上都要部署和运行Consul的agent,所有运行Consul agent节点的集合构成Consul Cluster。Consul agent有两种运行模式:Server和Client。这里的Server和Client只是Consul集群层面的区分,与搭建在Cluster之上 的应用服务无关。以Server模式运行的Consul agent节点用于维护Consul集群的状态,官方建议每个Consul Cluster至少有3个或以上的运行在Server mode的Agent,Client节点不限。
Consul架构说明

二.consul集群配置
两种配置方式:《1》通过进程管理器(supervisor)配置。前提是consul已经被supervisor管理,因此推荐新手使用命令行方式配置consul集群。
《2》通过consul命令行配置(此方式配置的consul集群将不被supervisor管理,在supervisor进程管理器上的状态会显示为FATAL)。
注:若是docker启动的,建议用命令行方式配置consul集群。
《1》通过进程管理器配置
1.修改启动脚本命令(以3台机器为例)
关闭防火墙
systemctl stop firewalld
vi /etc/supervisord.d/supervisor.ini
[program:consul]
priority = 1
user = yy
command
=
/usr/bin/consul
agent
-server
-data-
dir
=
"/data/yy-monitor-server/data/consul-data"
-bootstrap-expect=3
#集群中最少consul实例数为3
-client=
"0.0.0.0"
-bind=
"本机ip"
# 如 -bind="10.22.0.1002"
-ui
stdout_logfile =
/data/yy-monitor-server/log/consul
.log
stderr_logfile =
/data/yy-monitor-server/log/consul
.log
autorestart =
true
|
[program:consul]
priority = 1
user = yy
command
=
/usr/bin/consul
agent
-server
-data-
dir
=
"/data/yy-monitor-server/data/consul-data"
-client=
"0.0.0.0"
-bind=
"本机ip"
# 如-bind="10.22.0.1001"
join
=“其他consul ip”
# 如join=“10.22.0.1002”
-ui
stdout_logfile =
/data/yy-monitor-server/log/consul
.log
stderr_logfile =
/data/yy-monitor-server/log/consul
.log
autorestart =
true
|
2.重启配置
#systemctl restart supervisord
# supervisorctl restart consul
3.查看日志
tail -300f consul.log
==> Starting Consul agent...
==> Consul agent running!
Version:
'v0.9.2'
Node ID:
'035f828b-8db9-8cb1-a3a8-8eabf5e821c4'
Node name:
'monitor-test-02.yfb.sun.cn'
Datacenter:
'dc1'
Server:
true
(bootstrap:
false
)
Client Addr: 0.0.0.0 (HTTP: 8500, HTTPS: -1, DNS: 8600)
Cluster Addr: 10.22.0.1001 (LAN: 8301, WAN: 8302)
Gossip encrypt:
false
, RPC-TLS:
false
, TLS-Incoming:
false
==> Log data will now stream
in
as it occurs:
2018
/08/27
13:43:20 [INFO] raft: Restored from snapshot 1370-114688-1535337744933
2018
/08/27
13:43:20 [INFO] raft: Initial configuration (index=117709): [{Suffrage:Voter ID:10.22.0.1001:8300 Address:10.22.0.1001:8300} {Suffrage:Voter ID:10.22.0.1002:8300 Address:10.22.0.1002:8300} {Suffrage:Voter ID:10.22.0.1000:8300 Address:10.22.0.1000:8300}]
2018
/08/27
13:43:20 [INFO] serf: EventMemberJoin: monitor-
test
-02.yfb.sun.cn.dc1 10.22.0.1001
2018
/08/27
13:43:20 [INFO] raft: Node at 10.22.0.1001:8300 [Follower] entering Follower state (Leader:
""
)
2018
/08/27
13:43:20 [INFO] serf: Attempting re-
join
to previously known node: n2.dc1: 10.22.0.1001:8302
2018
/08/27
13:43:20 [INFO] serf: EventMemberJoin: monitor-
test
-02.yfb.sun.cn 10.22.0.1001
2018
/08/27
13:43:20 [INFO] agent: Started DNS server 0.0.0.0:8600 (udp)
2018
/08/27
13:43:20 [INFO] serf: Attempting re-
join
to previously known node: monitor-
test
-01.yfb.sun.cn: 10.22.0.1000:8301
2018
/08/27
13:43:20 [INFO] consul: Adding LAN server monitor-
test
-02.yfb.sun.cn (Addr: tcp
/10
.22.0.1001:8300) (DC: dc1)
2018
/08/27
13:43:20 [INFO] consul: Handled member-
join
event
for
server
"monitor-test-02.yfb.sun.cn.dc1"
in
area
"wan"
2018
/08/27
13:43:20 [INFO] agent: Started DNS server 0.0.0.0:8600 (tcp)
2018
/08/27
13:43:20 [INFO] agent: Started HTTP server on [::]:8500
2018
/08/27
13:43:20 [INFO] serf: Re-joined to previously known node: n2.dc1: 10.22.0.1001:8302
2018
/08/27
13:43:20 [WARN] memberlist: Refuting a suspect message (from: monitor-
test
-02.yfb.sun.cn)
2018
/08/27
13:43:20 [INFO] serf: EventMemberJoin: monitor-
test
-03.yfb.sun.cn 10.22.0.1002
2018
/08/27
13:43:20 [INFO] serf: EventMemberJoin: monitor-
test
-01.yfb.sun.cn 10.22.0.1000
2018
/08/27
13:43:20 [INFO] serf: Re-joined to previously known node: monitor-
test
-01.yfb.sun.cn: 10.22.0.1000:8301
2018
/08/27
13:43:20 [INFO] consul: Adding LAN server monitor-
test
-03.yfb.sun.cn (Addr: tcp
/10
.22.0.1002:8300) (DC: dc1)
2018
/08/27
13:43:20 [INFO] consul: Adding LAN server monitor-
test
-01.yfb.sun.cn (Addr: tcp
/10
.22.0.1000:8300) (DC: dc1)
2018
/08/27
13:43:20 [INFO] serf: EventMemberJoin: monitor-
test
-01.yfb.sun.cn.dc1 10.22.0.1000
2018
/08/27
13:43:20 [WARN] memberlist: Refuting an alive message
2018
/08/27
13:43:20 [INFO] serf: EventMemberJoin: monitor-
test
-03.yfb.sun.cn.dc1 10.22.0.1002
2018
/08/27
13:43:20 [INFO] consul: Handled member-
join
event
for
server
"monitor-test-01.yfb.sun.cn.dc1"
in
area
"wan"
2018
/08/27
13:43:20 [INFO] consul: Handled member-
join
event
for
server
"monitor-test-03.yfb.sun.cn.dc1"
in
area
"wan"
2018
/08/27
13:43:20 [INFO] agent: Synced node info
2018
/08/27
13:43:54 [INFO] memberlist: Suspect monitor-
test
-01.yfb.sun.cn has failed, no acks received
2018
/08/27
13:43:54 [INFO] serf: EventMemberUpdate: monitor-
test
-01.yfb.sun.cn
2018
/08/27
13:43:55 [INFO] serf: EventMemberUpdate: monitor-
test
-01.yfb.sun.cn.dc1
2018
/08/27
13:43:56 [INFO] serf: EventMemberUpdate: monitor-
test
-01.yfb.sun.cn.dc1
2018
/08/27
13:43:57 [INFO] serf: EventMemberUpdate: monitor-
test
-01.yfb.sun.cn
==> Failed to check
for
updates: Get https:
//checkpoint-api
.hashicorp.com
/v1/check/consul
?arch=amd64&os=linux&signature=143c3fc1-0a07-2f4e-79e3-528773c5260f&version=0.9.2: dial tcp 54.221.203.30:443: i
/o
timeout
2018
/08/27
13:44:01 [WARN] raft: Rejecting vote request from 10.22.0.1002:8300 since we have a leader: 10.22.0.1000:8300
2018
/08/27
13:44:01 [INFO] consul: New leader elected: monitor-
test
-03.yfb.sun.cn
|
注:更新异常可忽略,内网云桌面不能连接github。
4 查看当前节点信息
使用命令 consul info
每次启动之后,都会有一台部署有consul的机器被选举为leader节点:
state = Follower
|
5.ui页面
访问任意节点localhost:8500 ,如下图
《2》通过consul命令行配置
1.部署前
关闭防火墙
#因consul集群间需通信,关闭部署monitor的虚拟机的防火墙
systemctl stop firewalld
|
分别进入到monitor的容器中
#monitor-server为容器名,按实际情况修改
docker
exec
-
it monitor
-
server
/
bin
/
bash
#停止consul
supervisorctl stop consul
[root@monitor
-
test
-
01
~]
# supervisorctl
alertmanager RUNNING pid
22016
, uptime
21
:
28
:
23
consul RUNNING pid
11109
, uptime
0
:
00
:
07
dashboard
-
upgrade EXITED Aug
14
05
:
51
PM
yy
-
echo
-
webhook RUNNING pid
22025
, uptime
21
:
28
:
23
grafana RUNNING pid
22015
, uptime
21
:
28
:
23
nginx RUNNING pid
22017
, uptime
21
:
28
:
23
node_exporter RUNNING pid
22020
, uptime
21
:
28
:
23
prometheus RUNNING pid
22018
, uptime
21
:
28
:
23
supervisor> stop consul
consul: stopped
|
server:
1.1 部署主节点
运行命令:consul agent -server -bootstrap-expect 以sever方式运行的主机个数 -client 0.0.0.0 -bind 主机地址 -data-dir=/home/consul/ -ui &
[root@monitor-
test
-01 ~]
# consul agent -server -bootstrap-expect 3 -client 0.0.0.0 -bind 10.22.0.1000 -data-dir=/home/consul/ -ui &
==> WARNING: Expect Mode enabled, expecting 3 servers
==> Starting Consul agent...
==> Consul agent running!
Version:
'v0.9.2'
Node ID:
'8fea634e-c3d6-32ef-f8d4-4ed1a2c0ee42'
Node name:
'monitor-test-01.yfb.sun.cn'
Datacenter:
'dc1'
Server:
true
(bootstrap:
false
)
Client Addr: 0.0.0.0 (HTTP: 8500, HTTPS: -1, DNS: 8600)
Cluster Addr: 10.22.0.1000 (LAN: 8301, WAN: 8302)
Gossip encrypt:
false
, RPC-TLS:
false
, TLS-Incoming:
false
==> Log data will now stream
in
as it occurs:
2018
/08/15
16:40:26 [INFO] raft: Initial configuration (index=1): [{Suffrage:Voter ID:10.22.0.1000:8300 Address:10.22.0.1000:8300} {Suffrage:Voter ID:10.22.0.1001:8300 Address:10.22.0.1001:8300} {Suffrage:Voter ID:10.22.0.1002:8300 Address:10.22.0.1002:8300}]
2018
/08/15
16:40:26 [INFO] serf: EventMemberJoin: monitor-
test
-01.yfb.sun.cn.dc1 10.22.0.1000
2018
/08/15
16:40:26 [INFO] serf: EventMemberJoin: monitor-
test
-01.yfb.sun.cn 10.22.0.1000
2018
/08/15
16:40:26 [INFO] agent: Started DNS server 0.0.0.0:8600 (udp)
2018
/08/15
16:40:26 [INFO] raft: Node at 10.22.0.1000:8300 [Follower] entering Follower state (Leader:
""
)
2018
/08/15
16:40:26 [INFO] serf: Attempting re-
join
to previously known node: monitor-
test
-02.yfb.sun.cn.dc1: 10.22.0.1001:8302
2018
/08/15
16:40:26 [INFO] serf: Attempting re-
join
to previously known node: monitor-
test
-02.yfb.sun.cn: 10.22.0.1001:8301
2018
/08/15
16:40:26 [INFO] consul: Adding LAN server monitor-
test
-01.yfb.sun.cn (Addr: tcp
/10
.22.0.1000:8300) (DC: dc1)
|
1.2 其他机器运行命令:
consul agent -server -retry-join 主节点ip地址 -bind 本机ip地址 -client 0.0.0.0 -data-dir=/home/consul/ -ui &
[root@monitor-
test
-02 ~]
# consul agent -server -retry-join 10.22.0.1000 -bind 10.22.0.1001 -client 0.0.0.0 -data-dir=/home/consul/ -ui &
==> Starting Consul agent...
==> Consul agent running!
Version:
'v0.9.2'
Node ID:
'9e35e3c2-9226-de85-2202-2d70e516f7b7'
Node name:
'monitor-test-02.yfb.sun.cn'
Datacenter:
'dc1'
Server:
true
(bootstrap:
false
)
Client Addr: 127.0.0.1 (HTTP: 8500, HTTPS: -1, DNS: 8600)
Cluster Addr: 10.22.0.1001 (LAN: 8301, WAN: 8302)
Gossip encrypt:
false
, RPC-TLS:
false
, TLS-Incoming:
false
==> Log data will now stream
in
as it occurs:
2018
/08/15
16:32:51 [INFO] raft: Initial configuration (index=0): []
2018
/08/15
16:32:51 [INFO] serf: EventMemberJoin: monitor-
test
-02.yfb.sun.cn.dc1 10.22.0.1001
2018
/08/15
16:32:51 [INFO] serf: EventMemberJoin: monitor-
test
-02.yfb.sun.cn 10.22.0.1001
2018
/08/15
16:32:51 [INFO] agent: Started DNS server 127.0.0.1:8600 (udp)
2018
/08/15
16:32:51 [INFO] raft: Node at 10.22.0.1001:8300 [Follower] entering Follower state (Leader:
""
)
2018
/08/15
16:32:51 [WARN] serf: Failed to re-
join
any previously known node
2018
/08/15
16:32:51 [WARN] serf: Failed to re-
join
any previously known node
2018
/08/15
16:32:51 [INFO] consul: Adding LAN server monitor-
test
-02.yfb.sun.cn (Addr: tcp
/10
.22.0.1001:8300) (DC: dc1)
2018
/08/15
16:32:51 [INFO] consul: Handled member-
join
event
for
server
"monitor-test-02.yfb.sun.cn.dc1"
in
area
"wan"
2018
/08/15
16:32:51 [INFO] agent: Started DNS server 127.0.0.1:8600 (tcp)
2018
/08/15
16:32:51 [INFO] agent: Started HTTP server on 127.0.0.1:8500
2018
/08/15
16:32:51 [INFO] agent: Retry
join
is supported
for
: aws azure gce softlayer
2018
/08/15
16:32:51 [INFO] agent: Joining cluster...
2018
/08/15
16:32:51 [INFO] agent: (LAN) joining: [10.22.0.1000]
2018
/08/15
16:32:51 [INFO] serf: EventMemberJoin: monitor-
test
-01.yfb.
su
|
1.6 验证集群可用性
1、 consul leave 命令可优雅退出consul集群。
3、可分别访问consul ui界面进行验证。页面访问主机地址即可。
4、如果集群加入不成功,可以用 consul join 命令在非主节点机器上触发Cluster bootstrap过程:consul join 主节点ip地址。
三、代理配置
1.各主机做consul代理配置
# cat nginx.conf
upstream consul.cn{
server 其他机器ip:8500 backup;
server 主机ip:8500;
server 其他机器ip:8500 backup;
}
server {
# Consul UI
location
/consul/
{
proxy_pass http:
//consul
.cn
/ui/
;
}
# Consul API
location
/v1/
{
proxy_pass http:
//consul
.cn
/v1/
;
add_header X-Remote-IP $remote_addr;
add_header X-Server-Time $date_gmt;
}
}
|
2.nginx重启
supervisorctl restart nginx
|
3.验证配置
3.1 可以通过关闭其中一台机器的consul服务,查看日志或ui页面。
supervisorctl stop consul
|
集群中任一consul宕机不会影响consul的使用,数据不丢失。
注:如果其中两台的consul宕机,在集群中需要保证至少两台机器在线,否则无法自选出leader,因此consul不可用。
附录FAQ:
-
命令行参数
-advertise:通知展现地址用来改变我们给集群中的其他节点展现的地址,一般情况下-bind地址就是展现地址
-bootstrap:用来控制一个server是否在bootstrap模式,在一个datacenter中只能有一个server处于bootstrap模式,当一个server处于bootstrap模式时,可以自己选举为raft leader。
-bootstrap-expect:在一个datacenter中期望提供的server节点数目,当该值提供的时候,consul一直等到达到指定sever数目的时候才会引导整个集群,该标记不能和bootstrap公用。
-bind:该地址用来在集群内部的通讯,集群内的所有节点到地址都必须是可达的,默认是0.0.0.0。
-client:consul绑定在哪个client地址上,这个地址提供HTTP、DNS、RPC等服务,默认是127.0.0.1。
-config-file:明确的指定要加载哪个配置文件
-config-dir:配置文件目录,里面所有以.json结尾的文件都会被加载
-data-dir:提供一个目录用来存放agent的状态,所有的agent都需要该目录,该目录必须是稳定的,系统重启后都继续存在。
-dc:该标记控制agent的datacenter的名称,默认是dc1。
-encrypt:指定secret key,使consul在通讯时进行加密,key可以通过consul keygen生成,同一个集群中的节点必须使用相同的key。
-join:加入一个已经启动的agent的ip地址,可以多次指定多个agent的地址。如果consul不能加入任何指定的地址中,则agent会启动失败。默认agent启动时不会加入任何节点。
-retry-join:和join类似,但是允许你在第一次失败后进行尝试。
-retry-interval:两次join之间的时间间隔,默认是30s。
-retry-max:尝试重复join的次数,默认是0,也就是无限次尝试。
-log-level:consul agent启动后显示的日志信息级别。默认是info,可选:trace、debug、info、warn、err。
-node:节点在集群中的名称,在一个集群中必须是唯一的,默认是该节点的主机名。
-protocol:consul使用的协议版本。
-rejoin:使consul忽略先前的离开,在再次启动后仍旧尝试加入集群中。
-server:定义agent运行在server模式,每个集群至少有一个server,建议每个集群的server不要超过5个。
-syslog:开启系统日志功能,只在linux/osx上生效。
-ui-dir:提供存放web ui资源的路径,该目录必须是可读的。新版支持直接使用-ui。
-pid-file:提供一个路径来存放pid文件,可以使用该文件进行SIGINT/SIGHUP(关闭/更新)agent。
-enable-script-checks标志设置为true,以便启用可执行外部脚本的运行状况检查。 对于生产用途,需要将ACL配置为此以控制注册任意脚本的能力。
更多参数说明可参考:https://www.consul.io/docs/agent/options.html
[root@monitor-
test
-01 ~]
# consul agent -server -bootstrap-expect 3 -client 0.0.0.0 -bind 10.22.0.1000 -data-dir=/home/consul/ -ui
==> WARNING: Expect Mode enabled, expecting 3 servers
==> Starting Consul agent...
==> Consul agent running!
Version:
'v0.9.2'
Node ID:
'8fea634e-c3d6-32ef-f8d4-4ed1a2c0ee42'
Node name:
'monitor-test-01.yfb.sun.cn'
Datacenter:
'dc1'
Server:
true
(bootstrap:
false
)
Client Addr: 0.0.0.0 (HTTP: 8500, HTTPS: -1, DNS: 8600)
Cluster Addr: 10.22.0.1000 (LAN: 8301, WAN: 8302)
Gossip encrypt:
false
, RPC-TLS:
false
, TLS-Incoming:
false
==> Log data will now stream
in
as it occurs:
2018
/08/15
16:40:26 [INFO] raft: Initial configuration (index=1): [{Suffrage:Voter ID:10.22.0.1000:8300 Address:10.22.0.1000:8300} {Suffrage:Voter ID:10.22.0.1001:8300 Address:10.22.0.1001:8300} {Suffrage:Voter ID:10.22.0.1002:8300 Address:10.22.0.1002:8300}]
2018
/08/15
16:40:26 [INFO] serf: EventMemberJoin: monitor-
test
-01.yfb.sun.cn.dc1 10.22.0.1000
2018
/08/15
16:40:26 [INFO] serf: EventMemberJoin: monitor-
test
-01.yfb.sun.cn 10.22.0.1000
2018
/08/15
16:40:26 [INFO] agent: Started DNS server 0.0.0.0:8600 (udp)
2018
/08/15
16:40:26 [INFO] raft: Node at 10.22.0.1000:8300 [Follower] entering Follower state (Leader:
""
)
2018
/08/15
16:40:26 [INFO] serf: Attempting re-
join
to previously known node: monitor-
test
-02.yfb.sun.cn.dc1: 10.22.0.1001:8302
2018
/08/15
16:40:26 [INFO] serf: Attempting re-
join
to previously known node: monitor-
test
-02.yfb.sun.cn: 10.22.0.1001:8301
2018
/08/15
16:40:26 [INFO] consul: Adding LAN server monitor-
test
-01.yfb.sun.cn (Addr: tcp
/10
.22.0.1000:8300) (DC: dc1)
|