【阿圆实验】Consul HA 高可用方案



 一、建立Consul Cluster环境

利用Consul提供的服务实现服务的注册与发现,需要建立Consul Cluster。在Consul方案中,每个提供服务的节点上都要部署和运行Consul的agent,所有运行Consul agent节点的集合构成Consul Cluster。Consul agent有两种运行模式:Server和Client。这里的Server和Client只是Consul集群层面的区分,与搭建在Cluster之上 的应用服务无关。以Server模式运行的Consul agent节点用于维护Consul集群的状态,官方建议每个Consul Cluster至少有3个或以上的运行在Server mode的Agent,Client节点不限。

Consul架构说明

注:准备至少3台机器部署consul。

二.consul集群配置

两种配置方式:《1》通过进程管理器(supervisor)配置。前提是consul已经被supervisor管理,因此推荐新手使用命令行方式配置consul集群。

                         《2》通过consul命令行配置(此方式配置的consul集群将不被supervisor管理,在supervisor进程管理器上的状态会显示为FATAL)。

注:若是docker启动的,建议用命令行方式配置consul集群。

《1》通过进程管理器配置

1.修改启动脚本命令(以3台机器为例)

关闭防火墙

systemctl stop firewalld

vi /etc/supervisord.d/supervisor.ini

选取一台机器配置
[program:consul]
priority = 1
user = yy
command  =
     /usr/bin/consul
         agent
         -server
         -data- dir = "/data/yy-monitor-server/data/consul-data"
         -bootstrap-expect=3        #集群中最少consul实例数为3
         -client= "0.0.0.0"
         -bind= "本机ip"             # 如 -bind="10.22.0.1002" 
         -ui
stdout_logfile =  /data/yy-monitor-server/log/consul .log
stderr_logfile =  /data/yy-monitor-server/log/consul .log
autorestart =  true
其他机器配置
[program:consul]
priority = 1
user = yy
command  =
     /usr/bin/consul
         agent
         -server
         -data- dir = "/data/yy-monitor-server/data/consul-data"
         -client= "0.0.0.0"
         -bind= "本机ip"              # 如-bind="10.22.0.1001"
         join =“其他consul ip”      # 如join=“10.22.0.1002”
         -ui
stdout_logfile =  /data/yy-monitor-server/log/consul .log
stderr_logfile =  /data/yy-monitor-server/log/consul .log
autorestart =  true

2.重启配置

 #systemctl restart supervisord

# supervisorctl restart consul

3.查看日志

tail -300f consul.log

其中一台主机上的consul日志
==> Starting Consul agent...
==> Consul agent running!
            Version:  'v0.9.2'
            Node ID:  '035f828b-8db9-8cb1-a3a8-8eabf5e821c4'
          Node name:  'monitor-test-02.yfb.sun.cn'
         Datacenter:  'dc1'
             Server:  true  (bootstrap:  false )
        Client Addr: 0.0.0.0 (HTTP: 8500, HTTPS: -1, DNS: 8600)
       Cluster Addr: 10.22.0.1001 (LAN: 8301, WAN: 8302)
     Gossip encrypt:  false , RPC-TLS:  false , TLS-Incoming:  false
 
==> Log data will now stream  in  as it occurs:
 
     2018 /08/27  13:43:20 [INFO] raft: Restored from snapshot 1370-114688-1535337744933
     2018 /08/27  13:43:20 [INFO] raft: Initial configuration (index=117709): [{Suffrage:Voter ID:10.22.0.1001:8300 Address:10.22.0.1001:8300} {Suffrage:Voter ID:10.22.0.1002:8300 Address:10.22.0.1002:8300} {Suffrage:Voter ID:10.22.0.1000:8300 Address:10.22.0.1000:8300}]
     2018 /08/27  13:43:20 [INFO] serf: EventMemberJoin: monitor- test -02.yfb.sun.cn.dc1 10.22.0.1001
     2018 /08/27  13:43:20 [INFO] raft: Node at 10.22.0.1001:8300 [Follower] entering Follower state (Leader:  "" )
     2018 /08/27  13:43:20 [INFO] serf: Attempting re- join  to previously known node: n2.dc1: 10.22.0.1001:8302
     2018 /08/27  13:43:20 [INFO] serf: EventMemberJoin: monitor- test -02.yfb.sun.cn 10.22.0.1001
     2018 /08/27  13:43:20 [INFO] agent: Started DNS server 0.0.0.0:8600 (udp)
     2018 /08/27  13:43:20 [INFO] serf: Attempting re- join  to previously known node: monitor- test -01.yfb.sun.cn: 10.22.0.1000:8301
     2018 /08/27  13:43:20 [INFO] consul: Adding LAN server monitor- test -02.yfb.sun.cn (Addr: tcp /10 .22.0.1001:8300) (DC: dc1)
     2018 /08/27  13:43:20 [INFO] consul: Handled member- join  event  for  server  "monitor-test-02.yfb.sun.cn.dc1"  in  area  "wan"
     2018 /08/27  13:43:20 [INFO] agent: Started DNS server 0.0.0.0:8600 (tcp)
     2018 /08/27  13:43:20 [INFO] agent: Started HTTP server on [::]:8500
     2018 /08/27  13:43:20 [INFO] serf: Re-joined to previously known node: n2.dc1: 10.22.0.1001:8302
     2018 /08/27  13:43:20 [WARN] memberlist: Refuting a suspect message (from: monitor- test -02.yfb.sun.cn)
     2018 /08/27  13:43:20 [INFO] serf: EventMemberJoin: monitor- test -03.yfb.sun.cn 10.22.0.1002
     2018 /08/27  13:43:20 [INFO] serf: EventMemberJoin: monitor- test -01.yfb.sun.cn 10.22.0.1000
     2018 /08/27  13:43:20 [INFO] serf: Re-joined to previously known node: monitor- test -01.yfb.sun.cn: 10.22.0.1000:8301
     2018 /08/27  13:43:20 [INFO] consul: Adding LAN server monitor- test -03.yfb.sun.cn (Addr: tcp /10 .22.0.1002:8300) (DC: dc1)
     2018 /08/27  13:43:20 [INFO] consul: Adding LAN server monitor- test -01.yfb.sun.cn (Addr: tcp /10 .22.0.1000:8300) (DC: dc1)
     2018 /08/27  13:43:20 [INFO] serf: EventMemberJoin: monitor- test -01.yfb.sun.cn.dc1 10.22.0.1000
     2018 /08/27  13:43:20 [WARN] memberlist: Refuting an alive message
     2018 /08/27  13:43:20 [INFO] serf: EventMemberJoin: monitor- test -03.yfb.sun.cn.dc1 10.22.0.1002
     2018 /08/27  13:43:20 [INFO] consul: Handled member- join  event  for  server  "monitor-test-01.yfb.sun.cn.dc1"  in  area  "wan"
     2018 /08/27  13:43:20 [INFO] consul: Handled member- join  event  for  server  "monitor-test-03.yfb.sun.cn.dc1"  in  area  "wan"
     2018 /08/27  13:43:20 [INFO] agent: Synced node info
     2018 /08/27  13:43:54 [INFO] memberlist: Suspect monitor- test -01.yfb.sun.cn has failed, no acks received
     2018 /08/27  13:43:54 [INFO] serf: EventMemberUpdate: monitor- test -01.yfb.sun.cn
     2018 /08/27  13:43:55 [INFO] serf: EventMemberUpdate: monitor- test -01.yfb.sun.cn.dc1
     2018 /08/27  13:43:56 [INFO] serf: EventMemberUpdate: monitor- test -01.yfb.sun.cn.dc1
     2018 /08/27  13:43:57 [INFO] serf: EventMemberUpdate: monitor- test -01.yfb.sun.cn
==> Failed to check  for  updates: Get https: //checkpoint-api .hashicorp.com /v1/check/consul ?arch=amd64&os=linux&signature=143c3fc1-0a07-2f4e-79e3-528773c5260f&version=0.9.2: dial tcp 54.221.203.30:443: i /o  timeout
     2018 /08/27  13:44:01 [WARN] raft: Rejecting vote request from 10.22.0.1002:8300 since we have a leader: 10.22.0.1000:8300
     2018 /08/27  13:44:01 [INFO] consul: New leader elected: monitor- test -03.yfb.sun.cn

注:更新异常可忽略,内网云桌面不能连接github。

4 查看当前节点信息

使用命令  consul info

每次启动之后,都会有一台部署有consul的机器被选举为leader节点:

其他consul节点
state = Follower

 

5.ui页面

访问任意节点localhost:8500  ,如下图

 

《2》通过consul命令行配置

1.部署前

关闭防火墙

#因consul集群间需通信,关闭部署monitor的虚拟机的防火墙

systemctl stop firewalld


分别进入到monitor的容器中

#monitor-server为容器名,按实际情况修改
docker  exec  - it monitor - server  / bin / bash 
#停止consul
supervisorctl stop consul
 
 
[root@monitor - test - 01  ~] # supervisorctl
alertmanager                     RUNNING   pid  22016 , uptime  21 : 28 : 23
consul                           RUNNING   pid  11109 , uptime  0 : 00 : 07
dashboard - upgrade                EXITED    Aug  14  05 : 51  PM
yy - echo - webhook                RUNNING   pid  22025 , uptime  21 : 28 : 23
grafana                          RUNNING   pid  22015 , uptime  21 : 28 : 23
nginx                            RUNNING   pid  22017 , uptime  21 : 28 : 23
node_exporter                    RUNNING   pid  22020 , uptime  21 : 28 : 23
prometheus                       RUNNING   pid  22018 , uptime  21 : 28 : 23
supervisor> stop consul
consul: stopped

 

server:
1.1 部署主节点

运行命令:consul agent -server -bootstrap-expect 以sever方式运行的主机个数 -client 0.0.0.0 -bind 主机地址 -data-dir=/home/consul/ -ui &
[root@monitor- test -01 ~] # consul agent -server -bootstrap-expect 3 -client 0.0.0.0 -bind 10.22.0.1000 -data-dir=/home/consul/ -ui &
==> WARNING: Expect Mode enabled, expecting 3 servers
==> Starting Consul agent...
==> Consul agent running!
            Version:  'v0.9.2'
            Node ID:  '8fea634e-c3d6-32ef-f8d4-4ed1a2c0ee42'
          Node name:  'monitor-test-01.yfb.sun.cn'
         Datacenter:  'dc1'
             Server:  true  (bootstrap:  false )
        Client Addr: 0.0.0.0 (HTTP: 8500, HTTPS: -1, DNS: 8600)
       Cluster Addr: 10.22.0.1000 (LAN: 8301, WAN: 8302)
     Gossip encrypt:  false , RPC-TLS:  false , TLS-Incoming:  false
 
==> Log data will now stream  in  as it occurs:
 
     2018 /08/15  16:40:26 [INFO] raft: Initial configuration (index=1): [{Suffrage:Voter ID:10.22.0.1000:8300 Address:10.22.0.1000:8300} {Suffrage:Voter ID:10.22.0.1001:8300 Address:10.22.0.1001:8300} {Suffrage:Voter ID:10.22.0.1002:8300 Address:10.22.0.1002:8300}]
     2018 /08/15  16:40:26 [INFO] serf: EventMemberJoin: monitor- test -01.yfb.sun.cn.dc1 10.22.0.1000
     2018 /08/15  16:40:26 [INFO] serf: EventMemberJoin: monitor- test -01.yfb.sun.cn 10.22.0.1000
     2018 /08/15  16:40:26 [INFO] agent: Started DNS server 0.0.0.0:8600 (udp)
     2018 /08/15  16:40:26 [INFO] raft: Node at 10.22.0.1000:8300 [Follower] entering Follower state (Leader:  "" )
     2018 /08/15  16:40:26 [INFO] serf: Attempting re- join  to previously known node: monitor- test -02.yfb.sun.cn.dc1: 10.22.0.1001:8302
     2018 /08/15  16:40:26 [INFO] serf: Attempting re- join  to previously known node: monitor- test -02.yfb.sun.cn: 10.22.0.1001:8301
     2018 /08/15  16:40:26 [INFO] consul: Adding LAN server monitor- test -01.yfb.sun.cn (Addr: tcp /10 .22.0.1000:8300) (DC: dc1)

1.2 其他机器运行命令:

consul agent -server -retry-join 主节点ip地址 -bind 本机ip地址 -client 0.0.0.0 -data-dir=/home/consul/ -ui &
[root@monitor- test -02 ~] # consul agent -server -retry-join 10.22.0.1000 -bind 10.22.0.1001 -client 0.0.0.0 -data-dir=/home/consul/ -ui &
==> Starting Consul agent...
==> Consul agent running!
            Version:  'v0.9.2'
            Node ID:  '9e35e3c2-9226-de85-2202-2d70e516f7b7'
          Node name:  'monitor-test-02.yfb.sun.cn'
         Datacenter:  'dc1'
             Server:  true  (bootstrap:  false )
        Client Addr: 127.0.0.1 (HTTP: 8500, HTTPS: -1, DNS: 8600)
       Cluster Addr: 10.22.0.1001 (LAN: 8301, WAN: 8302)
     Gossip encrypt:  false , RPC-TLS:  false , TLS-Incoming:  false
 
==> Log data will now stream  in  as it occurs:
 
     2018 /08/15  16:32:51 [INFO] raft: Initial configuration (index=0): []
     2018 /08/15  16:32:51 [INFO] serf: EventMemberJoin: monitor- test -02.yfb.sun.cn.dc1 10.22.0.1001
     2018 /08/15  16:32:51 [INFO] serf: EventMemberJoin: monitor- test -02.yfb.sun.cn 10.22.0.1001
     2018 /08/15  16:32:51 [INFO] agent: Started DNS server 127.0.0.1:8600 (udp)
     2018 /08/15  16:32:51 [INFO] raft: Node at 10.22.0.1001:8300 [Follower] entering Follower state (Leader:  "" )
     2018 /08/15  16:32:51 [WARN] serf: Failed to re- join  any previously known node
     2018 /08/15  16:32:51 [WARN] serf: Failed to re- join  any previously known node
     2018 /08/15  16:32:51 [INFO] consul: Adding LAN server monitor- test -02.yfb.sun.cn (Addr: tcp /10 .22.0.1001:8300) (DC: dc1)
     2018 /08/15  16:32:51 [INFO] consul: Handled member- join  event  for  server  "monitor-test-02.yfb.sun.cn.dc1"  in  area  "wan"
     2018 /08/15  16:32:51 [INFO] agent: Started DNS server 127.0.0.1:8600 (tcp)
     2018 /08/15  16:32:51 [INFO] agent: Started HTTP server on 127.0.0.1:8500
     2018 /08/15  16:32:51 [INFO] agent: Retry  join  is supported  for : aws azure gce softlayer
     2018 /08/15  16:32:51 [INFO] agent: Joining cluster...
     2018 /08/15  16:32:51 [INFO] agent: (LAN) joining: [10.22.0.1000]
     2018 /08/15  16:32:51 [INFO] serf: EventMemberJoin: monitor- test -01.yfb. su

1.6 验证集群可用性

      1、 consul leave 命令可优雅退出consul集群。

 

      2、consul members 可查看集群状况。
 

      3、可分别访问consul ui界面进行验证。页面访问主机地址即可。

 

     4、如果集群加入不成功,可以用 consul join 命令在非主节点机器上触发Cluster bootstrap过程:consul join 主节点ip地址。

 

三、代理配置

1.各主机做consul代理配置

# cat nginx.conf
     upstream consul.cn{
          server 其他机器ip:8500 backup;
          server 主机ip:8500;
          server 其他机器ip:8500 backup;
     }
 
     server {
         # Consul UI
         location  /consul/  {
             proxy_pass      http: //consul .cn /ui/ ;
         }
         # Consul API
         location  /v1/  {
             proxy_pass      http: //consul .cn /v1/ ;
             add_header      X-Remote-IP $remote_addr;
             add_header      X-Server-Time $date_gmt;
         }
     }

2.nginx重启

supervisorctl restart nginx

3.验证配置

 3.1 可以通过关闭其中一台机器的consul服务,查看日志或ui页面。

supervisorctl stop consul

 集群中任一consul宕机不会影响consul的使用,数据不丢失。

 注:如果其中两台的consul宕机,在集群中需要保证至少两台机器在线,否则无法自选出leader,因此consul不可用。

附录FAQ:   

  • 命令行参数

  1. -advertise:通知展现地址用来改变我们给集群中的其他节点展现的地址,一般情况下-bind地址就是展现地址

  2. -bootstrap:用来控制一个server是否在bootstrap模式,在一个datacenter中只能有一个server处于bootstrap模式,当一个server处于bootstrap模式时,可以自己选举为raft leader。

  3. -bootstrap-expect:在一个datacenter中期望提供的server节点数目,当该值提供的时候,consul一直等到达到指定sever数目的时候才会引导整个集群,该标记不能和bootstrap公用。

  4. -bind:该地址用来在集群内部的通讯,集群内的所有节点到地址都必须是可达的,默认是0.0.0.0。

  5. -client:consul绑定在哪个client地址上,这个地址提供HTTP、DNS、RPC等服务,默认是127.0.0.1。

  6. -config-file:明确的指定要加载哪个配置文件

  7. -config-dir:配置文件目录,里面所有以.json结尾的文件都会被加载

  8. -data-dir:提供一个目录用来存放agent的状态,所有的agent都需要该目录,该目录必须是稳定的,系统重启后都继续存在。

  9. -dc:该标记控制agent的datacenter的名称,默认是dc1。

  10. -encrypt:指定secret key,使consul在通讯时进行加密,key可以通过consul keygen生成,同一个集群中的节点必须使用相同的key。

  11. -join:加入一个已经启动的agent的ip地址,可以多次指定多个agent的地址。如果consul不能加入任何指定的地址中,则agent会启动失败。默认agent启动时不会加入任何节点。

  12. -retry-join:和join类似,但是允许你在第一次失败后进行尝试。

  13. -retry-interval:两次join之间的时间间隔,默认是30s。

  14. -retry-max:尝试重复join的次数,默认是0,也就是无限次尝试。

  15. -log-level:consul agent启动后显示的日志信息级别。默认是info,可选:trace、debug、info、warn、err。

  16. -node:节点在集群中的名称,在一个集群中必须是唯一的,默认是该节点的主机名。

  17. -protocol:consul使用的协议版本。

  18. -rejoin:使consul忽略先前的离开,在再次启动后仍旧尝试加入集群中。

  19. -server:定义agent运行在server模式,每个集群至少有一个server,建议每个集群的server不要超过5个。

  20. -syslog:开启系统日志功能,只在linux/osx上生效。

  21. -ui-dir:提供存放web ui资源的路径,该目录必须是可读的。新版支持直接使用-ui。

  22. -pid-file:提供一个路径来存放pid文件,可以使用该文件进行SIGINT/SIGHUP(关闭/更新)agent。

  23. -enable-script-checks标志设置为true,以便启用可执行外部脚本的运行状况检查。 对于生产用途,需要将ACL配置为此以控制注册任意脚本的能力。

更多参数说明可参考:https://www.consul.io/docs/agent/options.html

[root@monitor- test -01 ~] # consul agent -server -bootstrap-expect 3 -client 0.0.0.0 -bind 10.22.0.1000 -data-dir=/home/consul/ -ui
==> WARNING: Expect Mode enabled, expecting 3 servers
==> Starting Consul agent...
==> Consul agent running!
            Version:  'v0.9.2'
            Node ID:  '8fea634e-c3d6-32ef-f8d4-4ed1a2c0ee42'
          Node name:  'monitor-test-01.yfb.sun.cn'
         Datacenter:  'dc1'
             Server:  true  (bootstrap:  false )
        Client Addr: 0.0.0.0 (HTTP: 8500, HTTPS: -1, DNS: 8600)
       Cluster Addr: 10.22.0.1000 (LAN: 8301, WAN: 8302)
     Gossip encrypt:  false , RPC-TLS:  false , TLS-Incoming:  false
 
==> Log data will now stream  in  as it occurs:
 
     2018 /08/15  16:40:26 [INFO] raft: Initial configuration (index=1): [{Suffrage:Voter ID:10.22.0.1000:8300 Address:10.22.0.1000:8300} {Suffrage:Voter ID:10.22.0.1001:8300 Address:10.22.0.1001:8300} {Suffrage:Voter ID:10.22.0.1002:8300 Address:10.22.0.1002:8300}]
     2018 /08/15  16:40:26 [INFO] serf: EventMemberJoin: monitor- test -01.yfb.sun.cn.dc1 10.22.0.1000
     2018 /08/15  16:40:26 [INFO] serf: EventMemberJoin: monitor- test -01.yfb.sun.cn 10.22.0.1000
     2018 /08/15  16:40:26 [INFO] agent: Started DNS server 0.0.0.0:8600 (udp)
     2018 /08/15  16:40:26 [INFO] raft: Node at 10.22.0.1000:8300 [Follower] entering Follower state (Leader:  "" )
     2018 /08/15  16:40:26 [INFO] serf: Attempting re- join  to previously known node: monitor- test -02.yfb.sun.cn.dc1: 10.22.0.1001:8302
     2018 /08/15  16:40:26 [INFO] serf: Attempting re- join  to previously known node: monitor- test -02.yfb.sun.cn: 10.22.0.1001:8301
     2018 /08/15  16:40:26 [INFO] consul: Adding LAN server monitor- test -01.yfb.sun.cn (Addr: tcp /10 .22.0.1000:8300) (DC: dc1)


免责声明!

本站转载的文章为个人学习借鉴使用,本站对版权不负任何法律责任。如果侵犯了您的隐私权益,请联系本站邮箱yoyou2525@163.com删除。



 
粤ICP备18138465号  © 2018-2025 CODEPRJ.COM