openstack高可用集群9-keepalived+haproxy VIP配置：

本文转载自查看原文 2020-01-27 12:59 1003 openstack

准备工作

关闭控制节点和计算节点的安全组配置，修改/etc/neutron/plugins/ml2/linuxbridge_agent.ini文件中配置为enable_security_group = false

下面以控制节点为例

[root@openstack1 ~]# grep -v '^#\|^$' /etc/neutron/plugins/ml2/linuxbridge_agent.ini

[DEFAULT]

[agent]

[linux_bridge]

physical_interface_mappings = default:eth1,external:eth2

[securitygroup]

firewall_driver = neutron.agent.linux.iptables_firewall.IptablesFirewallDriver

enable_security_group = false

[vxlan]

enable_vxlan = false

[root@openstack1 ~]# systemctl restart neutron-linuxbridge-agent.service

....做实验的过程中我们发现光重启服务是不行了，VIP还是不能ping通，最后我们在变更配置后重启了所有服务器，而且要注意各个云主机的selinux和iptables是否关闭，最好在各个云主机上都执行setenforce 0 && iptables -F

nova list

+--------------------------------------+-------------------------+--------+------------+-------------+---------------------------------------------+

+--------------------------------------+-------------------------+--------+------------+-------------+---------------------------------------------+

+--------------------------------------+-------------------------+--------+------------+-------------+---------------------------------------------+

15.3：keepalived+haproxy VIP配置：

新创建两台虚拟机，本次一台虚拟机IP地址为 172.16.99.114主机名为bj-test-114，另外一台为 172.16.99.115主机名为bj-test-115

15.3.1：安装haproxy+keepalived：

[root@bj-test-114 ~]# yum install haproxy keepalived haproxy –y #IP地址为 172.16.99.114

[root@bj-test-115 ~]# yum install haproxy keepalived haproxy –y #IP地址为 172.16.99.115

15.3.3：关联VIP指定实例：

15.3.3.1：将VIP关连至安全组：

[root@node1 ~]# neutron port-create --fixed-ip ip_address=172.16.99.210 --security-group 安全组ID或者名称网络ID或名称

[root@node1 ~]# neutron port-create --fixed-ip ip_address=172.16.99.210 --security-group 5bb5f2b1-9210-470f-a4a7-2715220b2920 5ac5c948-909f-47ff-beba-a2ffaf917c5f

[root@node1 ~]# openstack subnet list

+--------------------------------------+-------------+--------------------------------------+----------------+

+--------------------------------------+-------------+--------------------------------------+----------------+

+--------------------------------------+-------------+--------------------------------------+----------------+

[root@node1 ~]# openstack network list

+--------------------------------------+---------+--------------------------------------+

| ID | Name | Subnets |

+--------------------------------------+---------+--------------------------------------+

| 5ac5c948-909f-47ff-beba-a2ffaf917c5f | vlan99 | bbd536c6-a975-4841-8082-35b28de16ef0 |

| 98f5d807-80e0-48a3-9f40-97eb6ed15f33 | vlan809 | ffc3c430-e551-4c78-be5e-52e6aaf1484d |

+--------------------------------------+---------+--------------------------------------+

[root@node1 ~]# openstack security group list

+--------------------------------------+-----------+------------------------+----------------------------------+

+--------------------------------------+-----------+------------------------+----------------------------------+

+--------------------------------------+-----------+------------------------+----------------------------------+

15.3.3.3：列出各实例的port ID：

[root@node1 ~]# openstack port list | grep 172.16.99.114

[root@node1 ~]# openstack port list | grep 172.16.99.115

[root@node1 ~]# openstack port list

+--------------------------------------+------+-------------------+------------------------------------------------------------------------------+--------+

+--------------------------------------+------+-------------------+------------------------------------------------------------------------------+--------+

+--------------------------------------+------+-------------------+------------------------------------------------------------------------------+--------+

15.3.3.3：将VIP关联到实例：

[root@node1 ~]# neutron port-update 227caaa0-4421-44f3-8205-a6497404991e --allowed_address_pairs list=true type=dict ip_address=172.16.99.210

[root@node1 ~]# neutron port-update bbe39d0c-a961-4246-9238-a2fe5a6e4113 --allowed_address_pairs list=true type=dict ip_address=172.16.99.210

15.3.4：配置keepalived及组策略：

15.3.4.1：master配置：

[root@bj-test-114 ~]# vim /etc/keepalived/keepalived.conf

! Configuration File for keepalived

global_defs {

notification_email {

acassen@firewall.loc

failover@firewall.loc

sysadmin@firewall.loc

}

notification_email_from Alexandre.Cassen@firewall.loc

smtp_server 192.168.200.1

smtp_connect_timeout 30

router_id LVS_DEVEL

vrrp_skip_check_adv_addr

vrrp_strict

vrrp_garp_interval 0

vrrp_gna_interval 0

}

vrrp_instance VI_1 {

interface eth0

virtual_router_id 1

priority 100

advert_int 1

virtual_ipaddress {

172.16.99.210

}

15.3.4.2：slave配置：

[root@bj-test-115 ~]# cat /etc/keepalived/keepalived.conf

! Configuration File for keepalived

global_defs {

notification_email {

acassen@firewall.loc

failover@firewall.loc

sysadmin@firewall.loc

}

notification_email_from Alexandre.Cassen@firewall.loc

smtp_server 192.168.200.1

smtp_connect_timeout 30

router_id LVS_DEVEL

vrrp_skip_check_adv_addr

vrrp_strict

vrrp_garp_interval 0

vrrp_gna_interval 0

}

vrrp_instance VI_1 {

interface eth0

virtual_router_id 1

priority 50

advert_int 1

virtual_ipaddress {

172.16.99.210

}

15.3.5：配置安全组策略：

#keepalived给予vrrp协议进行报文的传播，需要在openstack组策略单独进行放开。

15.3.5.1：入口协议配置：

15.3.5.2：出口配置

15.3.5.3：最终状态：

15.3.6：启动keepalived并验证：

15.3.6.1：各实例启动keepalived：

[root@bj-test-114 ~]# systemctl start keepalived

[root@bj-test-114 ~]# systemctl enable keepalived

[root@bj-test-115 ~]# systemctl start keepalived

[root@bj-test-115 ~]# systemctl enable keepalived

15.3.6.2：验证VIP：

#master：

#backup：

15.3.6.3：验证VIP 通信：

15.3.6.4：验证VIP 切换：

#master停止VIP，在backup查看日志

15.3.7：haproxy配置：

15.3.7.1：配置haproxy：

[root@bj-test-114 ~]# cat /etc/haproxy/haproxy.cfg

global

maxconn 100000

uid 99

gid 99

daemon

nbproc 1

log 127.0.0.1 local0 info

defaults

option redispatch

option abortonclose

option http-keep-alive

option forwardfor

maxconn 100000

mode http

listen admin_stats

bind 0.0.0.0:1080

mode http

maxconn 10

stats refresh 30s

stats uri /stats

stats auth admin:admin

stats hide-version

#wen-port================================================================

frontend web

bind 172.16.99.210:80

mode http

default_backend web_http_nodes

backend web_http_nodes

mode http

balance roundrobin

server web1 172.16.99.110:8090 check inter 2000 fall 3 rise 5

server web2 172.16.99.111:8090 check inter 2000 fall 3 rise 5

15.3.7.2：配置内核参数：

各虚拟机配置允许监听非本地IP并开启转发

[root@bj-test-114 ~]# vim /etc/sysctl.conf

net.ipv4.ip_nonlocal_bind = 1

net.ipv4.ip_forward = 1

[root@bj-test-115 ~]# sysctl -p

net.ipv4.ip_nonlocal_bind = 1

net.ipv4.ip_forward = 1

两台虚拟机分别安装haproxy，结合keepalived实现高可用负载均衡。

15.3.8：配置web服务：

再次新建两台虚拟机

172.16.99.110 web1

172.16.99.111 web2

15.3.8.1:安装web服务

[root@bj-test-110 ~]# yum install httpd –y

[root@bj-test-110 ~]# echo "172.16.99.110" > /var/www/html/index.html

[root@bj-test-110 ~]# sed -i 's/Listen 80/Listen 8090/g' /etc/httpd/conf/httpd.conf

[root@bj-test-110 ~]# systemctl start httpd

[root@bj-test-111 ~]# yum install haproxy httpd -y

[root@bj-test-111 ~]# echo " 172.16.99.111" > /var/www/html/index.html

[root@bj-test-111 ~]# sed -i 's/Listen 80/Listen 8090/g' /etc/httpd/conf/httpd.conf

[root@bj-test-111 ~]# systemctl start httpd

15.3.8.2：启动http并验证访问：

[root@bj-test-114 ~]# systemctl start httpd

[root@bj-test-115 ~]# systemctl start httpd

报错：一切正常，但是ping不通VIP，通过抓包定位到问题应该在虚机中，到虚机中排查发现iptables不知道什么时候多了一条防火墙规则，删除后恢复正常

扩展：关于底层本质（openstack将VIP关连至安全组和实例）

VIP关连至安全组

neutron port-create --fixed-ip ip_address=172.16.99.210 --security-group 安全组ID或者名称网络ID或名称

neutron port-create --fixed-ip ip_address=172.16.99.210 --security-group 5bb5f2b1-9210-470f-a4a7-2715220b2920 5ac5c948-909f-47ff-beba-a2ffaf917c5f

15.3.3.3：将VIP关联到实例：

[root@node1 ~]# neutron port-update 227caaa0-4421-44f3-8205-a6497404991e --allowed_address_pairs list=true type=dict ip_address=172.16.99.210

[root@node1 ~]# neutron port-update bbe39d0c-a961-4246-9238-a2fe5a6e4113 --allowed_address_pairs list=true type=dict ip_address=172.16.99.210

底层原理剖析如下：

[root@node1 ~]# nova list

+--------------------------------------+-------------------------+--------+------------+-------------+---------------------------------------------+

+--------------------------------------+-------------------------+--------+------------+-------------+---------------------------------------------+

+--------------------------------------+-------------------------+--------+------------+-------------+---------------------------------------------+

[root@node4 ~]# virsh list --uuid --name

457c1731-52b4-4e2a-838e-1b04b0637589 instance-00000027

5e507e85-5410-4c30-91c2-276e990423c4 instance-00000023

[root@node4 ~]# virsh domiflist 5e507e85-5410-4c30-91c2-276e990423c4

Interface Type Source Model MAC

-------------------------------------------------------

tap227caaa0-44 bridge brq5ac5c948-90 virtio fa:16:3e:70:4f:38

tap77c530af-bd bridge brq98f5d807-80 virtio fa:16:3e:aa:6b:e6

[root@node4 ~]# ebtables-save | grep tap227caaa0-44

:neutronMAC-tap227caaa0-44 DROP

:neutronARP-tap227caaa0-44 DROP

-A PREROUTING -i tap227caaa0-44 -j neutronMAC-tap227caaa0-44

-A PREROUTING -p ARP -i tap227caaa0-44 -j neutronARP-tap227caaa0-44

-A neutronMAC-tap227caaa0-44 -i tap227caaa0-44 --among-src fa:16:3e:70:4f:38, -j RETURN

-A neutronARP-tap227caaa0-44 -p ARP --arp-ip-src 172.16.99.114 -j ACCEPT

-A neutronARP-tap227caaa0-44 -p ARP --arp-ip-src 172.16.99.210 -j ACCEPT

[root@node1 ~]# openstack port list

+--------------------------------------+------+-------------------+------------------------------------------------------------------------------+--------+

+--------------------------------------+------+-------------------+------------------------------------------------------------------------------+--------+

+--------------------------------------+------+-------------------+------------------------------------------------------------------------------+--------+

嗯，我们发现通过平台删除端口并没有什么用，不过是openstack port list中少了一个数据，VIP还是通的，于是我们直接删除ebtables规则，发现ping不通了，于是我们发现了openstack VIP底层的原理其实就是ebtables规则而已，

[root@node4 ~]# ebtables -t nat -D neutronARP-tap227caaa0-44 -p ARP --arp-ip-src 172.16.99.210 -j ACCEPT

[root@node4 ~]# ebtables-save | grep tap227caaa0-44

:neutronMAC-tap227caaa0-44 DROP

:neutronARP-tap227caaa0-44 DROP

-A PREROUTING -i tap227caaa0-44 -j neutronMAC-tap227caaa0-44

-A PREROUTING -p ARP -i tap227caaa0-44 -j neutronARP-tap227caaa0-44

-A neutronMAC-tap227caaa0-44 -i tap227caaa0-44 --among-src fa:16:3e:70:4f:38, -j RETURN

-A neutronARP-tap227caaa0-44 -p ARP --arp-ip-src 172.16.99.114 -j ACCEPT

于是我们尝试直接插入ebtables规则，发现可以恢复ping通

[root@node4 ~]# ebtables -t nat -A neutronARP-tap227caaa0-44 -p ARP --arp-ip-src 172.16.99.210 -j ACCEPT

[root@node4 ~]# ebtables-save | grep tap227caaa0-44

:neutronMAC-tap227caaa0-44 DROP

:neutronARP-tap227caaa0-44 DROP

-A PREROUTING -i tap227caaa0-44 -j neutronMAC-tap227caaa0-44

-A PREROUTING -p ARP -i tap227caaa0-44 -j neutronARP-tap227caaa0-44

-A neutronMAC-tap227caaa0-44 -i tap227caaa0-44 --among-src fa:16:3e:70:4f:38, -j RETURN

-A neutronARP-tap227caaa0-44 -p ARP --arp-ip-src 172.16.99.114 -j ACCEPT

-A neutronARP-tap227caaa0-44 -p ARP --arp-ip-src 172.16.99.210 -j ACCEPT

关于以前维护Zstack放开VIP访问的处理办法

[sys_admin@ph_ph_r730_6 ~]$ sudo ebtables-save | grep vnic10341

-A FORWARD -p IPv4 -i vnic10341.0 --ip-src 0.0.0.0 -j ACCEPT

-A FORWARD -p IPv4 -i vnic10341.0 --ip-src ! 45.113.33.6 -j DROP

[sys_admin@ph_ph_r730_6 ~]$ sudo ebtables -D FORWARD -p IPv4 -i vnic10341.0 --ip-src ! 45.113.33.6 -j DROP

作者：Dexter_Wang 工作岗位：某互联网公司资深云计算与存储工程师联系邮箱：993852246@qq.com

免责声明！

本站转载的文章为个人学习借鉴使用，本站对版权不负任何法律责任。如果侵犯了您的隐私权益，请联系本站邮箱yoyou2525@163.com删除。

猜您在找 高可用Mysql架构_Haproxy+keepalived+mycat集群的配置 openstack高可用haproxy配置 Rabbitmq +Haproxy +keepalived 实现高可用集群 Haproxy+keepalived高可用集群实战三主机配置 keepalived VIP高可用 RabbitMQ高可用配置（Haproxy + Keepalived）高可用OpenStack（Queen版）集群-3.高可用配置(pacemaker&haproxy) 在openstack 中部署keepalived 高可用集群 nginx keepalived vip 高可用 CentOS7 haproxy+keepalived实现高可用集群搭建