openstack高可用集群9-keepalived+haproxy VIP配置：

本文轉載自查看原文 2020-01-27 12:59 1003 openstack

准備工作

關閉控制節點和計算節點的安全組配置，修改/etc/neutron/plugins/ml2/linuxbridge_agent.ini文件中配置為enable_security_group = false

下面以控制節點為例

[root@openstack1 ~]# grep -v '^#\|^$' /etc/neutron/plugins/ml2/linuxbridge_agent.ini

[DEFAULT]

[agent]

[linux_bridge]

physical_interface_mappings = default:eth1,external:eth2

[securitygroup]

firewall_driver = neutron.agent.linux.iptables_firewall.IptablesFirewallDriver

enable_security_group = false

[vxlan]

enable_vxlan = false

[root@openstack1 ~]# systemctl restart neutron-linuxbridge-agent.service

....做實驗的過程中我們發現光重啟服務是不行了，VIP還是不能ping通，最后我們在變更配置后重啟了所有服務器，而且要注意各個雲主機的selinux和iptables是否關閉，最好在各個雲主機上都執行setenforce 0 && iptables -F

nova list

+--------------------------------------+-------------------------+--------+------------+-------------+---------------------------------------------+

+--------------------------------------+-------------------------+--------+------------+-------------+---------------------------------------------+

+--------------------------------------+-------------------------+--------+------------+-------------+---------------------------------------------+

15.3：keepalived+haproxy VIP配置：

新創建兩台虛擬機，本次一台虛擬機IP地址為 172.16.99.114主機名為bj-test-114，另外一台為 172.16.99.115主機名為bj-test-115

15.3.1：安裝haproxy+keepalived：

[root@bj-test-114 ~]# yum install haproxy keepalived haproxy –y #IP地址為 172.16.99.114

[root@bj-test-115 ~]# yum install haproxy keepalived haproxy –y #IP地址為 172.16.99.115

15.3.3：關聯VIP指定實例：

15.3.3.1：將VIP關連至安全組：

[root@node1 ~]# neutron port-create --fixed-ip ip_address=172.16.99.210 --security-group 安全組ID或者名稱網絡ID或名稱

[root@node1 ~]# neutron port-create --fixed-ip ip_address=172.16.99.210 --security-group 5bb5f2b1-9210-470f-a4a7-2715220b2920 5ac5c948-909f-47ff-beba-a2ffaf917c5f

[root@node1 ~]# openstack subnet list

+--------------------------------------+-------------+--------------------------------------+----------------+

+--------------------------------------+-------------+--------------------------------------+----------------+

+--------------------------------------+-------------+--------------------------------------+----------------+

[root@node1 ~]# openstack network list

+--------------------------------------+---------+--------------------------------------+

| ID | Name | Subnets |

+--------------------------------------+---------+--------------------------------------+

| 5ac5c948-909f-47ff-beba-a2ffaf917c5f | vlan99 | bbd536c6-a975-4841-8082-35b28de16ef0 |

| 98f5d807-80e0-48a3-9f40-97eb6ed15f33 | vlan809 | ffc3c430-e551-4c78-be5e-52e6aaf1484d |

+--------------------------------------+---------+--------------------------------------+

[root@node1 ~]# openstack security group list

+--------------------------------------+-----------+------------------------+----------------------------------+

+--------------------------------------+-----------+------------------------+----------------------------------+

+--------------------------------------+-----------+------------------------+----------------------------------+

15.3.3.3：列出各實例的port ID：

[root@node1 ~]# openstack port list | grep 172.16.99.114

[root@node1 ~]# openstack port list | grep 172.16.99.115

[root@node1 ~]# openstack port list

+--------------------------------------+------+-------------------+------------------------------------------------------------------------------+--------+

+--------------------------------------+------+-------------------+------------------------------------------------------------------------------+--------+

+--------------------------------------+------+-------------------+------------------------------------------------------------------------------+--------+

15.3.3.3：將VIP關聯到實例：

[root@node1 ~]# neutron port-update 227caaa0-4421-44f3-8205-a6497404991e --allowed_address_pairs list=true type=dict ip_address=172.16.99.210

[root@node1 ~]# neutron port-update bbe39d0c-a961-4246-9238-a2fe5a6e4113 --allowed_address_pairs list=true type=dict ip_address=172.16.99.210

15.3.4：配置keepalived及組策略：

15.3.4.1：master配置：

[root@bj-test-114 ~]# vim /etc/keepalived/keepalived.conf

! Configuration File for keepalived

global_defs {

notification_email {

acassen@firewall.loc

failover@firewall.loc

sysadmin@firewall.loc

}

notification_email_from Alexandre.Cassen@firewall.loc

smtp_server 192.168.200.1

smtp_connect_timeout 30

router_id LVS_DEVEL

vrrp_skip_check_adv_addr

vrrp_strict

vrrp_garp_interval 0

vrrp_gna_interval 0

}

vrrp_instance VI_1 {

interface eth0

virtual_router_id 1

priority 100

advert_int 1

virtual_ipaddress {

172.16.99.210

}

15.3.4.2：slave配置：

[root@bj-test-115 ~]# cat /etc/keepalived/keepalived.conf

! Configuration File for keepalived

global_defs {

notification_email {

acassen@firewall.loc

failover@firewall.loc

sysadmin@firewall.loc

}

notification_email_from Alexandre.Cassen@firewall.loc

smtp_server 192.168.200.1

smtp_connect_timeout 30

router_id LVS_DEVEL

vrrp_skip_check_adv_addr

vrrp_strict

vrrp_garp_interval 0

vrrp_gna_interval 0

}

vrrp_instance VI_1 {

interface eth0

virtual_router_id 1

priority 50

advert_int 1

virtual_ipaddress {

172.16.99.210

}

15.3.5：配置安全組策略：

#keepalived給予vrrp協議進行報文的傳播，需要在openstack組策略單獨進行放開。

15.3.5.1：入口協議配置：

15.3.5.2：出口配置

15.3.5.3：最終狀態：

15.3.6：啟動keepalived並驗證：

15.3.6.1：各實例啟動keepalived：

[root@bj-test-114 ~]# systemctl start keepalived

[root@bj-test-114 ~]# systemctl enable keepalived

[root@bj-test-115 ~]# systemctl start keepalived

[root@bj-test-115 ~]# systemctl enable keepalived

15.3.6.2：驗證VIP：

#master：

#backup：

15.3.6.3：驗證VIP 通信：

15.3.6.4：驗證VIP 切換：

#master停止VIP，在backup查看日志

15.3.7：haproxy配置：

15.3.7.1：配置haproxy：

[root@bj-test-114 ~]# cat /etc/haproxy/haproxy.cfg

global

maxconn 100000

uid 99

gid 99

daemon

nbproc 1

log 127.0.0.1 local0 info

defaults

option redispatch

option abortonclose

option http-keep-alive

option forwardfor

maxconn 100000

mode http

listen admin_stats

bind 0.0.0.0:1080

mode http

maxconn 10

stats refresh 30s

stats uri /stats

stats auth admin:admin

stats hide-version

#wen-port================================================================

frontend web

bind 172.16.99.210:80

mode http

default_backend web_http_nodes

backend web_http_nodes

mode http

balance roundrobin

server web1 172.16.99.110:8090 check inter 2000 fall 3 rise 5

server web2 172.16.99.111:8090 check inter 2000 fall 3 rise 5

15.3.7.2：配置內核參數：

各虛擬機配置允許監聽非本地IP並開啟轉發

[root@bj-test-114 ~]# vim /etc/sysctl.conf

net.ipv4.ip_nonlocal_bind = 1

net.ipv4.ip_forward = 1

[root@bj-test-115 ~]# sysctl -p

net.ipv4.ip_nonlocal_bind = 1

net.ipv4.ip_forward = 1

兩台虛擬機分別安裝haproxy，結合keepalived實現高可用負載均衡。

15.3.8：配置web服務：

再次新建兩台虛擬機

172.16.99.110 web1

172.16.99.111 web2

15.3.8.1:安裝web服務

[root@bj-test-110 ~]# yum install httpd –y

[root@bj-test-110 ~]# echo "172.16.99.110" > /var/www/html/index.html

[root@bj-test-110 ~]# sed -i 's/Listen 80/Listen 8090/g' /etc/httpd/conf/httpd.conf

[root@bj-test-110 ~]# systemctl start httpd

[root@bj-test-111 ~]# yum install haproxy httpd -y

[root@bj-test-111 ~]# echo " 172.16.99.111" > /var/www/html/index.html

[root@bj-test-111 ~]# sed -i 's/Listen 80/Listen 8090/g' /etc/httpd/conf/httpd.conf

[root@bj-test-111 ~]# systemctl start httpd

15.3.8.2：啟動http並驗證訪問：

[root@bj-test-114 ~]# systemctl start httpd

[root@bj-test-115 ~]# systemctl start httpd

報錯：一切正常，但是ping不通VIP，通過抓包定位到問題應該在虛機中，到虛機中排查發現iptables不知道什么時候多了一條防火牆規則，刪除后恢復正常

擴展：關於底層本質（openstack將VIP關連至安全組和實例）

VIP關連至安全組

neutron port-create --fixed-ip ip_address=172.16.99.210 --security-group 安全組ID或者名稱網絡ID或名稱

neutron port-create --fixed-ip ip_address=172.16.99.210 --security-group 5bb5f2b1-9210-470f-a4a7-2715220b2920 5ac5c948-909f-47ff-beba-a2ffaf917c5f

15.3.3.3：將VIP關聯到實例：

[root@node1 ~]# neutron port-update 227caaa0-4421-44f3-8205-a6497404991e --allowed_address_pairs list=true type=dict ip_address=172.16.99.210

[root@node1 ~]# neutron port-update bbe39d0c-a961-4246-9238-a2fe5a6e4113 --allowed_address_pairs list=true type=dict ip_address=172.16.99.210

底層原理剖析如下：

[root@node1 ~]# nova list

+--------------------------------------+-------------------------+--------+------------+-------------+---------------------------------------------+

+--------------------------------------+-------------------------+--------+------------+-------------+---------------------------------------------+

+--------------------------------------+-------------------------+--------+------------+-------------+---------------------------------------------+

[root@node4 ~]# virsh list --uuid --name

457c1731-52b4-4e2a-838e-1b04b0637589 instance-00000027

5e507e85-5410-4c30-91c2-276e990423c4 instance-00000023

[root@node4 ~]# virsh domiflist 5e507e85-5410-4c30-91c2-276e990423c4

Interface Type Source Model MAC

-------------------------------------------------------

tap227caaa0-44 bridge brq5ac5c948-90 virtio fa:16:3e:70:4f:38

tap77c530af-bd bridge brq98f5d807-80 virtio fa:16:3e:aa:6b:e6

[root@node4 ~]# ebtables-save | grep tap227caaa0-44

:neutronMAC-tap227caaa0-44 DROP

:neutronARP-tap227caaa0-44 DROP

-A PREROUTING -i tap227caaa0-44 -j neutronMAC-tap227caaa0-44

-A PREROUTING -p ARP -i tap227caaa0-44 -j neutronARP-tap227caaa0-44

-A neutronMAC-tap227caaa0-44 -i tap227caaa0-44 --among-src fa:16:3e:70:4f:38, -j RETURN

-A neutronARP-tap227caaa0-44 -p ARP --arp-ip-src 172.16.99.114 -j ACCEPT

-A neutronARP-tap227caaa0-44 -p ARP --arp-ip-src 172.16.99.210 -j ACCEPT

[root@node1 ~]# openstack port list

+--------------------------------------+------+-------------------+------------------------------------------------------------------------------+--------+

+--------------------------------------+------+-------------------+------------------------------------------------------------------------------+--------+

+--------------------------------------+------+-------------------+------------------------------------------------------------------------------+--------+

嗯，我們發現通過平台刪除端口並沒有什么用，不過是openstack port list中少了一個數據，VIP還是通的，於是我們直接刪除ebtables規則，發現ping不通了，於是我們發現了openstack VIP底層的原理其實就是ebtables規則而已，

[root@node4 ~]# ebtables -t nat -D neutronARP-tap227caaa0-44 -p ARP --arp-ip-src 172.16.99.210 -j ACCEPT

[root@node4 ~]# ebtables-save | grep tap227caaa0-44

:neutronMAC-tap227caaa0-44 DROP

:neutronARP-tap227caaa0-44 DROP

-A PREROUTING -i tap227caaa0-44 -j neutronMAC-tap227caaa0-44

-A PREROUTING -p ARP -i tap227caaa0-44 -j neutronARP-tap227caaa0-44

-A neutronMAC-tap227caaa0-44 -i tap227caaa0-44 --among-src fa:16:3e:70:4f:38, -j RETURN

-A neutronARP-tap227caaa0-44 -p ARP --arp-ip-src 172.16.99.114 -j ACCEPT

於是我們嘗試直接插入ebtables規則，發現可以恢復ping通

[root@node4 ~]# ebtables -t nat -A neutronARP-tap227caaa0-44 -p ARP --arp-ip-src 172.16.99.210 -j ACCEPT

[root@node4 ~]# ebtables-save | grep tap227caaa0-44

:neutronMAC-tap227caaa0-44 DROP

:neutronARP-tap227caaa0-44 DROP

-A PREROUTING -i tap227caaa0-44 -j neutronMAC-tap227caaa0-44

-A PREROUTING -p ARP -i tap227caaa0-44 -j neutronARP-tap227caaa0-44

-A neutronMAC-tap227caaa0-44 -i tap227caaa0-44 --among-src fa:16:3e:70:4f:38, -j RETURN

-A neutronARP-tap227caaa0-44 -p ARP --arp-ip-src 172.16.99.114 -j ACCEPT

-A neutronARP-tap227caaa0-44 -p ARP --arp-ip-src 172.16.99.210 -j ACCEPT

關於以前維護Zstack放開VIP訪問的處理辦法

[sys_admin@ph_ph_r730_6 ~]$ sudo ebtables-save | grep vnic10341

-A FORWARD -p IPv4 -i vnic10341.0 --ip-src 0.0.0.0 -j ACCEPT

-A FORWARD -p IPv4 -i vnic10341.0 --ip-src ! 45.113.33.6 -j DROP

[sys_admin@ph_ph_r730_6 ~]$ sudo ebtables -D FORWARD -p IPv4 -i vnic10341.0 --ip-src ! 45.113.33.6 -j DROP

作者：Dexter_Wang 工作崗位：某互聯網公司資深雲計算與存儲工程師聯系郵箱：993852246@qq.com

免責聲明！

本站轉載的文章為個人學習借鑒使用，本站對版權不負任何法律責任。如果侵犯了您的隱私權益，請聯系本站郵箱yoyou2525@163.com刪除。

猜您在找 高可用Mysql架構_Haproxy+keepalived+mycat集群的配置 openstack高可用haproxy配置 Rabbitmq +Haproxy +keepalived 實現高可用集群 Haproxy+keepalived高可用集群實戰三主機配置 keepalived VIP高可用 RabbitMQ高可用配置（Haproxy + Keepalived）高可用OpenStack（Queen版）集群-3.高可用配置(pacemaker&haproxy) 在openstack 中部署keepalived 高可用集群 nginx keepalived vip 高可用 CentOS7 haproxy+keepalived實現高可用集群搭建