keepalievd 高可用以及解决nginx宕机产生脑裂问题

Keepalived高可用

什么是高可用？

一般是指2台机器启动着完全相同的业务系统，当有一台系统宕机，另外一台服务器就能快速的接管，对于访问的用户是无感知的。

举例

通常做法是给路由器增加一台备节点，那么问题来了，如果我们的主网关master故障了，用户需要手动指向backup，如果用户过多修改起来会非常麻烦。

问题一：假设用户将指向都修改为backup路由器，那么master路由器修好了怎么办？
问题二：假设Master网关故障，我们将backup网关配置为master网关的ip是否可以？
 
其实是不行的，因为PC第一次通过ARP广播寻找到Master网关的MAC地址与IP地址后，会将信息写到ARP的缓存表中，那么PC之后连接都是通过那个缓存表的信息去连接，然后进行数据包的转发，即使我们修改了IP但是Mac地址是唯一的，pc的数据包依然会发送给master。（除非是PC的ARP缓存表过期，再次发起ARP广播的时候才能获取新的backup对应的Mac地址与IP地址）
 
如何才能做到出现故障自动转移，此时VRRP就出现了，我们的VRRP其实是通过软件或者硬件的形式在Master和Backup外面增加一个虚拟的MAC地址（VMAC）与虚拟IP地址（VIP），那么在这种情况下，PC请求VIP的时候，无论是Master处理还是Backup处理，PC仅会在ARP缓存表中记录VMAC与VIP的信息。

常用工具

硬件通常使用：F5
软件通常使用：keepalived

VRRP协议

VRRP协议会在一个局域网中进行广播
# 若lb01lb02均有vrrp协议，则相互广播各自状态

高可用keepalived核心概念

• 如何确定主节点(master)和备用节点(backup)
• 如果master故障，backup自动接管，那么master恢复后是否夺权(抢占式，非抢占式)
• 如果两台服务器都认为自己是master会如何？(脑裂)

---

keepalived搭建

环境准备

主机	IP	身份
lb01	172.16.1.5	Keepalived master
lb02	172.16.1.6	Keepalived backup
web01	172.16.1.7	web端
web02	172.16.1.8	web端
db01	172.16.1.61	数据库
网卡	10.0.0.8	VIP

1.lb01和lb02

[root@lb01 ~]# yum install keepalived -y
[root@lb02 ~]# yum install keepalived -y

2.编辑配置文件

[root@lb01 ~]# vim /etc/keepalived/keepalived.conf 
! Configuration File for keepalived

# 全局配置
global_defs {
   # 当前keepalived的唯一标识
   router_id lb01
}

# 配置VRRP协议
vrrp_instance VI_1 {
    # 状态，MASTER和BACKUP
    state MASTER
    # 绑定网卡
    interface eth0
    # 虚拟路由标示，可以理解为分组
    virtual_router_id 50
    # 优先级
    priority 100
    # 监测心跳间隔时间
    advert_int 1
    # 配置认证
    authentication {
        # 认证类型
        auth_type PASS
        # 认证的密码
        auth_pass 1111
    }
    # 设置VIP
    virtual_ipaddress {
        # 虚拟的VIP地址
        192.168.15.3
    }
}
[root@lb01 ~]# systemctl enable --now keepalived

[root@lb02 ~]# cat /etc/keepalived/keepalived.conf
! Configuration File for keepalived

# 全局配置
global_defs {
   # 当前keepalived的唯一标识
   router_id lb02
}

# 配置VRRP协议
vrrp_instance VI_1 {
    # 状态，MASTER和BACKUP
    state BACKUP
    # 绑定网卡
    interface eth0
    # 虚拟路由标示，可以理解为分组
    virtual_router_id 50
    # 优先级
    priority 90
    # 监测心跳间隔时间
    advert_int 1
    # 配置认证
    authentication {
        # 认证类型
        auth_type PASS
        # 认证的密码
        auth_pass 1111
    }
    # 设置VIP
    virtual_ipaddress {
        # 虚拟的VIP地址
        192.168.15.3
    }
}
 [root@lb02 ~]# systemctl enable --now keepalived

2.lb01和lb02的nginx保持一致

[root@lb01 ~]# ls
anaconda-ks.cfg  init.sh  nginx-1.20.2  nginx-1.20.2.tar.gz
[root@lb01 ~]# scp nginx-1.20.2.tar.gz 172.16.1.6:/root
root@172.16.1.6's password: 
nginx-1.20.2.tar.gz                                           100% 1037KB  20.5MB/s   00:00   

3.编译安装nginx，lb02

[root@lb02 ~]# ls
anaconda-ks.cfg  init.sh  nginx-1.20.2.tar.gz
[root@lb02 ~]# tar -xf nginx-1.20.2.tar.gz 
[root@lb02 ~]# ls
anaconda-ks.cfg  init.sh  nginx-1.20.2  nginx-1.20.2.tar.gz
[root@lb02 nginx-1.20.2]# ls
auto  CHANGES  CHANGES.ru  conf  configure  contrib  html  LICENSE  man  README  src
[root@lb02 nginx-1.20.2]# ./configure  --conf-path=/etc/nginx/nginx.conf --with-http_gzip_static_module --with-stream --with-http_ssl_module --with-http_sub_module

# 若执行配置文件结尾出现以下情况

./configure: error: SSL modules require the OpenSSL library.
You can either do not enable the modules, or install the OpenSSL library
into the system, or build the OpenSSL library statically from the source
with nginx by using --with-openssl=<path> option.

# 安装依赖
[root@lb02 nginx-1.20.2]# yum install openssl openssl-devel zlib zlib-devel -y

# 再次执行
[root@lb02 nginx-1.20.2]# ./configure  --conf-path=/etc/nginx/nginx.conf --with-http_gzip_static_module --with-stream --with-http_ssl_module --with-http_sub_module

[root@lb02 nginx-1.20.2]# make make install

[root@lb02 sbin]# ls
nginx
[root@lb02 sbin]# pwd
/usr/local/nginx/sbin
[root@lb02 sbin]# mv nginx /usr/sbin/

[root@lb02 sbin]# nginx -V
nginx version: nginx/1.20.2
built by gcc 4.8.5 20150623 (Red Hat 4.8.5-44) (GCC) 
built with OpenSSL 1.0.2k-fips  26 Jan 2017
TLS SNI support enabled
configure arguments: --conf-path=/etc/nginx/nginx.conf --with-http_gzip_static_module --with-stream --with-http_ssl_module --with-http_sub_module

[root@lb01 ~]# scp /usr/lib/systemd/system/nginx.service root@192.168.15.6:/usr/lib/systemd/system/
[root@lb01 ~]# scp -r ./* 192.168.15.6:/etc/nginx/ teoi

# 若启动nginx发生以下错误
[root@lb02 nginx]# systemctl start nginx
Job for nginx.service failed because the control process exited with error code. See "systemctl

# 查看状态及报错
[root@lb02 nginx]# systemctl status nginx.service -l
...
1月 10 18:45:43 lb02 nginx[77789]: nginx: [emerg] getpwnam("www") failed in /etc/nginx/nginx.conf:1
...

# 创建www用户
[root@lb02 nginx]# id www
id: www: no such user
[root@lb02 nginx]# groupadd www -g 666
[root@lb02 nginx]# useradd www -u 666 -g 666 -M -r
# 重载nginx 
[root@lb02 nginx]# systemctl daemon-reload

# 或者出现以下错误
...
1月 10 18:50:44 lb02 nginx[83557]: nginx: [emerg] open() "/var/log/nginx/error.log" failed (2: No such file or directory)
...

# 创建目录
[root@lb02 nginx]# mkdir /var/log/nginx
[root@lb02 nginx]# chown -R www.www /var/log/nginx
[root@lb02 nginx]# systemctl start nginx.service

# 由于之前做过静态分离，现在lb02也需要做挂载
[root@lb02 ~]# yum install nfs-utils -y
[root@lb02 ~]# mount -t nfs 172.16.1.31:/opt/img/ /opt/img/

lb01，lb02
[root@lb01 ~]# systemctl start keepalived
[root@lb02 ~]# systemctl start keepalived

# lb01关闭自动切换到lb02，但客户访问依旧是 192.168.15.3
[root@lb01 ~]# systemctl stop keepalived

keepalived脑裂问题

两台高可用服务器在指定时间内，无法互相检查到对方的心跳而各自启动故障转移功能。

即为脑裂

可能出于某种原因，如nginx宕机，防火墙打开... 导致VRRP无法相互广播各自状态，lb01和lb02都会产生一个
192.168.15.3的IP，是浏览器无法做出选择。

解决脑裂问题

思想：如何在nginx宕机时将状态告知keepalived，是keepalived自动关闭，从而启动故障转移功能

1、如果Nginx宕机怎么办？
想办法告诉keepalived，Nginx的情况。
# ps -ef | grep [n]ginx

$?  : 上一条命令执行的结果。

$? = 0 (nginx 正常启动)
$? != 0 (nginx 启动失败)

# 那么如何让将nginx状态告诉keepalived?

VRRP_script
# 编辑脚本

2、局域网之内，keepalived无法相互广播，怎么办？
判断VIP是否可以ping的通

解决nginx无法正常启动，如何调用keepalived高可用

• nginx宕机

拓展

#!/bin/bash

# 解决Nginx无法正常启动
ps -ef | grep -q [n]ginx 

if [ $? -ne 0 ];then
	# 代表Nginx未正常启动
	systemctl start nginx &>/dev/null
	sleep 2
	ps -ef | grep -q [n]ginx
	if [ $? -ne 0 ];then
		systemctl stop keepalived 
	fi
fi

解释ping -c 1 $VIP &>/dev/null

举例：

[root@lb01 conf.d]# ping -c 1 192.168.15.3 &>/dev/null 
[root@lb01 conf.d]# echo $?
0

[root@lb01 conf.d]# ping -c 1 192.168.15.30 &>/dev/null 
[root@lb01 conf.d]# echo $?
1

指定ping次数

[root@lb02 ~]# ping -c 1 192.168.15.3
[root@lb02 ~]# echo $?
0

[root@lb02 ~]# ping -c 1 192.168.15.30
[root@lb02 ~]# echo $?
1

实践

[root@lb01 ~]# vim check_nginx.sh
#!/bin/bash

# 解决Nginx无法正常启动
ps -ef | grep -q [n]ginx 

if [ $? -ne 0 ];then
	# 代表Nginx未正常启动
	systemctl start nginx &>/dev/null
	sleep 2
	ps -ef | grep -q [n]ginx
	if [ $? -ne 0 ];then
		systemctl stop keepalived 
	fi
fi
[root@lb01 ~]# chmod +x check_nginx.sh 
[root@lb01 ~]# systemctl status nginx
[root@lb01 ~]# systemctl status keepalived
[root@lb01 ~]# ip a
'分别启动nginx keepalived'

[root@lb01 ~]# vim /etc/nginx/nginx.conf
adasdasddas (顶部随意添加字样，使nginx运行错误)
user  www;
worker_processes  auto;

[root@lb01 ~]# systemctl stop nginx
[root@lb01 ~]# ./checkNG.sh 

至此，解决nginx宕机，实现高可用

此时，再对keepalived配置文件做编辑，与解决nginx宕机脚本连用

[root@lb01 keepalived]# vim /etc/keepalived/keepalived.conf 
! Configuration File for keepalived

# 全局配置
global_defs {
   # 当前keepalived的唯一标识
   router_id lb01
}

# 检测脚本
vrrp_script check_nginx {
	# 指定脚本路径
    script "/etc/keepalived/checkNG.sh"
    # 执行间隔
    interval 5
}

# 配置VRRP协议
vrrp_instance VI_1 {
    # 状态，MASTER和BACKUP
    state MASTER
    # 绑定网卡
    interface eth0
    # 虚拟路由标示，可以理解为分组
    virtual_router_id 50
    # 优先级
    priority 100
    # 监测心跳间隔时间
    advert_int 1
    # 配置认证
    authentication {
        # 认证类型
        auth_type PASS
        # 认证的密码
        auth_pass 1111
    }
    # 设置VIP
    virtual_ipaddress {
        # 虚拟的VIP地址
        192.168.15.3
    }
    # 调用检查
    track_script {
        check_nginx
    }
}

[root@lb01 keepalived]# mv checkNG.sh /etc/keepalived/
[root@lb01 keepalived]# scp /etc/keepalived/keepalived.conf /etc/keepalived/checkNG.sh 192.168.15.6:/etc/keepalived/

[root@lb02 ~]# vim /etc/keepalived/keepalived.conf 
! Configuration File for keepalived

# 全局配置
global_defs {
   # 当前keepalived的唯一标识
   router_id lb02
}

# 检测脚本
 vrrp_script check_nginx {
       # 指定脚本路径
       script "/etc/keepalived/checkNG.sh"
       # 执行间隔
       interval 5
 }

# 配置VRRP协议
vrrp_instance VI_1 {
    # 状态，MASTER和BACKUP
    state BACKUP
    # 绑定网卡
    interface eth0
    # 虚拟路由标示，可以理解为分组
    virtual_router_id 50
    # 优先级
    priority 90
    # 监测心跳间隔时间
    advert_int 1
    # 配置认证
    authentication {
        # 认证类型
        auth_type PASS
        # 认证的密码
        auth_pass 1111
    }
    # 设置VIP
    virtual_ipaddress {
        # 虚拟的VIP地址
        192.168.15.3
    }
    # 调用检查
    track_script {
        check_nginx
     }
"/etc/keepalived/keepalived.conf" 47L, 867C  

此时，在lb01中关闭nginx，可以看到，如图所示，由于编辑了脚本，检测到nginx关闭便会自动启动

至此，真正实现keepalived高可用