030：Cetus中間件和MHA讀寫分離

本文轉載自查看原文 2018-08-31 18:08 619

030：Cetus中間件和MHA讀寫分離

############################################

line：V1.1

mail: gczheng@139.com

date: 2018-08-30

############################################

一、主機環境

虛擬機配置

CPU	內存	硬盤	OS版本	MySQL版本	MHA版本	Cetus版本
2-core	4G	500G	CentOS 7.5.1804	5.7.18	0.57	v1.0.0-44

主機信息

主機名	IP地址	Server_ID	MHA Manager	MHA Node	Cetus	備注
node05	192.168.222.175		部署	部署	部署	監控/MySQL主庫的故障轉移
node02	192.168.222.172	172	-	部署	-
node03	192.168.222.173	173	-	部署	-
node04	192.168.222.174	174	-	部署	-

參考官方架構圖

二、搭建主從

1、配置主從

搭建基於GTID主從復制環境
若開啟主從延遲檢測需創建庫proxy_heart_beat和表tb_heartbeat
創建用戶和密碼（默認用戶對tb_heartbeat有讀寫權限）
確認Cetus可以遠程登錄MySQL

0.主庫創建同步帳號grant RELOAD,REPLICATION SLAVE, REPLICATION CLIENT on *.* to repl@'%' identified by password 'xxxxxx'; flush privileges;
1.mysqldump 出主庫數據，增加 master-data=2；
2.導入從庫；
3.change master to master_host="192.168.222.171", master_port=3306, master_user='repl',master_password='repl', master_auto_position=1;
4.start slave;
5.創建mysql cetus測試賬號 grant all on *.* to gcdb@'%' identified by password 'xxxxxx'; flush privileges;
6.備庫設置set global read_only=1;
7.創建延遲校庫proxy_heart_beat和表tb_heartbeat
CREATE DATABASE proxy_heart_beat;
USE proxy_heart_beat;
CREATE TABLE tb_heartbeat ( p_id varchar(128) NOT NULL, p_ts timestamp(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3), PRIMARY KEY (p_id) ) ENGINE = InnoDB DEFAULT CHARSET = utf8;
8.創建測試庫ttt
create database if not exists ttt;
create table ttt.t1(id int(4)primary key not null auto_increment,nums int(20) not null);
insert into ttt.t1(nums) values(1),(2),(3),(4),(5);
update ttt.t1 set nums=100 where id =3;
delete from ttt.t1 where id =4;
select * from ttt.t1;
9.安裝sendmail服務
yum install sendmail -y

2、配置 hosts

添加hosts

cat <<EOF >/etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

192.168.222.171 node01.test.com node01
192.168.222.172 node02.test.com node02
192.168.222.173 node03.test.com node03
192.168.222.174 node04.test.com node04
192.168.222.175 node05.test.com node05
192.168.222.176 node06.test.com node06
EOF

3、配置免秘鑰

mha管理節點：

ssh-keygen -t rsa

ssh-copy-id -i /root/.ssh/id_rsa.pub "root@192.168.222.172"
ssh-copy-id -i /root/.ssh/id_rsa.pub "root@192.168.222.173"
ssh-copy-id -i /root/.ssh/id_rsa.pub "root@192.168.222.174"

mysql節點各自生成公私鑰，並將公鑰拷貝給其他mysql節點

# 192.168.222.172
ssh-keygen -t rsa
ssh-copy-id -i /root/.ssh/id_rsa.pub "root@192.168.222.171"
ssh-copy-id -i /root/.ssh/id_rsa.pub "root@192.168.222.173"
ssh-copy-id -i /root/.ssh/id_rsa.pub "root@192.168.222.174"

# 192.168.222.173
ssh-keygen -t rsa
ssh-copy-id -i /root/.ssh/id_rsa.pub "root@192.168.222.172"
ssh-copy-id -i /root/.ssh/id_rsa.pub "root@192.168.222.171"
ssh-copy-id -i /root/.ssh/id_rsa.pub "root@192.168.222.174"

# 192.168.222.174
ssh-keygen -t rsa
ssh-copy-id -i /root/.ssh/id_rsa.pub "root@192.168.222.172"
ssh-copy-id -i /root/.ssh/id_rsa.pub "root@192.168.222.171"
ssh-copy-id -i /root/.ssh/id_rsa.pub "root@192.168.222.173"

測試ssh是否免密碼登錄

三、安裝MHA和Cetus

1、下載包和安裝依賴包

mha4mysql-manager-0.57.tar.gz和mha4mysql-node-0.57.tar.gz

下載地址

cetus源碼下載
git clone https://github.com/Lede-Inc/cetus.git

# 安裝Cetus依賴包
yum -y install cmake gcc glib2-devel flex libevent-devel mysql-devel gperftools-libs 
# 安裝mha依賴包
yum -y install perl-DBD-MySQL perl-Config-Tiny perl-Log-Dispatch perl-Parallel-ForkManager perl-ExtUtils-CBuilder perl-ExtUtils-MakeMaker perl-CPAN perl-ExtUtils-Manifest

2、安裝MHA Node

tar zxvf mha4mysql-node-0.57.tar.gz
cd mha4mysql-node-0.57
perl Makefile.PL
make && make install

-- Node安裝完成后會在/usr/local/bin/下生成這些文件
#save_binary_logs : 保存和復制master的二進制日志。 
#apply_diff_relay_logs : 識別差異的中繼日志事件並應用於其它slave。 
#filter_mysqlbinlog : 去除不必要的ROLLBACK事件(MHA已不再使用這個工具)。 
#purge_relay_logs : 清除中繼日志(不會阻塞SQL線程)。

3、安裝MHA manager

在所有的 MySQL上安裝 Node 節點；在 MHA 的管理節點安裝 manager 節點。

tar zvxf mha4mysql-manager-0.57.tar.gz 
cd mha4mysql-manager-0.57
perl Makefile.PL
make && make install

-- Manage安裝完成后會在/usr/local/bin/下生成這些文件
#masterha_check_ssh : 檢查MHA的SSH配置。 
#masterha_check_repl : 檢查MySQL復制。 
#masterha_manager : 啟動MHA。 
#masterha_check_status : 檢測當前MHA運行狀態。 
#masterha_master_monitor : 監測master是否宕機。 
#masterha_master_switch : 控制故障轉移(自動或手動)。 
#masterha_conf_host : 添加或刪除配置的server信息。

cp -r /software/mha4mysql-manager-0.57/samples/scripts/* /usr/local/bin/
-- 拷貝/software/mha4mysql-manager-0.57/samples/scripts/下的腳本到/usr/local/bin/

#master_ip_failover ：自動切換時vip管理的腳本，不是必須，如果我們使用keepalived的，我們可以自己編寫腳本完成對vip的管理，比如監控mysql，如果mysql異常，我們停止keepalived就行，這樣vip就會自動漂移
#master_ip_online_change：在線切換時vip的管理，不是必須，同樣可以可以自行編寫簡單的shell完成
#power_manager：故障發生后關閉主機的腳本，不是必須
#send_report：因故障切換后發送報警的腳本，不是必須，可自行編寫簡單的shell完成。

4、安裝Cetus

1.安裝說明

Cetus利用自動化建構系統CMake進行編譯安裝，其中描述構建過程的構建文件CMakeLists.txt已經在源碼中的主目錄和子目錄中，下載源碼並解壓后具體安裝步驟如下：

創建編譯目錄：在源碼主目錄下創建獨立的目錄build，並轉到該目錄下

mkdir build/
cd build/

編譯：利用cmake進行編譯，指令如下

讀寫分離版本：
cmake ../ -DCMAKE_BUILD_TYPE=Debug -DCMAKE_INSTALL_PREFIX=/home/user/cetus_install -DSIMPLE_PARSER=ON

分庫版本：
cmake ../ -DCMAKE_BUILD_TYPE=Debug -DCMAKE_INSTALL_PREFIX=/home/user/cetus_install -DSIMPLE_PARSER=OFF

其中CMAKE_BUILD_TYPE變量可以選擇生成 debug 版和或release 版的程序，CMAKE_INSTALL_PREFIX變量確定軟件的實際安裝目錄的絕對路徑，安裝目錄建議以/home/user/日期.編譯版本.分支.commit_id的方式命名；SIMPLE_PARSER變量確定軟件的編譯版本，設置為ON則編譯讀寫分離版本，否則編譯分庫版本。

該過程會檢查您的系統是否缺少一些依賴庫和依賴軟件，可以根據錯誤代碼安裝相應依賴。

安裝：執行make install進行安裝

make install

配置：Cetus運行前還需要編輯配置文件

cd /home/user/cetus_install/conf/
cp XXX.json.example XXX.json
cp XXX.conf.example XXX.conf
vi XXX.json
vi XXX.conf

配置文件在make insatll后存在示例文件，以.example結尾，目錄為/home/user/cetus_install/conf/，包括用戶設置文件（users.json）、變量處理配置文件（variables.json）、分庫版本的分片規則配置文件（sharding.json）、讀寫分離版本的啟動配置文件（proxy.conf）和分庫版本的啟動配置文件（shard.conf）。

根據具體編譯安裝的版本編輯相關配置文件，若使用讀寫分離功能則需配置users.json和proxy.conf，若使用sharding功能則需配置users.json、sharding.json和shard.conf，其中兩個版本的variables.json均可選配。

配置文件的具體說明見Cetus 讀寫分離版配置文件說明和Cetus 分庫(sharding)版配置文件說明。

啟動：Cetus可以利用bin/cetus啟動

讀寫分離版本：
bin/cetus --defaults-file=conf/proxy.conf [--conf-dir＝/home/user/cetus_install/conf/]

分庫版本：
bin/cetus --defaults-file=conf/shard.conf [--conf-dir＝/home/user/cetus_install/conf/]

其中Cetus啟動時可以添加命令行選項，--defaults-file選項用來加載啟動配置文件（proxy.conf或者shard.conf），且在啟動前保證啟動配置文件的權限為660；--conf-dir是可選項，用來加載其他配置文件(.json文件)，默認為當前目錄下conf文件夾。

Cetus可起動守護進程后台運行，也可在進程意外終止自動啟動一個新進程，可通過啟動配置選項進行設置。

2.安裝實施

#進入源碼目錄
[root@node05 software]# git clone https://github.com/Lede-Inc/cetus.git
[root@node05 software]# cd cetus/
[root@node05 cetus]# mkdir build/ && cd build
[root@node05 build]# cmake ../ -DCMAKE_BUILD_TYPE=Debug -DCMAKE_INSTALL_PREFIX=/usr/local/cetus -DSIMPLE_PARSER=ON
[root@node05 build]# make install
[root@node05 build]# ll /usr/local/cetus/
total 0
drwxr-xr-x 2 root root  19 Aug 31 09:29 bin
drwxr-xr-x 2 root root 143 Aug 31 09:29 conf
drwxr-xr-x 4 root root 210 Aug 31 09:29 lib
drwxr-xr-x 2 root root  19 Aug 31 09:29 libexec
drwxr-xr-x 2 root root  23 Aug 31 09:29 logs

[root@node05 build]# ll /usr/local/share/perl5/MHA/
total 388
-r--r--r-- 1 root root 12997 May 31  2015 BinlogHeaderParser.pm
-r--r--r-- 1 root root 18783 May 31  2015 BinlogManager.pm
-r--r--r-- 1 root root  2251 May 31  2015 BinlogPosFinderElp.pm
-r--r--r-- 1 root root  1648 May 31  2015 BinlogPosFinder.pm
-r--r--r-- 1 root root  3130 May 31  2015 BinlogPosFinderXid.pm
-r--r--r-- 1 root root  5684 May 31  2015 BinlogPosFindManager.pm
-r--r--r-- 1 root root 17480 May 31  2015 Config.pm
-r--r--r-- 1 root root 27019 May 31  2015 DBHelper.pm
-r--r--r-- 1 root root  3075 May 31  2015 FileStatus.pm
-r--r--r-- 1 root root 20370 May 31  2015 HealthCheck.pm
-r--r--r-- 1 root root 10560 May 31  2015 ManagerAdmin.pm
-r--r--r-- 1 root root  3679 May 31  2015 ManagerAdminWrapper.pm
-r--r--r-- 1 root root  3508 May 31  2015 ManagerConst.pm
-r--r--r-- 1 root root  4612 May 31  2015 ManagerUtil.pm
-r--r--r-- 1 root root 74140 May 31  2015 MasterFailover.pm
-r--r--r-- 1 root root 23654 May 31  2015 MasterMonitor.pm
-r--r--r-- 1 root root 23466 May 31  2015 MasterRotate.pm
-r--r--r-- 1 root root  1308 May 31  2015 NodeConst.pm
-r--r--r-- 1 root root  6689 May 31  2015 NodeUtil.pm
-r--r--r-- 1 root root 44325 May 31  2015 ServerManager.pm
-r--r--r-- 1 root root 33213 May 31  2015 Server.pm
-r--r--r-- 1 root root  6531 May 31  2015 SlaveUtil.pm
-r--r--r-- 1 root root  4874 May 31  2015 SSHCheck.pm
[root@node05 build]#

5、cetus替換mha部分文件

#使用 mha_ld/src 替換所有文件/usr/share/perl5/vendor_perl/MHA/目錄的所有同名文件
[root@node05 build]# cd ../mha_ld/src/
[root@node05 src]# ls
BinlogHeaderParser.pm  BinlogPosFindManager.pm  HealthCheck.pm          MasterFailover.pm         NodeUtil.pm      ServerManager.pm
BinlogManager.pm       cetus.cnf                ManagerAdmin.pm         masterha_secondary_check  ProxyManager.pm  Server.pm
BinlogPosFinderElp.pm  Config.pm                ManagerAdminWrapper.pm  MasterMonitor.pm          sample.cnf       SlaveUtil.pm
BinlogPosFinder.pm     DBHelper.pm              ManagerConst.pm         MasterRotate.pm           sendMail.sh      SSHCheck.pm
BinlogPosFinderXid.pm  FileStatus.pm            ManagerUtil.pm          NodeConst.pm              sendmail.txt

[root@node05 src]# ls /usr/local/share/perl5/MHA/
BinlogHeaderParser.pm  BinlogPosFinderXid.pm    FileStatus.pm           ManagerConst.pm    MasterRotate.pm   Server.pm
BinlogManager.pm       BinlogPosFindManager.pm  HealthCheck.pm          ManagerUtil.pm     NodeConst.pm      SlaveUtil.pm
BinlogPosFinderElp.pm  Config.pm                ManagerAdmin.pm         MasterFailover.pm  NodeUtil.pm       SSHCheck.pm
BinlogPosFinder.pm     DBHelper.pm              ManagerAdminWrapper.pm  MasterMonitor.pm   ServerManager.pm

[root@node05 src]# rsync /software/cetus/mha_ld/src/* /usr/local/share/perl5/MHA/
[root@node05 src]#  ls /usr/local/share/perl5/MHA/
BinlogHeaderParser.pm  BinlogPosFindManager.pm  HealthCheck.pm          MasterFailover.pm         NodeUtil.pm      ServerManager.pm
BinlogManager.pm       cetus.cnf                ManagerAdmin.pm         masterha_secondary_check  ProxyManager.pm  Server.pm
BinlogPosFinderElp.pm  Config.pm                ManagerAdminWrapper.pm  MasterMonitor.pm          sample.cnf       SlaveUtil.pm
BinlogPosFinder.pm     DBHelper.pm              ManagerConst.pm         MasterRotate.pm           sendMail.sh      SSHCheck.pm
BinlogPosFinderXid.pm  FileStatus.pm            ManagerUtil.pm          NodeConst.pm              sendmail.txt

#使用 mha_ld/masterha_secondary_check替換masterha_secondary_check命令 which masterha_secondary_check
[root@node05 src]# which masterha_secondary_check
/usr/local/bin/masterha_secondary_check
[root@node05 src]# rsync /software/cetus/mha_ld/src/masterha_secondary_check /usr/local/bin/
[root@node05 src]# chmod +x /usr/local/bin/masterha_secondary_check
[root@node05 src]# ll /usr/local/bin/masterha_secondary_check
-r-xr-xr-x 1 root root 5186 Aug 31 11:48 /usr/local/bin/masterha_secondary_check

四、MHA和cetus配置

1、創建和修改配置文件

在manager節點上創建cetus配置文件（cetus.cnf）

[root@node05 src]#  cp /software/cetus/mha_ld/cetus.cnf /etc/cetus.cnf
[root@node05 src]#  vim /etc/cetus.cnf
[root@node05 src]#  cat /etc/cetus.cnf 
middle_ipport=192.168.222.175:23306
middle_user=admin
middle_pass=admin

在manager節點上創建cetus的user和proxy.配置文件（users.json和proxy.conf）

[root@node05 src]# cp /usr/local/cetus/conf/proxy.conf.example /usr/local/cetus/conf/proxy.conf
[root@node05 src]# cp /usr/local/cetus/conf/users.json.example  /usr/local/cetus/conf/users.json
[root@node05 src]# vim /usr/local/cetus/conf/users.json
[root@node05 src]# cat /usr/local/cetus/conf/users.json
{
	"users":	[{
			"user":	"gcdb",
			"client_pwd":	"iforgot",
			"server_pwd":	"iforgot"
		}, {
			"user":	"cetus_app1",
			"client_pwd":	"cetus_app1",
			"server_pwd":	"cetus_app1"
		}]
}

[root@node05 src]# vim  /usr/local/cetus/conf/proxy.conf
[root@node05 src]# cat /usr/local/cetus/conf/proxy.conf
[cetus]
# For mode-switch
daemon = true

# Loaded Plugins
plugins=proxy,admin

# Proxy Configuration, For example: MySQL master and salve host ip are both 192.0.0.1
proxy-address=192.168.222.175:13306
proxy-backend-addresses=192.168.222.172:3306
proxy-read-only-backend-addresses=192.168.222.173:3306,192.168.222.174:3306


# Admin Configuration
admin-address=192.168.222.175:23306
admin-username=admin
admin-password=admin

# Backend Configuration, use test db and username created
default-db=ttt
default-username=gcdb
default-pool-size=100
max-resp-size=10485760
long-query-time=100

# File and Log Configuration, put log in /data and marked by proxy port, /data/cetus needs to be created manually and has rw authority for cetus os user
max-open-files = 65536
pid-file = cetus6001.pid
plugin-dir=lib/cetus/plugins
log-file=/var/log/cetus.log
log-level=debug

# Check salve delay
disable-threads=false
check-slave-delay=true
slave-delay-down=5
slave-delay-recover=1

# For trouble
keepalive=true
verbose-shutdown=true
log-backtrace-on-crash=true

在manager節點創建全局配置文件(masterha_default.cnf)

[root@node05 software]# cat /etc/masterha_default.cnf
[server default]
#用於指定mha manager產生相關狀態文件全路徑
manager_workdir=/var/log/masterha

#指定mha manager的絕對路徑的文件名日志文件
manager_log=/var/log/masterha/mha.log

# cetus porxy文件絕對路徑
proxy_conf=/etc/cetus.cnf

# 登陸mysql數據庫賬戶及密碼
user=gcdb
password=iforgot

# ssh用戶
ssh_user=root
ssh_port=22

# mysql數據庫master節點binlog的位置，該參數用於當master節點死掉后通過ssh方式順序讀取binlog event，需要配置，因為master節點死掉后無法通過replication機制來自動獲取binlog日志位置
master_binlog_dir=/r2/mysqldata
#master_binlog_dir= /r2/mysqldata

# 用於檢測各節點間的連接性，此處詳細可參考MHA parameters描述部分
secondary_check_script= masterha_secondary_check -s 192.168.222.173  -s 192.168.222.174
ping_interval=3
ping_type=select
remote_workdir=/tmp

#定義用於實現VIP漂移的腳本，后面的是shutdown以及report腳本
master_ip_failover_script=/usr/local/bin/master_ip_failover
#shutdown_script=/usr/local/bin/power_manager
shutdown_script=
report_script="/usr/local/share/perl5/MHA/sendMail.sh"


#MySQL用於復制的賬號
repl_user=repl
repl_password=repl

在manager節點上創建配置文件(mha01.cnf)

[root@node05 software]#  cat /etc/mha.cnf
[server default]
manager_log=/var/log/masterha/mha.log
manager_workdir=/var/log/masterha

[server1]
hostname=192.168.222.172
master_binlog_dir=/r2/mysqldata
port=3306

[server2]
hostname=192.168.222.173
master_binlog_dir=/r2/mysqldata
port=3306

[server3]
hostname=192.168.222.174
master_binlog_dir=/r2/mysqldata
port=3306

在manager節點上修改故障轉移腳本（master_ip_failover）

[root@node05 mha4mysql-manager-0.57]# vim  /usr/local/bin/master_ip_failover
#!/usr/bin/env perl

use strict;
use warnings FATAL => 'all';
use Getopt::Long;
use MHA::DBHelper;

my (
    $command,          $ssh_user,        $orig_master_host, $orig_master_ip,
    $orig_master_port, $new_master_host, $new_master_ip,    $new_master_port
);

#添加切換vip
my $vip = '192.168.222.99/24';
my $key = '1';
my $ssh_start_vip = "/sbin/ifconfig ens224:$key $vip";
my $ssh_stop_vip = "/sbin/ifconfig ens224:$key down";
$ssh_user = "root";

GetOptions(
    'command=s'          => \$command,
    'ssh_user=s'         => \$ssh_user,
    'orig_master_host=s' => \$orig_master_host,
    'orig_master_ip=s'   => \$orig_master_ip,
    'orig_master_port=i' => \$orig_master_port,
    'new_master_host=s'  => \$new_master_host,
    'new_master_ip=s'    => \$new_master_ip,
    'new_master_port=i'  => \$new_master_port,
);

exit &main();

sub main {

    print "\n\nIN SCRIPT TEST====$ssh_stop_vip==$ssh_start_vip===\n\n";

    if ( $command eq "stop" || $command eq "stopssh" ) {

        my $exit_code = 1;
        eval {
            print "Disabling the VIP on old master: $orig_master_host \n";
            &stop_vip();
            $exit_code = 0;
        };
        if ($@) {
            warn "Got Error: $@\n";
            exit $exit_code;
        }
        exit $exit_code;
    }
    elsif ( $command eq "start" ) {

        my $exit_code = 10;
        eval {
            print "Enabling the VIP - $vip on the new master - $new_master_host \n";
            &start_vip();
            $exit_code = 0;
        };
        if ($@) {
            warn $@;
            exit $exit_code;
        }
        exit $exit_code;
    }
    elsif ( $command eq "status" ) {
        print "Checking the Status of the script.. OK \n";
        exit 0;
    }
    else {
        &usage();
        exit 1;
    }
}

sub start_vip() {
    `ssh $ssh_user\@$new_master_host \" $ssh_start_vip \"`;
}
sub stop_vip() {
     return 0  unless  ($ssh_user);
    `ssh $ssh_user\@$orig_master_host \" $ssh_stop_vip \"`;
}

sub usage {
    print
    "Usage: master_ip_failover --command=start|stop|stopssh|status --orig_master_host=host --orig_master_ip=ip --orig_master_port=port --new_master_host=host --new_master_ip=ip --new_master_port=port\n";
}

2、檢查 SSH/復制/MHA Manager 的配置

檢查SSH 情況：masterha_check_ssh --conf=/etc/mha.cnf

[root@node05 src]# masterha_check_ssh --conf=/etc/mha.cnf
Fri Aug 31 13:55:51 2018 - [info] Reading default configuration from /etc/masterha_default.cnf..
Fri Aug 31 13:55:51 2018 - [info] Reading application default configuration from /etc/mha.cnf..
Fri Aug 31 13:55:51 2018 - [info] Reading server configuration from /etc/mha.cnf..
Fri Aug 31 13:55:51 2018 - [info] Starting SSH connection tests..
Fri Aug 31 13:55:52 2018 - [debug] 
Fri Aug 31 13:55:51 2018 - [debug]  Connecting via SSH from root@192.168.222.172(192.168.222.172:22) to root@192.168.222.173(192.168.222.173:22)..
Fri Aug 31 13:55:51 2018 - [debug]   ok.
Fri Aug 31 13:55:51 2018 - [debug]  Connecting via SSH from root@192.168.222.172(192.168.222.172:22) to root@192.168.222.174(192.168.222.174:22)..
Fri Aug 31 13:55:51 2018 - [debug]   ok.
Fri Aug 31 13:55:52 2018 - [debug] 
Fri Aug 31 13:55:51 2018 - [debug]  Connecting via SSH from root@192.168.222.173(192.168.222.173:22) to root@192.168.222.172(192.168.222.172:22)..
Fri Aug 31 13:55:51 2018 - [debug]   ok.
Fri Aug 31 13:55:51 2018 - [debug]  Connecting via SSH from root@192.168.222.173(192.168.222.173:22) to root@192.168.222.174(192.168.222.174:22)..
Fri Aug 31 13:55:52 2018 - [debug]   ok.
Fri Aug 31 13:55:53 2018 - [debug] 
Fri Aug 31 13:55:52 2018 - [debug]  Connecting via SSH from root@192.168.222.174(192.168.222.174:22) to root@192.168.222.172(192.168.222.172:22)..
Fri Aug 31 13:55:52 2018 - [debug]   ok.
Fri Aug 31 13:55:52 2018 - [debug]  Connecting via SSH from root@192.168.222.174(192.168.222.174:22) to root@192.168.222.173(192.168.222.173:22)..
Fri Aug 31 13:55:52 2018 - [debug]   ok.
Fri Aug 31 13:55:53 2018 - [info] All SSH connection tests passed successfully.
[root@node05 src]#

檢查復制情況：masterha_check_repl --conf=/etc/mha.cnf

[root@node05 src]# masterha_check_repl --conf=/etc/mha.cnf
Fri Aug 31 13:59:17 2018 - [info] Reading default configuration from /etc/masterha_default.cnf..
Fri Aug 31 13:59:17 2018 - [info] Reading application default configuration from /etc/mha.cnf..
Fri Aug 31 13:59:17 2018 - [info] Reading server configuration from /etc/mha.cnf..
Fri Aug 31 13:59:17 2018 - [info] MHA::MasterMonitor version 0.56.
Fri Aug 31 13:59:17 2018 - [info] g_workdir: /var/log/masterha
Fri Aug 31 13:59:17 2018 - [info] proxy_conf: /etc/cetus.cnf
Fri Aug 31 13:59:17 2018 - [info] -------------ManagerUtil::check_node_version-----------
Fri Aug 31 13:59:18 2018 - [info] GTID failover mode = 1
Fri Aug 31 13:59:18 2018 - [info] Dead Servers:
Fri Aug 31 13:59:18 2018 - [info] Alive Servers:
Fri Aug 31 13:59:18 2018 - [info]   192.168.222.172(192.168.222.172:3306)
Fri Aug 31 13:59:18 2018 - [info]   192.168.222.173(192.168.222.173:3306)
Fri Aug 31 13:59:18 2018 - [info]   192.168.222.174(192.168.222.174:3306)
Fri Aug 31 13:59:18 2018 - [info] Alive Slaves:
Fri Aug 31 13:59:18 2018 - [info]   192.168.222.173(192.168.222.173:3306)  Version=5.7.18-log (oldest major version between slaves) log-bin:enabled
Fri Aug 31 13:59:18 2018 - [info]     GTID ON
Fri Aug 31 13:59:18 2018 - [info]     Replicating from 192.168.222.172(192.168.222.172:3306)
Fri Aug 31 13:59:18 2018 - [info]   192.168.222.174(192.168.222.174:3306)  Version=5.7.18-log (oldest major version between slaves) log-bin:enabled
Fri Aug 31 13:59:18 2018 - [info]     GTID ON
Fri Aug 31 13:59:18 2018 - [info]     Replicating from 192.168.222.172(192.168.222.172:3306)
Fri Aug 31 13:59:18 2018 - [info] Current Alive Master: 192.168.222.172(192.168.222.172:3306)
Fri Aug 31 13:59:18 2018 - [info] Checking slave configurations..
Fri Aug 31 13:59:18 2018 - [info] Checking replication filtering settings..
Fri Aug 31 13:59:18 2018 - [info]  binlog_do_db= , binlog_ignore_db= 
Fri Aug 31 13:59:18 2018 - [info]  Replication filtering check ok.
Fri Aug 31 13:59:18 2018 - [info] GTID (with auto-pos) is supported. Skipping all SSH and Node package checking.
Fri Aug 31 13:59:18 2018 - [info] Checking SSH publickey authentication settings on the current master..
Fri Aug 31 13:59:18 2018 - [info] HealthCheck: SSH to 192.168.222.172 is reachable.
Fri Aug 31 13:59:18 2018 - [info] 
192.168.222.172(192.168.222.172:3306) (current master)
 +--192.168.222.173(192.168.222.173:3306)
 +--192.168.222.174(192.168.222.174:3306)

Fri Aug 31 13:59:18 2018 - [info] Checking replication health on 192.168.222.173..
Fri Aug 31 13:59:18 2018 - [info]  ok.
Fri Aug 31 13:59:18 2018 - [info] Checking replication health on 192.168.222.174..
Fri Aug 31 13:59:18 2018 - [info]  ok.
Fri Aug 31 13:59:18 2018 - [info] Checking master_ip_failover_script status:
Fri Aug 31 13:59:18 2018 - [info]   /usr/local/bin/master_ip_failover --command=status --ssh_user=root --orig_master_host=192.168.222.172 --orig_master_ip=192.168.222.172 --orig_master_port=3306 

IN SCRIPT TEST====/sbin/ifconfig ens224:1 down==/sbin/ifconfig ens224:1 192.168.222.99/24===
Checking the Status of the script.. OK 
Fri Aug 31 13:59:18 2018 - [info]  OK.
Fri Aug 31 13:59:18 2018 - [warning] shutdown_script is not defined.
Fri Aug 31 13:59:18 2018 - [info] Got exit code 0 (Not master dead).

MySQL Replication Health is OK.
[root@node05 src]#

檢查MHA 狀態：masterha_check_status --conf=/etc/mha.cnf

[root@node05 src]# masterha_check_status --conf=/etc/mha.cnf
mha is stopped(2:NOT_RUNNING).

檢查masterha_secondary_check腳本中mysql客戶端安裝位置

[root@node05 software]# which mysql
/usr/local/mysql/bin/mysql
[root@node05 software]# grep -w /usr/bin/mysql  /usr/local/bin/*
/usr/local/bin/masterha_secondary_check:        . "/usr/bin/mysql -u$master_user -p$master_password -h$master_host -P$master_port "
[root@node05 software]# ln -s /usr/local/mysql/bin/mysql /usr/bin/mysql
[root@node05 software]# which mysql
/usr/bin/mysql

3、啟動MHA manager

session A

[root@node05 src]# mkdir -p /var/log/masterha/

session B

[root@node05 mha4mysql-manager-0.57]# nohup masterha_manager --conf=/etc/masterha/app1.cnf --remove_dead_master_conf --ignore_last_failover < /dev/null > /var/log/masterha/app1/manager.log 2>&1 &
[1] 19246

session A

[root@node05 ~]# tail -f /var/log/masterha/app1/manager.log

IN SCRIPT TEST====/sbin/ifconfig ens224:1 down==/sbin/ifconfig ens224:1 192.168.222.99/24===

Checking the Status of the script.. OK 
Fri Aug 31 14:06:45 2018 - [info]  OK.
Fri Aug 31 14:06:45 2018 - [warning] shutdown_script is not defined.
Fri Aug 31 14:06:45 2018 - [info] Set master ping interval 3 seconds.
Fri Aug 31 14:06:45 2018 - [info] Set secondary check script: masterha_secondary_check  -s 192.168.222.172 -s 192.168.222.173  -s 192.168.222.174
Fri Aug 31 14:06:45 2018 - [info] Starting ping health check on 192.168.222.172(192.168.222.172:3306)..
Fri Aug 31 14:06:45 2018 - [info] Ping(SELECT) succeeded, waiting until MySQL doesn't respond..
[root@node05 src]# cat /var/log/masterha/mha.log
Fri Aug 31 14:06:44 2018 - [info] Reading default configuration from /etc/masterha_default.cnf..
Fri Aug 31 14:06:44 2018 - [info] Reading application default configuration from /etc/mha.cnf..
Fri Aug 31 14:06:44 2018 - [info] Reading server configuration from /etc/mha.cnf..
Fri Aug 31 14:06:44 2018 - [info] MHA::MasterMonitor version 0.56.
Fri Aug 31 14:06:44 2018 - [info] g_workdir: /var/log/masterha
Fri Aug 31 14:06:44 2018 - [info] proxy_conf: /etc/cetus.cnf
Fri Aug 31 14:06:44 2018 - [info] -------------ManagerUtil::check_node_version-----------
Fri Aug 31 14:06:45 2018 - [info] GTID failover mode = 1
Fri Aug 31 14:06:45 2018 - [info] Dead Servers:
Fri Aug 31 14:06:45 2018 - [info] Alive Servers:
Fri Aug 31 14:06:45 2018 - [info]   192.168.222.172(192.168.222.172:3306)
Fri Aug 31 14:06:45 2018 - [info]   192.168.222.173(192.168.222.173:3306)
Fri Aug 31 14:06:45 2018 - [info]   192.168.222.174(192.168.222.174:3306)
Fri Aug 31 14:06:45 2018 - [info] Alive Slaves:
Fri Aug 31 14:06:45 2018 - [info]   192.168.222.173(192.168.222.173:3306)  Version=5.7.18-log (oldest major version between slaves) log-bin:enabled
Fri Aug 31 14:06:45 2018 - [info]     GTID ON
Fri Aug 31 14:06:45 2018 - [info]     Replicating from 192.168.222.172(192.168.222.172:3306)
Fri Aug 31 14:06:45 2018 - [info]   192.168.222.174(192.168.222.174:3306)  Version=5.7.18-log (oldest major version between slaves) log-bin:enabled
Fri Aug 31 14:06:45 2018 - [info]     GTID ON
Fri Aug 31 14:06:45 2018 - [info]     Replicating from 192.168.222.172(192.168.222.172:3306)
Fri Aug 31 14:06:45 2018 - [info] Current Alive Master: 192.168.222.172(192.168.222.172:3306)
Fri Aug 31 14:06:45 2018 - [info] Checking slave configurations..
Fri Aug 31 14:06:45 2018 - [info] Checking replication filtering settings..
Fri Aug 31 14:06:45 2018 - [info]  binlog_do_db= , binlog_ignore_db= 
Fri Aug 31 14:06:45 2018 - [info]  Replication filtering check ok.
Fri Aug 31 14:06:45 2018 - [info] GTID (with auto-pos) is supported. Skipping all SSH and Node package checking.
Fri Aug 31 14:06:45 2018 - [info] Checking SSH publickey authentication settings on the current master..
Fri Aug 31 14:06:45 2018 - [info] HealthCheck: SSH to 192.168.222.172 is reachable.
Fri Aug 31 14:06:45 2018 - [info] 
192.168.222.172(192.168.222.172:3306) (current master)
 +--192.168.222.173(192.168.222.173:3306)
 +--192.168.222.174(192.168.222.174:3306)

Fri Aug 31 14:06:45 2018 - [info] Checking master_ip_failover_script status:
Fri Aug 31 14:06:45 2018 - [info]   /usr/local/bin/master_ip_failover --command=status --ssh_user=root --orig_master_host=192.168.222.172 --orig_master_ip=192.168.222.172 --orig_master_port=3306 


IN SCRIPT TEST====/sbin/ifconfig ens224:1 down==/sbin/ifconfig ens224:1 192.168.222.99/24===

Checking the Status of the script.. OK 
Fri Aug 31 14:06:45 2018 - [info]  OK.
Fri Aug 31 14:06:45 2018 - [warning] shutdown_script is not defined.
Fri Aug 31 14:06:45 2018 - [info] Set master ping interval 3 seconds.
Fri Aug 31 14:06:45 2018 - [info] Set secondary check script: masterha_secondary_check  -s 192.168.222.172 -s 192.168.222.173  -s 192.168.222.174
Fri Aug 31 14:06:45 2018 - [info] Starting ping health check on 192.168.222.172(192.168.222.172:3306)..
Fri Aug 31 14:06:45 2018 - [info] Ping(SELECT) succeeded, waiting until MySQL doesn't respond..  --表示已開啟監聽

session B

[root@node05 software]# masterha_check_status --conf=/etc/mha.cnf
mha (pid:25386) is running(0:PING_OK), master:192.168.222.172
[root@node05 software]#

4、啟動cetus

啟動前保證啟動配置文件的權限為660

[root@node05 src]# chmod 660 /usr/local/cetus/conf/*
[root@node05 src]# ll /usr/local/cetus/conf/
total 28
-rw-rw---- 1 root root 1053 Aug 31 12:12 proxy.conf
-rw-rw---- 1 root root 1011 Aug 31 09:21 proxy.conf.example
-rw-rw---- 1 root root 1199 Aug 31 09:21 shard.conf.example
-rw-rw---- 1 root root 1030 Aug 31 09:21 sharding.json.example
-rw-rw---- 1 root root  189 Aug 31 12:04 users.json
-rw-rw---- 1 root root  198 Aug 31 09:21 users.json.example
-rw-rw---- 1 root ro

守護進程模式啟動Cetus
session A

[root@node01 software]# ll mha*
-rw-r--r-- 1 root root 118521 Aug 28 15:41 mha4mysql-manager-0.57.tar.gz
-rw-r--r-- 1 root root  54484 Aug 28 15:41 mha4mysql-node-0.57.tar.gz
[root@node05 src]# /usr/local/cetus/bin/cetus --defaults-file=/usr/local/cetus/conf/proxy.conf
[root@node05 src]#  ps -ef |grep cetusb
root     26580     1  0 14:23 ?        00:00:00 /usr/local/cetus/libexec/cetus --defaults-file=/usr/local/cetus/conf/proxy.conf
root     26581 26580  0 14:23 ?        00:00:00 /usr/local/cetus/libexec/cetus --defaults-file=/usr/local/cetus/conf/proxy.conf
root     26624  2272  0 14:23 pts/0    00:00:00 grep --color=auto cetus
[root@node05 src]#

查看Cetus狀態
session B

[root@node05 src]# mysql -ugcdb  -piforgot -h192.168.222.175 -P13306
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 1
Server version: 5.7.18-log (cetus) MySQL Community Server (GPL)

Copyright (c) 2000, 2017, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

(gcdb@192.168.222.175) 14:26:55 [(none)]> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mysql              |
| performance_schema |
| proxy_heart_beat   |
| sys                |
| ttt                |
+--------------------+
6 rows in set (0.00 sec)

(gcdb@192.168.222.175) 14:27:02 [(none)]> exit
Bye
[root@node05 src]# mysql -uadmin  -padmin -h192.168.222.175 -P23306
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 1
Server version: 5.7 admin

Copyright (c) 2000, 2017, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

(admin@192.168.222.175) 14:27:41 [(none)]> select conn_details from backends;select * from backends;                                         
+-------------+----------+------------+------------+-------------+
| backend_ndx | username | idle_conns | used_conns | total_conns |
+-------------+----------+------------+------------+-------------+
| 0           | gcdb     | 100        | 0          | 100         |
| 1           | gcdb     | 100        | 0          | 100         |
| 2           | gcdb     | 100        | 0          | 100         |
+-------------+----------+------------+------------+-------------+
3 rows in set (0.00 sec)

+-------------+----------------------+-------+------+-------------+------+------------+------------+-------------+
| backend_ndx | address              | state | type | slave delay | uuid | idle_conns | used_conns | total_conns |
+-------------+----------------------+-------+------+-------------+------+------------+------------+-------------+
| 1           | 192.168.222.172:3306 | up    | rw   | NULL        | NULL | 100        | 0          | 100         | --172，讀寫
| 2           | 192.168.222.173:3306 | up    | ro   | 426         | NULL | 100        | 0          | 100         | --173，只讀
| 3           | 192.168.222.174:3306 | up    | ro   | 427         | NULL | 100        | 0          | 100         | --174，只讀
+-------------+----------------------+-------+------+-------------+------+------------+------------+-------------+
3 rows in set (0.00 sec)

(admin@192.168.222.175) 14:58:37 [(none)]> cetus;
+--------------------------+---------------------+
| Status                   | Value               |
+--------------------------+---------------------+
| Cetus version            | v1.0.0-44-g5b1bd43  |
| Startup time             | 2018-08-31 14:23:07 |
| Loaded modules           | proxy admin         |
| Idle backend connections | 300                 |
| Used backend connections | 0                   |
| Client connections       | 4                   |
| Query count              | 5                   |
| QPS (1min, 5min, 15min)  | 0.00, 0.01, 0.00    |
| TPS (1min, 5min, 15min)  | 0.00, 0.00, 0.00    |
+--------------------------+---------------------+
9 rows in set (0.00 sec)

(admin@192.168.222.175) 14:58:39 [(none)]>

5、驗證cetus讀寫分離功能

session A

[root@node05 software]# mysql -ugcdb  -piforgot -h192.168.222.175 -P13306
(gcdb@192.168.222.175) 15:18:37 [(none)]> select sleep(10) from ttt.t1;

session B

(admin@192.168.222.175) 15:18:44 [(none)]> select conn_details from backends;select * from backends;
+-------------+----------+------------+------------+-------------+
| backend_ndx | username | idle_conns | used_conns | total_conns |
+-------------+----------+------------+------------+-------------+
| 0           | gcdb     | 100        | 0          | 100         |
| 1           | gcdb     | 100        | 0          | 100         |
| 2           | gcdb     | 99         | 1          | 100         | --讀負載到backend_ndx=3這台
+-------------+----------+------------+------------+-------------+
3 rows in set (0.00 sec)

+-------------+----------------------+-------+------+-------------+------+------------+------------+-------------+
| backend_ndx | address              | state | type | slave delay | uuid | idle_conns | used_conns | total_conns |
+-------------+----------------------+-------+------+-------------+------+------------+------------+-------------+
| 1           | 192.168.222.172:3306 | up    | rw   | NULL        | NULL | 100        | 0          | 100         |
| 2           | 192.168.222.173:3306 | up    | ro   | 403         | NULL | 100        | 0          | 100         |
| 3           | 192.168.222.174:3306 | up    | ro   | 404         | NULL | 99         | 1          | 100         | --讀負載到192.168.222.174
+-------------+----------------------+-------+------+-------------+------+------------+------------+-------------+
3 rows in set (0.00 sec)

(admin@192.168.222.175) 15:18:46 [(none)]>

session A

(gcdb@192.168.222.175) 15:18:37 [(none)]> select sleep(10) from ttt.t1;
+-----------+
| sleep(10) |
+-----------+
|         0 |
|         0 |
|         0 |
|         0 |
+-----------+
4 rows in set (40.00 sec)

(gcdb@192.168.222.175) 15:19:20 [(none)]> 

#再次執行查詢看后端負載到那個服務器
(gcdb@192.168.222.175) 15:19:20 [(none)]> (gcdb@192.168.222.175) 15:19:20 [(none)]> select sleep(10) from ttt.t1;
+-----------+
| sleep(10) |
+-----------+
|         0 |
|         0 |
|         0 |
|         0 |
+-----------+
4 rows in set (40.01 sec)

session B

(admin@192.168.222.175) 15:19:22 [(none)]> select conn_details from backends;select * from backends;
+-------------+----------+------------+------------+-------------+
| backend_ndx | username | idle_conns | used_conns | total_conns |
+-------------+----------+------------+------------+-------------+
| 0           | gcdb     | 100        | 0          | 100         |
| 1           | gcdb     | 99         | 1          | 100         |  --讀負載到backend_ndx=2這台
| 2           | gcdb     | 100        | 0          | 100         |
+-------------+----------+------------+------------+-------------+
3 rows in set (0.00 sec)

+-------------+----------------------+-------+------+-------------+------+------------+------------+-------------+
| backend_ndx | address              | state | type | slave delay | uuid | idle_conns | used_conns | total_conns |
+-------------+----------------------+-------+------+-------------+------+------------+------------+-------------+
| 1           | 192.168.222.172:3306 | up    | rw   | NULL        | NULL | 100        | 0          | 100         |
| 2           | 192.168.222.173:3306 | up    | ro   | 783         | NULL | 99         | 1          | 100         |  --讀負載到192.168.222.173
| 3           | 192.168.222.174:3306 | up    | ro   | 432         | NULL | 100        | 0          | 100         |
+-------------+----------------------+-------+------+-------------+------+------------+------------+-------------+
3 rows in set (0.00 sec)

驗證簡單讀寫

(gcdb@192.168.222.175) 15:23:11 [(none)]> select sleep(5);insert into ttt.t1(nums) values(6),(7),(8),(9),(10);
+----------+
| sleep(5) |
+----------+
|        0 |
+----------+
1 row in set (5.00 sec)

Query OK, 5 rows affected (0.05 sec)
Records: 5  Duplicates: 0  Warnings: 0

(gcdb@192.168.222.175) 15:30:35 [(none)]> select * from ttt.t1;
+----+------+
| id | nums |
+----+------+
|  1 |    1 |
|  2 |    2 |
|  3 |  100 |
|  5 |    5 |
|  6 |    6 |
|  7 |    7 |
|  8 |    8 |
|  9 |    9 |
| 10 |   10 |
+----+------+
9 rows in set (0.01 sec)

(gcdb@192.168.222.175) 15:31:02 [(none)]>

五 MHA和Cetu聯動切換

1、添加mail告警功能

修改配置

[root@node05 software]# vim /etc/mail.rc 
#最后追加以下內容
set from=xxxxxx@163.com  		    #此處設置發件人的信息
set smtp=smtp.163.com     			#此處配置郵件服務地址，因為郵箱是163的，所以此處配置為smtp.163.com
set smtp-auth-user=xxxxxx@163.com    #此處配置發件人郵箱地址
set smtp-auth-password=xxxxxx        #切記此處配置abc是客戶端授權碼，不是發件人郵箱地址密碼
set smtp-auth=login                  #郵件認證方式

啟動sendmail測試

[root@node05 ~]# systemctl start sendmail
[root@node05 ~]# mail -s "status" xxxxxx@139.com < /etc/hosts

修改sendmail腳本

[root@node05 ~]# chmod 755 /usr/local/share/perl5/MHA/sendMail.sh
[root@node05 ~]# ll /usr/local/share/perl5/MHA/sendMail.sh
-rwxr-xr-x 1 root root 593 Sep  3 09:27 /usr/local/share/perl5/MHA/sendMail.sh
[root@node05 ~]# cat /usr/local/share/perl5/MHA/sendMail.sh 
#!/bin/bash
## author : cch
## desc: sendmail
############ variable part ########################
conf=/etc/mha.cnf

if [ $# -ne 1 ];then
   mailsubject="mha--failover--`date +%Y%m%d%H%M`"
else
   mailsubject=$1
fi
############# main #########################
find_flag=`cat $conf|grep -v '^#'|grep "manager_workdir"|awk -F= '{print $2}'|wc -l`
if [  ${find_flag} -eq 1 ];then
   manager_workdir=`cat $conf|grep -v '^#'|grep "manager_workdir"|awk -F= '{print $2}'|sed 's/ //g'`
fi
#修改郵箱地址
mail -s "${mailsubject}"  xxxx@139.com < ${manager_workdir}/sendmail.txt
echo `date` >> ./sendMail.log
[root@node05 ~]#

2、切換主庫

模擬主庫故障

1、node02 會話

[root@node02 scripts]# systemctl stop mysql
[root@node02 scripts]# ps -ef |grep mysql |grep -v grep
root     26319  2289  0 Aug31 pts/0    00:00:00 mysql -uroot -px xxxxx
[root@node02 scripts]#

2、node05 會話

切換前cetus中查看讀寫狀態

session B

(admin@192.168.222.175) 10:04:01 [(none)]> select * from backends;
+-------------+----------------------+-------+------+-------------+------+------------+------------+-------------+
| backend_ndx | address              | state | type | slave delay | uuid | idle_conns | used_conns | total_conns |
+-------------+----------------------+-------+------+-------------+------+------------+------------+-------------+
| 1           | 192.168.222.172:3306 | up    | rw   | NULL        | NULL | 100        | 0          | 100         |
| 2           | 192.168.222.173:3306 | up    | ro   | 423         | NULL | 100        | 0          | 100         |
| 3           | 192.168.222.174:3306 | up    | ro   | 423         | NULL | 100        | 0          | 100         |
+-------------+----------------------+-------+------+-------------+------+------------+------------+-------------+
3 rows in set (0.00 sec)
(admin@192.168.222.175) 10:10:44 [(none)]>

session A

  [root@node05 ~]# tailf /var/log/masterha/mha.log 
  Mon Sep  3 10:06:24 2018 - [warning] Got error on MySQL select ping: 2013 (Lost connection to MySQL server during query)
  Mon Sep  3 10:06:24 2018 - [info] Executing secondary network check script: masterha_secondary_check  -s 192.168.222.173  -s 192.168.222.174  --user=root  --master_host=192.168.222.172  --master_ip=192.168.222.172  --master_port=3306 --master_user=gcdb --master_password=iforgot --ping_type=SELECT
  Mon Sep  3 10:06:24 2018 - [info] Executing SSH check script: exit 0
  Mon Sep  3 10:06:24 2018 - [info] HealthCheck: SSH to 192.168.222.172 is reachable.
  Monitoring server 192.168.222.173 is reachable, Master is not reachable from 192.168.222.173. OK.
  Monitoring server 192.168.222.174 is reachable, Master is not reachable from 192.168.222.174. OK.
  Mon Sep  3 10:06:24 2018 - [info] Master is not reachable from all other monitoring servers. Failover should start.
  Mon Sep  3 10:06:27 2018 - [warning] Got error on MySQL connect: 2003 (Can't connect to MySQL server on '192.168.222.172' (111))
  Mon Sep  3 10:06:27 2018 - [warning] Connection failed 2 time(s)..
  Mon Sep  3 10:06:30 2018 - [warning] Got error on MySQL connect: 2003 (Can't connect to MySQL server on '192.168.222.172' (111))
  Mon Sep  3 10:06:30 2018 - [warning] Connection failed 3 time(s)..
  Mon Sep  3 10:06:33 2018 - [warning] Got error on MySQL connect: 2003 (Can't connect to MySQL server on '192.168.222.172' (111))
  Mon Sep  3 10:06:33 2018 - [warning] Connection failed 4 time(s)..
  Mon Sep  3 10:06:33 2018 - [warning] Master is not reachable from health checker!
  Mon Sep  3 10:06:33 2018 - [warning] Master 192.168.222.172(192.168.222.172:3306) is not reachable!
  Mon Sep  3 10:06:33 2018 - [warning] SSH is reachable.
  Mon Sep  3 10:06:33 2018 - [info] Connecting to a master server failed. Reading configuration file /etc/masterha_default.cnf and /etc/mha.cnf again, and trying to connect to all servers to check server status..
  Mon Sep  3 10:06:33 2018 - [info] Reading default configuration from /etc/masterha_default.cnf..
  Mon Sep  3 10:06:33 2018 - [info] Reading application default configuration from /etc/mha.cnf..
  Mon Sep  3 10:06:33 2018 - [info] Reading server configuration from /etc/mha.cnf..
  Mon Sep  3 10:06:34 2018 - [info] GTID failover mode = 1
  Mon Sep  3 10:06:34 2018 - [info] Dead Servers:
  Mon Sep  3 10:06:34 2018 - [info]   192.168.222.172(192.168.222.172:3306)
  Mon Sep  3 10:06:34 2018 - [info] Alive Servers:
  Mon Sep  3 10:06:34 2018 - [info]   192.168.222.173(192.168.222.173:3306)
  Mon Sep  3 10:06:34 2018 - [info]   192.168.222.174(192.168.222.174:3306)
  Mon Sep  3 10:06:34 2018 - [info] Alive Slaves:
  Mon Sep  3 10:06:34 2018 - [info]   192.168.222.173(192.168.222.173:3306)  Version=5.7.18-log (oldest major version between slaves) log-bin:enabled
  Mon Sep  3 10:06:34 2018 - [info]     GTID ON
  Mon Sep  3 10:06:34 2018 - [info]     Replicating from 192.168.222.172(192.168.222.172:3306)
  Mon Sep  3 10:06:34 2018 - [info]   192.168.222.174(192.168.222.174:3306)  Version=5.7.18-log (oldest major version between slaves) log-bin:enabled
  Mon Sep  3 10:06:34 2018 - [info]     GTID ON
  Mon Sep  3 10:06:34 2018 - [info]     Replicating from 192.168.222.172(192.168.222.172:3306)
  Mon Sep  3 10:06:34 2018 - [info] Checking slave configurations..
  Mon Sep  3 10:06:34 2018 - [info]  read_only=1 is not set on slave 192.168.222.173(192.168.222.173:3306).
  Mon Sep  3 10:06:34 2018 - [info]  read_only=1 is not set on slave 192.168.222.174(192.168.222.174:3306).
  Mon Sep  3 10:06:34 2018 - [info] Checking replication filtering settings..
  Mon Sep  3 10:06:34 2018 - [info]  Replication filtering check ok.
  Mon Sep  3 10:06:34 2018 - [info] Master is down!
  Mon Sep  3 10:06:34 2018 - [info] Terminating monitoring script.
  Mon Sep  3 10:06:34 2018 - [info] Got exit code 20 (Master dead).
  Mon Sep  3 10:06:34 2018 - [info] MHA::MasterFailover version 0.56.
  Mon Sep  3 10:06:34 2018 - [info] g_interactive: 0
  Mon Sep  3 10:06:34 2018 - [info] g_logfile: /var/log/masterha/mha.log
  Mon Sep  3 10:06:34 2018 - [info] g_global_config_file:  /etc/masterha_default.cnf
  Mon Sep  3 10:06:34 2018 - [info] g_config_file:  /etc/mha.cnf
  Mon Sep  3 10:06:34 2018 - [info] g_proxy_config_file: /etc/cetus.cnf
  Mon Sep  3 10:06:34 2018 - [info] Starting master failover.
  Mon Sep  3 10:06:34 2018 - [info] 
  Mon Sep  3 10:06:34 2018 - [info] * Phase 1: Configuration Check Phase..
  Mon Sep  3 10:06:34 2018 - [info] 
  Mon Sep  3 10:06:34 2018 - [info] -------------ManagerUtil::check_node_version-----------
  Mon Sep  3 10:06:35 2018 - [info] GTID failover mode = 1
  Mon Sep  3 10:06:35 2018 - [info] Dead Servers:
  Mon Sep  3 10:06:35 2018 - [info]   192.168.222.172(192.168.222.172:3306)
  Mon Sep  3 10:06:35 2018 - [info] Checking master reachability via MySQL(double check)...
  Mon Sep  3 10:06:35 2018 - [info]  ok.
  Mon Sep  3 10:06:35 2018 - [info] Dead server: 192.168.222.172(192.168.222.172:3306)
  Mon Sep  3 10:06:35 2018 - [info] deadaddr: 192.168.222.172:3306
  Mon Sep  3 10:06:35 2018 - [info] Reading default configuration from /etc/masterha_default.cnf..
  Mon Sep  3 10:06:35 2018 - [info] Reading application default configuration from /etc/mha.cnf..
  Mon Sep  3 10:06:35 2018 - [info] Reading server configuration from /etc/mha.cnf..
  Mon Sep  3 10:06:35 2018 - [info] [ProxyManager::setproxy] proxy_conf: /etc/cetus.cnf 
  Mon Sep  3 10:06:35 2018 - [info] type   : failover
  Mon Sep  3 10:06:35 2018 - [info] status : down
  Mon Sep  3 10:06:35 2018 - [info] addr   : 192.168.222.172:3306
  Mon Sep  3 10:06:35 2018 - [info] dbtype : ro
  Mon Sep  3 10:06:35 2018 - [info] Reading default configuration from /etc/masterha_default.cnf..
  Mon Sep  3 10:06:35 2018 - [info] Reading application default configuration from /etc/mha.cnf..
  Mon Sep  3 10:06:35 2018 - [info] Reading server configuration from /etc/mha.cnf..
  Mon Sep  3 10:06:35 2018 - [info] [ProxyManager::setdb] proxy_conf: /etc/cetus.cnf 
  Mon Sep  3 10:06:35 2018 - [info] ** 192.168.222.175:23306 started, pid: 5144
  Mon Sep  3 10:06:35 2018 - [info] exec command: /usr/bin/mysql -h192.168.222.175 -uadmin -P23306 -padmin  -e " update backends set state='down' , type='ro' where address='192.168.222.172:3306';"
  Mon Sep  3 10:06:35 2018 - [info] ......setdb.....
  Mon Sep  3 10:06:35 2018 - [info] Alive Servers:
  Mon Sep  3 10:06:35 2018 - [info]   192.168.222.173(192.168.222.173:3306)
  Mon Sep  3 10:06:35 2018 - [info]   192.168.222.174(192.168.222.174:3306)
  Mon Sep  3 10:06:35 2018 - [info] Alive Slaves:
  Mon Sep  3 10:06:35 2018 - [info]   192.168.222.173(192.168.222.173:3306)  Version=5.7.18-log (oldest major version between slaves) log-bin:enabled
  Mon Sep  3 10:06:35 2018 - [info]     GTID ON
  Mon Sep  3 10:06:35 2018 - [info]     Replicating from 192.168.222.172(192.168.222.172:3306)
  Mon Sep  3 10:06:35 2018 - [info]   192.168.222.174(192.168.222.174:3306)  Version=5.7.18-log (oldest major version between slaves) log-bin:enabled
  Mon Sep  3 10:06:35 2018 - [info]     GTID ON
  Mon Sep  3 10:06:35 2018 - [info]     Replicating from 192.168.222.172(192.168.222.172:3306)
  Mon Sep  3 10:06:35 2018 - [info] Starting GTID based failover.
  Mon Sep  3 10:06:35 2018 - [info] 
  Mon Sep  3 10:06:35 2018 - [info] ** Phase 1: Configuration Check Phase completed.
  Mon Sep  3 10:06:35 2018 - [info] 
  Mon Sep  3 10:06:35 2018 - [info] * Phase 2: Dead Master Shutdown Phase..
  Mon Sep  3 10:06:35 2018 - [info] 
  Mon Sep  3 10:06:35 2018 - [info] Forcing shutdown so that applications never connect to the current master..
  Mon Sep  3 10:06:35 2018 - [info] Executing master IP deactivation script:
  Mon Sep  3 10:06:35 2018 - [info]   /usr/local/bin/master_ip_failover --orig_master_host=192.168.222.172 --orig_master_ip=192.168.222.172 --orig_master_port=3306 --command=stopssh --ssh_user=root  
   
  IN SCRIPT TEST====/sbin/ifconfig ens224:1 down==/sbin/ifconfig ens224:1 192.168.222.99/24===
  
  Disabling the VIP on old master: 192.168.222.172 
  Mon Sep  3 10:06:35 2018 - [info]  done.
  Mon Sep  3 10:06:35 2018 - [warning] shutdown_script is not set. Skipping explicit shutting down of the dead master.
  Mon Sep  3 10:06:35 2018 - [info] * Phase 2: Dead Master Shutdown Phase completed.
  Mon Sep  3 10:06:35 2018 - [info] 
  Mon Sep  3 10:06:35 2018 - [info] * Phase 3: Master Recovery Phase..
  Mon Sep  3 10:06:35 2018 - [info] 
  Mon Sep  3 10:06:35 2018 - [info] * Phase 3.1: Getting Latest Slaves Phase..
  Mon Sep  3 10:06:35 2018 - [info] 
  Mon Sep  3 10:06:35 2018 - [info] The latest binary log file/position on all slaves is binlog.000006:3477689
  Mon Sep  3 10:06:35 2018 - [info] Retrieved Gtid Set: d3f2e80d-acf1-11e8-869e-005056abaf95:8953-15333
  Mon Sep  3 10:06:35 2018 - [info] Latest slaves (Slaves that received relay log files to the latest):
  Mon Sep  3 10:06:35 2018 - [info]   192.168.222.173(192.168.222.173:3306)  Version=5.7.18-log (oldest major version between slaves) log-bin:enabled
  Mon Sep  3 10:06:35 2018 - [info]     GTID ON
  Mon Sep  3 10:06:35 2018 - [info]     Replicating from 192.168.222.172(192.168.222.172:3306)
  Mon Sep  3 10:06:35 2018 - [info]   192.168.222.174(192.168.222.174:3306)  Version=5.7.18-log (oldest major version between slaves) log-bin:enabled
  Mon Sep  3 10:06:35 2018 - [info]     GTID ON
  Mon Sep  3 10:06:35 2018 - [info]     Replicating from 192.168.222.172(192.168.222.172:3306)
  Mon Sep  3 10:06:35 2018 - [info] The oldest binary log file/position on all slaves is binlog.000006:3477689
  Mon Sep  3 10:06:35 2018 - [info] Retrieved Gtid Set: d3f2e80d-acf1-11e8-869e-005056abaf95:8953-15333
  Mon Sep  3 10:06:35 2018 - [info] Oldest slaves:
  Mon Sep  3 10:06:35 2018 - [info]   192.168.222.173(192.168.222.173:3306)  Version=5.7.18-log (oldest major version between slaves) log-bin:enabled
  Mon Sep  3 10:06:35 2018 - [info]     GTID ON
  Mon Sep  3 10:06:35 2018 - [info]     Replicating from 192.168.222.172(192.168.222.172:3306)
  Mon Sep  3 10:06:35 2018 - [info]   192.168.222.174(192.168.222.174:3306)  Version=5.7.18-log (oldest major version between slaves) log-bin:enabled
  Mon Sep  3 10:06:35 2018 - [info]     GTID ON
  Mon Sep  3 10:06:35 2018 - [info]     Replicating from 192.168.222.172(192.168.222.172:3306)
  Mon Sep  3 10:06:35 2018 - [info] 
  Mon Sep  3 10:06:35 2018 - [info] * Phase 3.3: Determining New Master Phase..
  Mon Sep  3 10:06:35 2018 - [info] 
  Mon Sep  3 10:06:35 2018 - [info] Searching new master from slaves..
  Mon Sep  3 10:06:35 2018 - [info]  Candidate masters from the configuration file:
  Mon Sep  3 10:06:35 2018 - [info]  Non-candidate masters:
  Mon Sep  3 10:06:35 2018 - [info] New master is 192.168.222.173(192.168.222.173:3306)
  Mon Sep  3 10:06:35 2018 - [info] Starting master failover..
  Mon Sep  3 10:06:35 2018 - [info] 
  From:
  192.168.222.172(192.168.222.172:3306) (current master)
   +--192.168.222.173(192.168.222.173:3306)
   +--192.168.222.174(192.168.222.174:3306)
  
  To:
  192.168.222.173(192.168.222.173:3306) (new master)
   +--192.168.222.174(192.168.222.174:3306)
  Mon Sep  3 10:06:35 2018 - [info] 
  Mon Sep  3 10:06:35 2018 - [info] * Phase 3.3: New Master Recovery Phase..
  Mon Sep  3 10:06:35 2018 - [info] 
  Mon Sep  3 10:06:35 2018 - [info]  Waiting all logs to be applied.. 
  Mon Sep  3 10:06:35 2018 - [info]   done.
  Mon Sep  3 10:06:35 2018 - [info] Getting new master's binlog name and position..
  Mon Sep  3 10:06:35 2018 - [info]  binlog.000003:2361204
  Mon Sep  3 10:06:35 2018 - [info]  All other slaves should start replication from here. Statement should be: CHANGE MASTER TO MASTER_HOST='192.168.222.173', MASTER_PORT=3306, MASTER_AUTO_POSITION=1, MASTER_USER='repl', MASTER_PASSWORD='xxx';
  Mon Sep  3 10:06:35 2018 - [info] Master Recovery succeeded. File:Pos:Exec_Gtid_Set: binlog.000003, 2361204, 75ee239c-aaa5-11e8-84db-005056aba7b5:1-2795,
  d3f2e80d-acf1-11e8-869e-005056abaf95:1-15333
  Mon Sep  3 10:06:35 2018 - [info] Executing master IP activate script:
  Mon Sep  3 10:06:35 2018 - [info]   /usr/local/bin/master_ip_failover --command=start --ssh_user=root --orig_master_host=192.168.222.172 --orig_master_ip=192.168.222.172 --orig_master_port=3306 --new_master_host=192.168.222.173 --new_master_ip=192.168.222.173 --new_master_port=3306 --new_master_user='gcdb' --new_master_password='iforgot'  
  Unknown option: new_master_user
  Unknown option: new_master_password
    
  IN SCRIPT TEST====/sbin/ifconfig ens224:1 down==/sbin/ifconfig ens224:1 192.168.222.99/24===
  
  Enabling the VIP - 192.168.222.99/24 on the new master - 192.168.222.173 
  Mon Sep  3 10:06:36 2018 - [info]  OK.
  Mon Sep  3 10:06:36 2018 - [info] ** Finished master recovery successfully.
  Mon Sep  3 10:06:36 2018 - [info] MHA::Server=HASH(0x274df80)->get_hostinfo()
  Mon Sep  3 10:06:36 2018 - [info] before  newmaster_addr: 192.168.222.173(192.168.222.173:3306)
  Mon Sep  3 10:06:36 2018 - [info] Reading default configuration from /etc/masterha_default.cnf..
  Mon Sep  3 10:06:36 2018 - [info] Reading application default configuration from /etc/mha.cnf..
  Mon Sep  3 10:06:36 2018 - [info] Reading server configuration from /etc/mha.cnf..
  Mon Sep  3 10:06:36 2018 - [info] [ProxyManager::setproxy] proxy_conf: /etc/cetus.cnf 
  Mon Sep  3 10:06:36 2018 - [info] type   : setmaster
  Mon Sep  3 10:06:36 2018 - [info] status : up
  Mon Sep  3 10:06:36 2018 - [info] addr   : 192.168.222.173:3306
  Mon Sep  3 10:06:36 2018 - [info] dbtype : rw
  Mon Sep  3 10:06:36 2018 - [info] Reading default configuration from /etc/masterha_default.cnf..
  Mon Sep  3 10:06:36 2018 - [info] Reading application default configuration from /etc/mha.cnf..
  Mon Sep  3 10:06:36 2018 - [info] Reading server configuration from /etc/mha.cnf..
  Mon Sep  3 10:06:36 2018 - [info] [ProxyManager::setdb] proxy_conf: /etc/cetus.cnf 
  Mon Sep  3 10:06:36 2018 - [info] ** 192.168.222.175:23306 started, pid: 5157
  Mon Sep  3 10:06:36 2018 - [info] exec command: /usr/bin/mysql -h192.168.222.175 -uadmin -P23306 -padmin  -e " update backends set state='up' , type='rw' where address='192.168.222.173:3306';"
  Mon Sep  3 10:06:36 2018 - [info] ......setdb.....
  Mon Sep  3 10:06:36 2018 - [info] * Phase 3: Master Recovery Phase completed.
  Mon Sep  3 10:06:36 2018 - [info] 
  Mon Sep  3 10:06:36 2018 - [info] * Phase 4: Slaves Recovery Phase..
  Mon Sep  3 10:06:36 2018 - [info] 
  Mon Sep  3 10:06:36 2018 - [info] 
  Mon Sep  3 10:06:36 2018 - [info] * Phase 4.1: Starting Slaves in parallel..
  Mon Sep  3 10:06:36 2018 - [info] 
  Mon Sep  3 10:06:36 2018 - [info] -- Slave recovery on host 192.168.222.174(192.168.222.174:3306) started, pid: 5159. Check tmp log /var/log/masterha/192.168.222.174_3306_20180903100634.log if it takes time..
  Mon Sep  3 10:06:37 2018 - [info] 
  Mon Sep  3 10:06:37 2018 - [info] Log messages from 192.168.222.174 ...
  Mon Sep  3 10:06:37 2018 - [info] 
  Mon Sep  3 10:06:36 2018 - [info]  Resetting slave 192.168.222.174(192.168.222.174:3306) and starting replication from the new master 192.168.222.173(192.168.222.173:3306)..
  Mon Sep  3 10:06:36 2018 - [info]  Executed CHANGE MASTER.
  Mon Sep  3 10:06:36 2018 - [info]  Slave started.
  Mon Sep  3 10:06:36 2018 - [info]  gtid_wait(75ee239c-aaa5-11e8-84db-005056aba7b5:1-2795,
  d3f2e80d-acf1-11e8-869e-005056abaf95:1-15333) completed on 192.168.222.174(192.168.222.174:3306). Executed 0 events.
  Mon Sep  3 10:06:37 2018 - [info] End of log messages from 192.168.222.174.
  Mon Sep  3 10:06:37 2018 - [info] -- Slave on host 192.168.222.174(192.168.222.174:3306) started.
  Mon Sep  3 10:06:37 2018 - [info] All new slave servers recovered successfully.
  Mon Sep  3 10:06:37 2018 - [info] 
  Mon Sep  3 10:06:37 2018 - [info] * Phase 5: New master cleanup phase..
  Mon Sep  3 10:06:37 2018 - [info] 
  Mon Sep  3 10:06:37 2018 - [info] Resetting slave info on the new master..
  Mon Sep  3 10:06:37 2018 - [info]  192.168.222.173: Resetting slave info succeeded.
  Mon Sep  3 10:06:37 2018 - [info] Master failover to 192.168.222.173(192.168.222.173:3306) completed successfully.
  Mon Sep  3 10:06:37 2018 - [info] Deleted server1 entry from /etc/mha.cnf .
  Mon Sep  3 10:06:37 2018 - [info] 
  
  ----- Failover Report -----
  
  mha: MySQL Master failover 192.168.222.172(192.168.222.172:3306) to 192.168.222.173(192.168.222.173:3306) succeeded
  
  Master 192.168.222.172(192.168.222.172:3306) is down.
  
  Check MHA Manager logs at node05.test.com:/var/log/masterha/mha.log for details.
  
  Started automated(non-interactive) failover.
  Invalidated master IP address on 192.168.222.172(192.168.222.172:3306)
  Selected 192.168.222.173(192.168.222.173:3306) as a new master.
  192.168.222.173(192.168.222.173:3306): OK: Applying all logs succeeded.
  192.168.222.173(192.168.222.173:3306): OK: Activated master IP address.
  192.168.222.174(192.168.222.174:3306): OK: Slave started, replicating from 192.168.222.173(192.168.222.173:3306)
  192.168.222.173(192.168.222.173:3306): Resetting slave info succeeded.
  Master failover to 192.168.222.173(192.168.222.173:3306) completed successfully.
  Mon Sep  3 10:06:37 2018 - [info] g_workdir: /var/log/masterha
  Mon Sep  3 10:06:37 2018 - [info] sendmail:  /var/log/masterha/sendmail.txt 
  Mon Sep  3 10:06:37 2018 - [info] Sending mail..

session B
切換后Cetus中查看讀寫狀態

(admin@192.168.222.175) 10:04:01 [(none)]> select * from backends;
+-------------+----------------------+-------+------+-------------+------+------------+------------+-------------+
| backend_ndx | address              | state | type | slave delay | uuid | idle_conns | used_conns | total_conns |
+-------------+----------------------+-------+------+-------------+------+------------+------------+-------------+
| 1           | 192.168.222.172:3306 | down  | ro   | 15177       | NULL | 0          | 0          | 0           | --172，狀態已經down
| 2           | 192.168.222.173:3306 | up    | rw   | NULL        | NULL | 100        | 0          | 100         | --173，讀寫
| 3           | 192.168.222.174:3306 | up    | ro   | 402         | NULL | 100        | 0          | 100         | 
+-------------+----------------------+-------+------+-------------+------+------------+------------+-------------+
3 rows in set (0.00 sec)

(admin@192.168.222.175) 10:10:44 [(none)]>

session A
切換后mha.cnf 配置里面原主庫會被清除掉

[root@node05 software]# cat  /etc/mha.cnf 
[server default]
manager_log=/var/log/masterha/mha.log
manager_workdir=/var/log/masterha

[server2]
hostname=192.168.222.173
master_binlog_dir=/r2/mysqldata
port=3306

[server3]
hostname=192.168.222.174
master_binlog_dir=/r2/mysqldata
port=3306
[root@node05 software]# ll /var/log/masterha/mha.failover.complete    --mha成功執行切換，會生成該文件，如果重新構建MHA把這個文件刪除掉
-rw-r--r-- 1 root root 0 Sep  3 10:06 /var/log/masterha/mha.failover.complete

3 、郵件告警

六總結

1 、目錄的樹形結構

[root@node05 software]# tree -fp  /usr/local/share/perl5/MHA/
/usr/local/share/perl5/MHA
├── [-r--r--r--]  /usr/local/share/perl5/MHA/BinlogHeaderParser.pm
├── [-r--r--r--]  /usr/local/share/perl5/MHA/BinlogManager.pm
├── [-r--r--r--]  /usr/local/share/perl5/MHA/BinlogPosFinderElp.pm
├── [-r--r--r--]  /usr/local/share/perl5/MHA/BinlogPosFinder.pm
├── [-r--r--r--]  /usr/local/share/perl5/MHA/BinlogPosFinderXid.pm
├── [-r--r--r--]  /usr/local/share/perl5/MHA/BinlogPosFindManager.pm
├── [-rw-r--r--]  /usr/local/share/perl5/MHA/cetus.cnf
├── [-r--r--r--]  /usr/local/share/perl5/MHA/Config.pm
├── [-r--r--r--]  /usr/local/share/perl5/MHA/DBHelper.pm
├── [-r--r--r--]  /usr/local/share/perl5/MHA/FileStatus.pm
├── [-r--r--r--]  /usr/local/share/perl5/MHA/HealthCheck.pm
├── [-r--r--r--]  /usr/local/share/perl5/MHA/ManagerAdmin.pm
├── [-r--r--r--]  /usr/local/share/perl5/MHA/ManagerAdminWrapper.pm
├── [-r--r--r--]  /usr/local/share/perl5/MHA/ManagerConst.pm
├── [-r--r--r--]  /usr/local/share/perl5/MHA/ManagerUtil.pm
├── [-r--r--r--]  /usr/local/share/perl5/MHA/MasterFailover.pm
├── [-rw-r--r--]  /usr/local/share/perl5/MHA/masterha_secondary_check
├── [-r--r--r--]  /usr/local/share/perl5/MHA/MasterMonitor.pm
├── [-r--r--r--]  /usr/local/share/perl5/MHA/MasterRotate.pm
├── [-r--r--r--]  /usr/local/share/perl5/MHA/NodeConst.pm
├── [-r--r--r--]  /usr/local/share/perl5/MHA/NodeUtil.pm
├── [-rw-r--r--]  /usr/local/share/perl5/MHA/ProxyManager.pm
├── [-rw-r--r--]  /usr/local/share/perl5/MHA/sample.cnf
├── [-rwxr-xr-x]  /usr/local/share/perl5/MHA/sendMail.sh
├── [-rw-r--r--]  /usr/local/share/perl5/MHA/sendmail.txt
├── [-r--r--r--]  /usr/local/share/perl5/MHA/ServerManager.pm
├── [-r--r--r--]  /usr/local/share/perl5/MHA/Server.pm
├── [-r--r--r--]  /usr/local/share/perl5/MHA/SlaveUtil.pm
└── [-r--r--r--]  /usr/local/share/perl5/MHA/SSHCheck.pm
0 directories, 29 files
[root@node05 software]# tree -fp  /var/log/masterha/
/var/log/masterha
├── [-rw-r--r--]  /var/log/masterha/mha.failover.complete
├── [-rw-r--r--]  /var/log/masterha/mha.log
└── [-rw-r--r--]  /var/log/masterha/sendmail.txt
0 directories, 3 files
[root@node05 software]#  tree -fp  /etc/ |grep .cnf
├── [-rw-r--r--]  /etc/cetus.cnf
├── [-rw-r--r--]  /etc/masterha_default.cnf
├── [-rw-r--r--]  /etc/mha.cnf
[root@node05 software]# tree -fp /usr/local/cetus/
[root@node05 software]# tree -fp /usr/local/cetus/conf/
/usr/local/cetus/conf
├── [-rw-rw----]  /usr/local/cetus/conf/proxy.conf
├── [-rw-rw----]  /usr/local/cetus/conf/proxy.conf.example
├── [-rw-rw----]  /usr/local/cetus/conf/shard.conf.example
├── [-rw-rw----]  /usr/local/cetus/conf/sharding.json.example
├── [-rw-rw----]  /usr/local/cetus/conf/users.json
├── [-rw-rw----]  /usr/local/cetus/conf/users.json.example
└── [-rw-rw----]  /usr/local/cetus/conf/variables.json.example

2、mysql客戶端問題

[root@node05 software]# grep -w /usr/bin/mysql  /usr/local/bin/*
/usr/local/bin/masterha_secondary_check:        . "/usr/bin/mysql -u$master_user -p$master_password -h$master_host -P$master_port "
See online reference (http://code.google.com/p/mysql-master-ha/wiki/Parameters#secondary_check_script) for details.
See online reference (http://code.google.com/p/mysql-master-ha/wiki/Parameters#secondary_check_script) for details.
[root@node05 software]# ln -s /usr/local/mysql/bin/mysql /usr/bin/mysql   --做鏈接來解決
[root@node05 software]# which mysql
/usr/bin/mysql

3、master_ip_failover failed

Fri Aug 31 13:47:40 2018 - [info]   /usr/local/bin/master_ip_failover --command=status --ssh_user=root --orig_master_host=192.168.222.172 --orig_master_ip=192.168.222.172 --orig_master_port=3306 
Bareword "FIXME_xxx" not allowed while "strict subs" in use at /usr/local/bin/master_ip_failover line 93.
Execution of /usr/local/bin/master_ip_failover aborted due to compilation errors.
Fri Aug 31 13:47:40 2018 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln267]  Failed to get master_ip_failover_script status with return code 255:0.
Fri Aug 31 13:47:40 2018 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln485] Error happened on checking configurations.  at /usr/local/bin/masterha_check_repl line 48.
Fri Aug 31 13:47:40 2018 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln585] Error happened on monitoring servers.
Fri Aug 31 13:47:40 2018 - [info] Got exit code 1 (Not master dead).

master_ip_failover里面添加切換vip

#添加切換vip
my $vip = '192.168.222.99/24';
my $key = '1';
my $ssh_start_vip = "/sbin/ifconfig ens224:$key $vip";
my $ssh_stop_vip = "/sbin/ifconfig ens224:$key down";
$ssh_user = "root";

4、多網卡cetus配置問題

[root@node05 software]# cat /etc/cetus.cnf 
middle_ipport=192.168.222.175:23306  --要配置同一網段，且與proxy.conf文件里面admin-address的端口和ip相對應

5、重新構建MHA

刪除mha.failover.complete文件
mha.cnf 把主機信息重新添加
masterha_default.cnf里面修改 masterha_secondary_check -s 連接測試

免責聲明！

本站轉載的文章為個人學習借鑒使用，本站對版權不負任何法律責任。如果侵犯了您的隱私權益，請聯系本站郵箱yoyou2525@163.com刪除。

猜您在找 MySQL中間件之ProxySQL(2)：初試讀寫分離 mycat中間件進行MySQL的數據讀寫分離 mysql讀寫分離——中間件ProxySQL的簡介與配置 Mycat數據庫中間件對Mysql讀寫分離和分庫分表配置 .netcore實現一個讀寫分離的數據庫訪問中間件 laravel 中間件及路由分離 MaxScale：實現MySQL讀寫分離與負載均衡的中間件利器中間件,前后端分離思想分布式數據層中間件：實現分庫分表+動態數據源+讀寫分離

030：Cetus中間件和MHA讀寫分離

一、主機環境

二、搭建主從

1、配置主從

2、配置 hosts

3、配置免秘鑰

三、安裝MHA和Cetus

1、下載包和安裝依賴包

2、安裝MHA Node

3、安裝MHA manager

4、安裝Cetus

1.安裝說明

2.安裝實施

5、cetus替換mha部分文件

四、MHA和cetus配置

1、創建和修改配置文件

2、檢查 SSH/復制/MHA Manager 的配置

3、啟動MHA manager

4、啟動cetus

5、驗證cetus讀寫分離功能

五 MHA和Cetu聯動切換

1、添加mail告警功能

2、切換主庫

1、node02 會話

2、node05 會話

3 、郵件告警

六 總結

1 、目錄的樹形結構

2、mysql客戶端問題

3、master_ip_failover failed

4、多網卡cetus配置問題

5、重新構建MHA

免責聲明！

六總結