本文主要在MHA 切換日志的角度分析MHA切換的過。MHA故障切換過程如下圖所示
第一部分：開啟MHA 監控
通過分析日志，得到以下步驟：
1、讀取MHA manager 節點的配置文件，並檢查配置文件中參數設置的正確性。
Sat Jun 22 20:16:29 2019 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping. Sat Jun 22 20:16:29 2019 - [info] Reading application default configuration from /etc/masterha/app1.conf.. Sat Jun 22 20:16:29 2019 - [info] Reading server configuration from /etc/masterha/app1.conf.. #找到manager的 配置文件 Creating /tmp if not exists.. ok. Checking output directory is accessible or not.. ok. Binlog found at /usr/local/mysql/data, up to mysql_bin.000024 #根據配置文件，找到主庫binlog Sat Jun 22 20:18:10 2019 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping. Sat Jun 22 20:18:10 2019 - [info] Reading application default configuration from /etc/masterha/app1.conf.. Sat Jun 22 20:18:10 2019 - [info] Reading server configuration from /etc/masterha/app1.conf.. #通過配置文件找到當前的主從架構 Sat Jun 22 20:16:30 2019 - [info] 172.16.13.15(172.16.13.15:3306) Version=5.7.16-log (oldest major version between slaves) log-bin:enabled Sat Jun 22 20:16:30 2019 - [info] Replicating from 172.16.15.3(172.16.15.3:3306) Sat Jun 22 20:16:30 2019 - [info] Primary candidate for the new Master (candidate_master is set) Sat Jun 22 20:16:30 2019 - [info] 172.16.15.2(172.16.15.2:3306) Version=5.7.16-log (oldest major version between slaves) log-bin:enabled Sat Jun 22 20:16:30 2019 - [info] Replicating from 172.16.15.3(172.16.15.3:3306) Sat Jun 22 20:16:30 2019 - [info] Current Alive Master: 172.16.15.3(172.16.15.3:3306) #找到當前的主庫
2、檢查從庫的配置，判斷 slave 是否對某些數據庫或表進行過濾
Sat Jun 22 20:16:30 2019 - [info] Checking slave configurations.. Sat Jun 22 20:16:30 2019 - [info] Checking replication filtering(過濾) settings.. Sat Jun 22 20:16:30 2019 - [info] binlog_do_db= , binlog_ignore_db= Sat Jun 22 20:16:30 2019 - [info] Replication filtering check ok.
3、檢查是否支持GTID主從，MHA Node版本，SSH連接測試，測試主庫上的恢復腳本
Sat Jun 22 20:16:30 2019 - [info] GTID (with auto-pos) is not supported Sat Jun 22 20:16:30 2019 - [info] Starting SSH connection tests.. Sat Jun 22 20:16:32 2019 - [info] All SSH connection tests passed successfully. Sat Jun 22 20:16:32 2019 - [info] Checking MHA Node version.. Sat Jun 22 20:16:32 2019 - [info] Version check ok. Sat Jun 22 20:16:32 2019 - [info] Checking SSH publickey authentication settings on the current master.. Sat Jun 22 20:16:32 2019 - [info] HealthCheck: SSH to 172.16.15.3 is reachable. Sat Jun 22 20:16:32 2019 - [info] Master MHA Node version is 0.56. Sat Jun 22 20:16:32 2019 - [info] Checking recovery script configurations on 172.16.15.3(172.16.15.3:3306).. Sat Jun 22 20:16:32 2019 - [info] Executing command: save_binary_logs --command=test --start_pos=4 --binlog_dir=/usr/local/mysql/data --output_file=/tmp/save_binary_logs_test --manager_version=0.56 --start_file=mysql_bin.000024
4、分別通過SSH連接到主庫和從庫，檢查binlog和relay log，測試應用差異日志的腳本是否正常，測試MySQL連接和權限
Sat Jun 22 20:16:32 2019 - [info] Connecting to root@172.16.15.3(172.16.15.3:22).. Creating /tmp if not exists.. ok. Checking output directory is accessible or not.. ok. Binlog found at /usr/local/mysql/data, up to mysql_bin.000024 Sat Jun 22 20:16:33 2019 - [info] Binlog setting check done. Sat Jun 22 20:16:33 2019 - [info] Checking SSH publickey authentication and checking recovery script configurations on all alive slave servers.. Sat Jun 22 20:16:33 2019 - [info] Executing command : apply_diff_relay_logs --command=test --slave_user='root' --slave_host=172.16.13.15 --slave_ip=172.16.13.15 --slave_port=3306 --workdir=/tmp --target_version=5.7.16-log --manager_version=0.56 --relay_log_info=/usr/local/mysql/data/relay-log.info --relay_dir=/usr/local/mysql/data/ --slave_pass=xxx Sat Jun 22 20:16:33 2019 - [info] Connecting to root@172.16.13.15(172.16.13.15:22).. Checking slave recovery environment settings.. Opening /usr/local/mysql/data/relay-log.info ... ok. Relay log found at /usr/local/mysql/data, up to mysqlserver-relay-bin.000010 Temporary relay log file is /usr/local/mysql/data/mysqlserver-relay-bin.000010 Testing mysql connection and privileges.. done. Testing mysqlbinlog output.. done. Cleaning up test file(s).. done. Sat Jun 22 20:16:33 2019 - [info] Executing command : apply_diff_relay_logs --command=test --slave_user='root' --slave_host=172.16.15.2 --slave_ip=172.16.15.2 --slave_port=3306 --workdir=/tmp --target_version=5.7.16-log --manager_version=0.56 --relay_log_info=/usr/local/mysql/data/relay-log.info --relay_dir=/usr/local/mysql/data/ --slave_pass=xxx Sat Jun 22 20:16:33 2019 - [info] Connecting to root@172.16.15.2(172.16.15.2:22).. Checking slave recovery environment settings.. Opening /usr/local/mysql/data/relay-log.info ... ok. Relay log found at /usr/local/mysql/data, up to A2-relay-bin.000011 Temporary relay log file is /usr/local/mysql/data/A2-relay-bin.000011 Testing mysql connection and privileges.. done. Testing mysqlbinlog output.. done. Cleaning up test file(s).. done. Sat Jun 22 20:16:33 2019 - [info] Slaves settings check done.
5、得到主從架構信息，檢查 master_ip_failover_script腳本是否正常。開始監控主庫master 的運行情況
Sat Jun 22 20:16:33 2019 - [info] 172.16.15.3(172.16.15.3:3306) (current master) +--172.16.13.15(172.16.13.15:3306) +--172.16.15.2(172.16.15.2:3306) Sat Jun 22 20:16:33 2019 - [info] Checking master_ip_failover_script status: Sat Jun 22 20:16:33 2019 - [info] /var/log/masterha/scripts/master_ip_failover --command=status --ssh_user=root --orig_master_host=172.16.15.3 --orig_master_ip=172.16.15.3 --orig_master_port=3306 Checking the Status of the script.. OK Sat Jun 22 20:16:33 2019 - [info] OK. Sat Jun 22 20:16:33 2019 - [warning] shutdown_script is not defined. Sat Jun 22 20:16:33 2019 - [info] Set master ping interval 1 seconds. Sat Jun 22 20:16:33 2019 - [warning] secondary_check_script is not defined. It is highly recommended setting it to check master reachability from two or more routes. Sat Jun 22 20:16:33 2019 - [info] Starting ping health check on 172.16.15.3(172.16.15.3:3306).. Sat Jun 22 20:16:33 2019 - [info] Ping(SELECT) succeeded, waiting until MySQL doesn't respond..
第二部分：故障切換分析
1、發現主庫故障，測試和主庫的SSH連接，連續多次確認主庫狀態。根據配置文件信息，檢查所有主機的狀態
Sat Jun 22 20:18:06 2019 - [warning] Got error on MySQL select ping: 2006 (MySQL server has gone away) Sat Jun 22 20:18:06 2019 - [info] Executing SSH check script: save_binary_logs --command=test --start_pos=4 --binlog_dir=/usr/local/mysql/data --output_file=/tmp/save_binary_logs_test --manager_version=0.56 --binlog_prefix=mysql_bin Sat Jun 22 20:18:07 2019 - [info] HealthCheck: SSH to 172.16.15.3 is reachable. Sat Jun 22 20:18:07 2019 - [warning] Got error on MySQL connect: 2013 (Lost connection to MySQL server at 'reading initial communication packet', system error: 111) Sat Jun 22 20:18:07 2019 - [warning] Connection failed 2 time(s).. Sat Jun 22 20:18:08 2019 - [warning] Got error on MySQL connect: 2013 (Lost connection to MySQL server at 'reading initial communication packet', system error: 111) Sat Jun 22 20:18:08 2019 - [warning] Connection failed 3 time(s).. Sat Jun 22 20:18:09 2019 - [warning] Got error on MySQL connect: 2013 (Lost connection to MySQL server at 'reading initial communication packet', system error: 111) Sat Jun 22 20:18:09 2019 - [warning] Connection failed 4 time(s).. Sat Jun 22 20:18:09 2019 - [warning] Master is not reachable from health checker! Sat Jun 22 20:18:09 2019 - [warning] Master 172.16.15.3(172.16.15.3:3306) is not reachable! Sat Jun 22 20:18:09 2019 - [warning] SSH is reachable. Sat Jun 22 20:18:09 2019 - [info] Connecting to a master server failed. Reading configuration file /etc/masterha_default.cnf and /etc/masterha/app1.conf again, and trying to connect to all servers to check server status.. Sat Jun 22 20:18:09 2019 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping. Sat Jun 22 20:18:09 2019 - [info] Reading application default configuration from /etc/masterha/app1.conf.. Sat Jun 22 20:18:09 2019 - [info] Reading server configuration from /etc/masterha/app1.conf.. Sat Jun 22 20:18:10 2019 - [info] GTID failover mode = 0 Sat Jun 22 20:18:10 2019 - [info] Dead Servers: Sat Jun 22 20:18:10 2019 - [info] 172.16.15.3(172.16.15.3:3306) Sat Jun 22 20:18:10 2019 - [info] Alive Servers: Sat Jun 22 20:18:10 2019 - [info] 172.16.13.15(172.16.13.15:3306) Sat Jun 22 20:18:10 2019 - [info] 172.16.15.2(172.16.15.2:3306) Sat Jun 22 20:18:10 2019 - [info] Alive Slaves: Sat Jun 22 20:18:10 2019 - [info] 172.16.13.15(172.16.13.15:3306) Version=5.7.16-log (oldest major version between slaves) log-bin:enabled Sat Jun 22 20:18:10 2019 - [info] Replicating from 172.16.15.3(172.16.15.3:3306) Sat Jun 22 20:18:10 2019 - [info] Primary candidate for the new Master (candidate_master is set) Sat Jun 22 20:18:10 2019 - [info] 172.16.15.2(172.16.15.2:3306) Version=5.7.16-log (oldest major version between slaves) log-bin:enabled Sat Jun 22 20:18:10 2019 - [info] Replicating from 172.16.15.3(172.16.15.3:3306)
2、檢查從庫配置，MHA Node版本，再次確認Master 不可用
Sat Jun 22 20:18:10 2019 - [info] Checking slave configurations.. Sat Jun 22 20:18:10 2019 - [info] Checking replication filtering settings.. Sat Jun 22 20:18:10 2019 - [info] Replication filtering check ok. Sat Jun 22 20:18:10 2019 - [info] Master is down! Sat Jun 22 20:18:10 2019 - [info] Terminating monitoring script. Sat Jun 22 20:18:10 2019 - [info] Got exit code 20 (Master dead). Sat Jun 22 20:18:10 2019 - [info] MHA::MasterFailover version 0.56.
3、開始故障切換 Starting master failover
Phase 1: Configuration Check Phase..配置檢查（得到當前主機狀態）
Sat Jun 22 20:18:10 2019 - [info] Starting master failover. Sat Jun 22 20:18:10 2019 - [info] Sat Jun 22 20:18:10 2019 - [info] * Phase 1: Configuration Check Phase.. Sat Jun 22 20:18:10 2019 - [info] Sat Jun 22 20:18:12 2019 - [info] GTID failover mode = 0 Sat Jun 22 20:18:12 2019 - [info] Dead Servers: Sat Jun 22 20:18:12 2019 - [info] 172.16.15.3(172.16.15.3:3306) Sat Jun 22 20:18:12 2019 - [info] Checking master reachability via MySQL(double check)... Sat Jun 22 20:18:12 2019 - [info] ok. Sat Jun 22 20:18:12 2019 - [info] Alive Servers: Sat Jun 22 20:18:12 2019 - [info] 172.16.13.15(172.16.13.15:3306) Sat Jun 22 20:18:12 2019 - [info] 172.16.15.2(172.16.15.2:3306) Sat Jun 22 20:18:12 2019 - [info] Alive Slaves: Sat Jun 22 20:18:12 2019 - [info] 172.16.13.15(172.16.13.15:3306) Version=5.7.16-log (oldest major version between slaves) log-bin:enabled Sat Jun 22 20:18:12 2019 - [info] Replicating from 172.16.15.3(172.16.15.3:3306) Sat Jun 22 20:18:12 2019 - [info] Primary candidate for the new Master (candidate_master is set) Sat Jun 22 20:18:12 2019 - [info] 172.16.15.2(172.16.15.2:3306) Version=5.7.16-log (oldest major version between slaves) log-bin:enabled Sat Jun 22 20:18:12 2019 - [info] Replicating from 172.16.15.3(172.16.15.3:3306) Sat Jun 22 20:18:12 2019 - [info] Starting Non-GTID based failover. Sat Jun 22 20:18:12 2019 - [info] Sat Jun 22 20:18:12 2019 - [info] ** Phase 1: Configuration Check Phase completed. Sat Jun 22 20:18:12 2019 - [info]
Phase 2: Dead Master Shutdown Phase..嘗試關閉從庫（將虛擬IP關掉，並嘗試啟動shutdown腳本）
Sat Jun 22 20:18:12 2019 - [info] * Phase 2: Dead Master Shutdown Phase.. Sat Jun 22 20:18:12 2019 - [info] Sat Jun 22 20:18:12 2019 - [info] Forcing shutdown so that applications never connect to the current master.. Sat Jun 22 20:18:12 2019 - [info] Executing master IP deactivation script: Sat Jun 22 20:18:12 2019 - [info] /var/log/masterha/scripts/master_ip_failover --orig_master_host=172.16.15.3 --orig_master_ip=172.16.15.3 --orig_master_port=3306 --command=stopssh --ssh_user=root *************************************************************** Disabling the VIP - 172.16.13.141/16 on old master: 172.16.15.3 *************************************************************** Sat Jun 22 20:18:12 2019 - [info] done. Sat Jun 22 20:18:12 2019 - [warning] shutdown_script is not set. Skipping explicit shutting down of the dead master. Sat Jun 22 20:18:12 2019 - [info] * Phase 2: Dead Master Shutdown Phase completed. Sat Jun 22 20:18:12 2019 - [info]
Phase 3: Master Recovery Phase..
Phase 3.1: Getting Latest Slaves Phase..得到距離舊主binlog最近（least slave）和最遠的binlog位置
Sat Jun 22 20:18:12 2019 - [info] * Phase 3: Master Recovery Phase.. Sat Jun 22 20:18:12 2019 - [info] Sat Jun 22 20:18:12 2019 - [info] * Phase 3.1: Getting Latest Slaves Phase.. Sat Jun 22 20:18:12 2019 - [info] Sat Jun 22 20:18:12 2019 - [info] The latest binary log file/position on all slaves is mysql_bin.000024:19243814 Sat Jun 22 20:18:12 2019 - [info] Latest slaves (Slaves that received relay log files to the latest): Sat Jun 22 20:18:12 2019 - [info] 172.16.13.15(172.16.13.15:3306) Version=5.7.16-log (oldest major version between slaves) log-bin:enabled Sat Jun 22 20:18:12 2019 - [info] Replicating from 172.16.15.3(172.16.15.3:3306) Sat Jun 22 20:18:12 2019 - [info] Primary candidate for the new Master (candidate_master is set) Sat Jun 22 20:18:12 2019 - [info] 172.16.15.2(172.16.15.2:3306) Version=5.7.16-log (oldest major version between slaves) log-bin:enabled Sat Jun 22 20:18:12 2019 - [info] Replicating from 172.16.15.3(172.16.15.3:3306) Sat Jun 22 20:18:12 2019 - [info] The oldest binary log file/position on all slaves is mysql_bin.000024:19243814 Sat Jun 22 20:18:12 2019 - [info] Oldest slaves: Sat Jun 22 20:18:12 2019 - [info] 172.16.13.15(172.16.13.15:3306) Version=5.7.16-log (oldest major version between slaves) log-bin:enabled Sat Jun 22 20:18:12 2019 - [info] Replicating from 172.16.15.3(172.16.15.3:3306) Sat Jun 22 20:18:12 2019 - [info] Primary candidate for the new Master (candidate_master is set) Sat Jun 22 20:18:12 2019 - [info] 172.16.15.2(172.16.15.2:3306) Version=5.7.16-log (oldest major version between slaves) log-bin:enabled Sat Jun 22 20:18:12 2019 - [info] Replicating from 172.16.15.3(172.16.15.3:3306) Sat Jun 22 20:18:12 2019 - [info]
Phase 3.2: Saving Dead Master's Binlog Phase..保存（截取）舊主庫的binlog和從庫最接近（least slave）的這段 binlog，保存到 tmp目錄
Sat Jun 22 20:18:12 2019 - [info] * Phase 3.2: Saving Dead Master's Binlog Phase.. Sat Jun 22 20:18:12 2019 - [info] Sat Jun 22 20:18:12 2019 - [info] Fetching dead master's binary logs.. Sat Jun 22 20:18:12 2019 - [info] Executing command on the dead master 172.16.15.3(172.16.15.3:3306): save_binary_logs --command=save --start_file=mysql_bin.000024 --start_pos=19243814 --binlog_dir=/usr/local/mysql/data --output_file=/tmp/saved_master_binlog_from_172.16.15.3_3306_20190622201810.binlog --handle_raw_binlog=1 --disable_log_bin=0 --manager_version=0.56 Creating /tmp if not exists.. ok. Concat binary/relay logs from mysql_bin.000024 pos 19243814 to mysql_bin.000024 EOF into /tmp/saved_master_binlog_from_172.16.15.3_3306_20190622201810.binlog .. Binlog Checksum enabled Dumping binlog format description event, from position 0 to 194.. ok. No need to dump effective binlog data from /usr/local/mysql/data/mysql_bin.000024 (pos starts 19243814, filesize 19243814). Skipping. Binlog Checksum enabled /tmp/saved_master_binlog_from_172.16.15.3_3306_20190622201810.binlog has no effective data events. Event not exists. Sat Jun 22 20:18:12 2019 - [info] Additional events were not found from the orig master. No need to save.
Phase 3.3: Determining New Master Phase..決定提升新的主庫
Phase 3.3: New Master Diff Log Generation Phase..生成新主和最新least binlog 之間的差異日志文件
Sat Jun 22 20:18:12 2019 - [info] * Phase 3.3: Determining New Master Phase.. Sat Jun 22 20:18:12 2019 - [info] Sat Jun 22 20:18:12 2019 - [info] Finding the latest slave that has all relay logs for recovering other slaves.. Sat Jun 22 20:18:12 2019 - [info] All slaves received relay logs to the same position. No need to resync each other. Sat Jun 22 20:18:12 2019 - [info] Searching new master from slaves.. Sat Jun 22 20:18:12 2019 - [info] Candidate masters from the configuration file: Sat Jun 22 20:18:12 2019 - [info] 172.16.13.15(172.16.13.15:3306) Version=5.7.16-log (oldest major version between slaves) log-bin:enabled Sat Jun 22 20:18:12 2019 - [info] Replicating from 172.16.15.3(172.16.15.3:3306) Sat Jun 22 20:18:12 2019 - [info] Primary candidate for the new Master (candidate_master is set) Sat Jun 22 20:18:12 2019 - [info] Non-candidate masters: Sat Jun 22 20:18:12 2019 - [info] Searching from candidate_master slaves which have received the latest relay log events.. Sat Jun 22 20:18:12 2019 - [info] New master is 172.16.13.15(172.16.13.15:3306) Sat Jun 22 20:18:12 2019 - [info] Starting master failover.. Sat Jun 22 20:18:12 2019 - [info] From: 172.16.15.3(172.16.15.3:3306) (current master) +--172.16.13.15(172.16.13.15:3306) +--172.16.15.2(172.16.15.2:3306) To: 172.16.13.15(172.16.13.15:3306) (new master) +--172.16.15.2(172.16.15.2:3306) Sat Jun 22 20:18:12 2019 - [info] Sat Jun 22 20:18:12 2019 - [info] * Phase 3.3: New Master Diff Log Generation Phase.. Sat Jun 22 20:18:12 2019 - [info] Sat Jun 22 20:18:12 2019 - [info] This server has all relay logs. No need to generate diff files from the latest slave. Sat Jun 22 20:18:12 2019 - [info]
Phase 3.4: Master Log Apply Phase..新的主庫應用日志（包括新主與least slave之間的差異日志和least slave與舊主之間的日志），使新主到達和舊主一致的狀態，得到新的主庫的binlog name和position，用於后面在新的從庫上執行 change master to，指向新的主庫
在新的主庫上面開啟虛擬IP
Sat Jun 22 20:18:12 2019 - [info] * Phase 3.4: Master Log Apply Phase.. Sat Jun 22 20:18:12 2019 - [info] Sat Jun 22 20:18:12 2019 - [info] *NOTICE: If any error happens from this phase, manual recovery is needed. Sat Jun 22 20:18:12 2019 - [info] Starting recovery on 172.16.13.15(172.16.13.15:3306).. Sat Jun 22 20:18:12 2019 - [info] This server has all relay logs. Waiting all logs to be applied.. Sat Jun 22 20:19:02 2019 - [info] done. Sat Jun 22 20:19:02 2019 - [info] All relay logs were successfully applied. Sat Jun 22 20:19:02 2019 - [info] Getting new master's binlog name and position.. Sat Jun 22 20:19:02 2019 - [info] mysql_bin.000056:127086216 Sat Jun 22 20:19:02 2019 - [info] All other slaves should start replication from here. Statement should be: CHANGE MASTER TO MASTER_HOST='172.16.13.15', MASTER_PORT=3306, MASTER_LOG_FILE='mysql_bin.000056', MASTER_LOG_POS=127086216, MASTER_USER='root', MASTER_PASSWORD='xxx'; Sat Jun 22 20:19:02 2019 - [info] Executing master IP activate script: Sat Jun 22 20:19:02 2019 - [info] /var/log/masterha/scripts/master_ip_failover --command=start --ssh_user=root --orig_master_host=172.16.15.3 --orig_master_ip=172.16.15.3 --orig_master_port=3306 --new_master_host=172.16.13.15 --new_master_ip=172.16.13.15 --new_master_port=3306 --new_master_user='root' --new_master_password='115433' Unknown option: new_master_user Unknown option: new_master_password *************************************************************** Enabling the VIP - 172.16.13.141/16 on new master: 172.16.13.15 *************************************************************** Sat Jun 22 20:19:02 2019 - [info] OK. Sat Jun 22 20:19:02 2019 - [info] Setting read_only=0 on 172.16.13.15(172.16.13.15:3306).. Sat Jun 22 20:19:02 2019 - [info] ok. Sat Jun 22 20:19:02 2019 - [info] ** Finished master recovery successfully. Sat Jun 22 20:19:02 2019 - [info] * Phase 3: Master Recovery Phase completed. Sat Jun 22 20:19:02 2019 - [info]
至此，新的主庫恢復完成
Phase 4: Slaves Recovery Phase..開始進行從庫的恢復
Phase 4.1: Starting Parallel Slave Diff Log Generation Phase..並行生成各個從庫和least slave之間的差異日志
Sat Jun 22 20:19:02 2019 - [info] * Phase 4: Slaves Recovery Phase.. Sat Jun 22 20:19:02 2019 - [info] Sat Jun 22 20:19:02 2019 - [info] * Phase 4.1: Starting Parallel Slave Diff Log Generation Phase.. Sat Jun 22 20:19:02 2019 - [info] Sat Jun 22 20:19:02 2019 - [info] -- Slave diff file generation on host 172.16.15.2(172.16.15.2:3306) started, pid: 10087. Check tmp log /var/log/masterha/app1/172.16.15.2_3306_20190622201810.log if it takes time.. Sat Jun 22 20:19:03 2019 - [info] Sat Jun 22 20:19:03 2019 - [info] Log messages from 172.16.15.2 ... Sat Jun 22 20:19:03 2019 - [info] Sat Jun 22 20:19:02 2019 - [info] This server has all relay logs. No need to generate diff files from the latest slave. Sat Jun 22 20:19:03 2019 - [info] End of log messages from 172.16.15.2. Sat Jun 22 20:19:03 2019 - [info] -- 172.16.15.2(172.16.15.2:3306) has the latest relay log events. Sat Jun 22 20:19:03 2019 - [info] Generating relay diff files from the latest slave succeeded. Sat Jun 22 20:19:03 2019 - [info]
Phase 4.2: Starting Parallel Slave Log Apply Phase..從庫並行應用日志（包括從庫與least slave之間的差異日志和least slave與舊主之間的差異日志）
執行change master to，使新從庫指向新的主庫
Sat Jun 22 20:19:03 2019 - [info] * Phase 4.2: Starting Parallel Slave Log Apply Phase.. Sat Jun 22 20:19:03 2019 - [info] Sat Jun 22 20:19:03 2019 - [info] -- Slave recovery on host 172.16.15.2(172.16.15.2:3306) started, pid: 10089. Check tmp log /var/log/masterha/app1/172.16.15.2_3306_20190622201810.log if it takes time.. Sat Jun 22 20:19:04 2019 - [info] Sat Jun 22 20:19:04 2019 - [info] Log messages from 172.16.15.2 ... Sat Jun 22 20:19:04 2019 - [info] Sat Jun 22 20:19:03 2019 - [info] Starting recovery on 172.16.15.2(172.16.15.2:3306).. Sat Jun 22 20:19:03 2019 - [info] This server has all relay logs. Waiting all logs to be applied.. Sat Jun 22 20:19:03 2019 - [info] done. Sat Jun 22 20:19:03 2019 - [info] All relay logs were successfully applied. Sat Jun 22 20:19:03 2019 - [info] Resetting slave 172.16.15.2(172.16.15.2:3306) and starting replication from the new master 172.16.13.15(172.16.13.15:3306).. Sat Jun 22 20:19:03 2019 - [info] Executed CHANGE MASTER. Sat Jun 22 20:19:03 2019 - [info] Slave started. Sat Jun 22 20:19:04 2019 - [info] End of log messages from 172.16.15.2. Sat Jun 22 20:19:04 2019 - [info] -- Slave recovery on host 172.16.15.2(172.16.15.2:3306) succeeded. Sat Jun 22 20:19:04 2019 - [info] All new slave servers recovered successfully.
Phase 5: New master cleanup phase..在新的主庫上重新設置slave信息
Sat Jun 22 20:19:04 2019 - [info] * Phase 5: New master cleanup phase.. Sat Jun 22 20:19:04 2019 - [info] Sat Jun 22 20:19:04 2019 - [info] Resetting slave info on the new master.. Sat Jun 22 20:19:06 2019 - [info] 172.16.13.15: Resetting slave info succeeded. Sat Jun 22 20:19:06 2019 - [info] Master failover to 172.16.13.15(172.16.13.15:3306) completed successfully. Sat Jun 22 20:19:06 2019 - [info]
生成故障切換報告
----- Failover Report ----- app1: MySQL Master failover 172.16.15.3(172.16.15.3:3306) to 172.16.13.15(172.16.13.15:3306) succeeded Master 172.16.15.3(172.16.15.3:3306) is down! Check MHA Manager logs at A2:/var/log/masterha/app1/manager.log for details. Started automated(non-interactive) failover. Invalidated master IP address on 172.16.15.3(172.16.15.3:3306) The latest slave 172.16.13.15(172.16.13.15:3306) has all relay logs for recovery. Selected 172.16.13.15(172.16.13.15:3306) as a new master. 172.16.13.15(172.16.13.15:3306): OK: Applying all logs succeeded. 172.16.13.15(172.16.13.15:3306): OK: Activated master IP address. 172.16.15.2(172.16.15.2:3306): This host has the latest relay log events. Generating relay diff files from the latest slave succeeded. 172.16.15.2(172.16.15.2:3306): OK: Applying all logs succeeded. Slave started, replicating from 172.16.13.15(172.16.13.15:3306) 172.16.13.15(172.16.13.15:3306): Resetting slave info succeeded. Master failover to 172.16.13.15(172.16.13.15:3306) completed successfully.