常用的mysql 高可用有下面幾種方案：
| 名稱 | 原理 | 特點 |
| mysqlmha | Perl腳本對mysql master做心跳，master down了以后，選舉new master | ，是要改代理層的路由信息。 |
| mmm | 多主復制管理器， 主備通過VIP共享ip ， 監控節點做輪詢定期檢查master是否可用，不可用時 ，切換主備， | 輪詢有時間間隔， 這段時間內如果master down ， 則事務會大量失敗， 不建議 |
| heartbeat+ drbd | 主備通過VIP共享ip ,只有一個master提供服務，stand by master通過drbd進行塊的復制，類似於raid1 | 不適合大集群 |
| mysql cluster | ndb 存儲引擎提供存儲集群，mysqld 集群提供訪問服務， | 用的公司少，沒有大規模部署的經驗，問題多。 |
| percona galera cluster | galera 為innodb提供數據同步 ，所有mysql節點都是讀寫節點，原理有點類似於mysql cluster ,都是在存儲集群上做文章。 | 理論上節點間的復制隨着節點的增加會變慢。因為是1對 n-1的一個復制關系。
|
從上表可以看到，只剩下mysqlmha + percona galera cluster 作為我們的備選方案 ，后者我們單獨找時間來測試galera cluster。
我們先測試mysqlmha
測試環境：
OS: centos 6.5
mysql: 5.6.27-log
mysql-master-ha: 0.56
和https://code.google.com/p/mysql-master-ha/wiki/TableOfContents?tm=6提供的文檔一樣，采用了如下mysql測試拓撲結構：
MySQL master(一台 ）
mysql slave (2 台 ）
mysql-master-ha-manager(一台 )
在所有機器上安裝 mysql-master-ha node 節點，安裝mysql-master-ha node manager; 安裝方法參考這里： https://code.google.com/p/mysql-master-ha/wiki/Tutorial
配置好mysql master 到 MySQL slave 的主從復制關系以后，我們在manger節點啟動 masterha_manager
masterha_manager --conf=./app1.conf
然后我們在master節點上killall -9 mysqld mysqld_safe把master殺掉， 觀察manager節點上的日志輸出 ， 來理解mysql-master-ha 是如何做mysql master的高可用的：
[root@server9 mysql-master-ha-conf]# masterha_manager --conf=./app1.conf
Tue Nov 10 10:49:40 2015 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Tue Nov 10 10:49:40 2015 - [info] Reading application default configuration from ./app1.conf..
Tue Nov 10 10:49:40 2015 - [info] Reading server configuration from ./app1.conf..
Tue Nov 10 10:49:40 2015 - [info] MHA::MasterMonitor version 0.56.
Tue Nov 10 10:49:40 2015 - [info] GTID failover mode = 0
1\發現old master不可用。
Tue Nov 10 10:49:40 2015 - [info] Dead Servers:
Tue Nov 10 10:49:40 2015 - [info] server7(server7:3307)
Tue Nov 10 10:49:40 2015 - [info] Alive Servers:
Tue Nov 10 10:49:40 2015 - [info] server6(server6:3307)
Tue Nov 10 10:49:40 2015 - [info] server8(server8:3307)
Tue Nov 10 10:49:40 2015 - [info] Alive Slaves:
Tue Nov 10 10:49:40 2015 - [info] server6(server6:3307) Version=5.6.27-log (oldest major version between slaves) log-bin:enabled
Tue Nov 10 10:49:40 2015 - [info] Replicating from server7(server7:3307)
Tue Nov 10 10:49:40 2015 - [info] server8(server8:3307) Version=5.6.27-log (oldest major version between slaves) log-bin:enabled
Tue Nov 10 10:49:40 2015 - [info] Replicating from server7(server7:3307)
Tue Nov 10 10:49:40 2015 - [warning] MySQL master is not currently alive!
2\檢查slave配置， 檢查master 的ssh連通性，檢查master binlog備份腳本的有效性。
檢查slave 的ssh聯通性： 進入slave檢查relay-log 的apply_diff_relay_logs腳本：
Tue Nov 10 10:49:40 2015 - [info] Checking slave configurations..
Tue Nov 10 10:49:40 2015 - [info] read_only=1 is not set on slave server6(server6:3307).
Tue Nov 10 10:49:40 2015 - [warning] relay_log_purge=0 is not set on slave server6(server6:3307).
Tue Nov 10 10:49:40 2015 - [info] read_only=1 is not set on slave server8(server8:3307).
Tue Nov 10 10:49:40 2015 - [info] Checking replication filtering settings..
Tue Nov 10 10:49:40 2015 - [info] Replication filtering check ok.
Tue Nov 10 10:49:40 2015 - [info] GTID (with auto-pos) is not supported
Tue Nov 10 10:49:40 2015 - [info] Starting SSH connection tests..
Tue Nov 10 10:49:41 2015 - [info] All SSH connection tests passed successfully.
Tue Nov 10 10:49:41 2015 - [info] Checking MHA Node version..
Tue Nov 10 10:49:42 2015 - [info] Version check ok.
Tue Nov 10 10:49:42 2015 - [info] Getting current master (maybe dead) info ..
Tue Nov 10 10:49:42 2015 - [info] Identified master is server7(server7:3307).
Tue Nov 10 10:49:42 2015 - [info] Checking SSH publickey authentication settings on the current master..
Tue Nov 10 10:49:42 2015 - [info] HealthCheck: SSH to server7 is reachable.
Tue Nov 10 10:49:43 2015 - [info] Master MHA Node version is 0.56.
Tue Nov 10 10:49:43 2015 - [info] Checking recovery script configurations on server7(server7:3307)..
Tue Nov 10 10:49:43 2015 - [info] Executing command: save_binary_logs --command=test --start_pos=4 --binlog_dir=/var/lib/mysql,/var/log/mysql --output_file=/var/log/masterha/app1/save_binary_logs_test --manager_version=0.56 --start_file=master-bin.000004
Tue Nov 10 10:49:43 2015 - [info] Connecting to root@server7(server7:22)..
Creating /var/log/masterha/app1 if not exists.. ok.
Checking output directory is accessible or not..
ok.
Binlog found at /var/lib/mysql, up to master-bin.000004
Tue Nov 10 10:49:43 2015 - [info] Binlog setting check done.
Tue Nov 10 10:49:43 2015 - [info] Checking SSH publickey authentication and checking recovery script configurations on all alive slave servers..
Tue Nov 10 10:49:43 2015 - [info] Executing command : apply_diff_relay_logs --command=test --slave_user='root' --slave_host=server6 --slave_ip=server6 --slave_port=3307 --workdir=/var/log/masterha/app1 --target_version=5.6.27-log --manager_version=0.56 --relay_log_info=/var/lib/mysql/relay-log.info --relay_dir=/var/lib/mysql/ --slave_pass=xxx
Tue Nov 10 10:49:43 2015 - [info] Connecting to root@server6(server6:22)..
Checking slave recovery environment settings..
Opening /var/lib/mysql/relay-log.info ... ok.
Relay log found at /var/lib/mysql, up to mysqld-relay-bin.000007
Temporary relay log file is /var/lib/mysql/mysqld-relay-bin.000007
Testing mysql connection and privileges..Warning: Using a password on the command line interface can be insecure.
done.
Testing mysqlbinlog output.. done.
Cleaning up test file(s).. done.
Tue Nov 10 10:49:43 2015 - [info] Executing command : apply_diff_relay_logs --command=test --slave_user='root' --slave_host=server8 --slave_ip=server8 --slave_port=3307 --workdir=/var/log/masterha/app1 --target_version=5.6.27-log --manager_version=0.56 --relay_log_info=/var/lib/mysql/relay-log.info --relay_dir=/var/lib/mysql/ --slave_pass=xxx
Tue Nov 10 10:49:43 2015 - [info] Connecting to root@server8(server8:22)..
Checking slave recovery environment settings..
Opening /var/lib/mysql/relay-log.info ... ok.
Relay log found at /var/lib/mysql, up to slave-relay-bin.000004
Temporary relay log file is /var/lib/mysql/slave-relay-bin.000004
Testing mysql connection and privileges..Warning: Using a password on the command line interface can be insecure.
done.
Testing mysqlbinlog output.. done.
Cleaning up test file(s).. done.
Tue Nov 10 10:49:44 2015 - [info] Slaves settings check done.
Tue Nov 10 10:49:44 2015 - [info]
server7(server7:3307) (current master)
+--server6(server6:3307)
+--server8(server8:3307)
3、確認 old master的mysql不可到達 （ping 3307端口不可用， 建議設置二級檢查， 從
多個路由確認old master不可到達。）， 連續多次ping old master的3307端口不可到達以后，
准備啟動master failover;
----
Tue Nov 10 10:49:44 2015 - [warning] master_ip_failover_script is not defined.
Tue Nov 10 10:49:44 2015 - [warning] shutdown_script is not defined.
Tue Nov 10 10:49:44 2015 - [error][/usr/local/share/perl5/MHA/Server.pm, ln457] Checking slave status failed on server6(server6:3307). err=Got error when executing SHOW SLAVE STATUS. MySQL server has gone away
Tue Nov 10 10:49:44 2015 - [info] Set master ping interval 3 seconds.
Tue Nov 10 10:49:44 2015 - [warning] secondary_check_script is not defined. It is highly recommended setting it to check master reachability from two or more routes.
Tue Nov 10 10:49:44 2015 - [info] Starting ping health check on server7(server7:3307)..
Tue Nov 10 10:49:44 2015 - [warning] Got error on MySQL connect: 2013 (Lost connection to MySQL server at 'reading initial communication packet', system error: 111)
Tue Nov 10 10:49:44 2015 - [warning] Connection failed 1 time(s)..
Tue Nov 10 10:49:44 2015 - [info] Executing SSH check script: save_binary_logs --command=test --start_pos=4 --binlog_dir=/var/lib/mysql,/var/log/mysql --output_file=/var/log/masterha/app1/save_binary_logs_test --manager_version=0.56 --binlog_prefix=master-bin
Creating /var/log/masterha/app1 if not exists.. ok.
Checking output directory is accessible or not..
ok.
Binlog found at /var/lib/mysql, up to master-bin.000004
Tue Nov 10 10:49:44 2015 - [info] HealthCheck: SSH to server7 is reachable.
Tue Nov 10 10:49:47 2015 - [warning] Got error on MySQL connect: 2013 (Lost connection to MySQL server at 'reading initial communication packet', system error: 111)
Tue Nov 10 10:49:47 2015 - [warning] Connection failed 2 time(s)..
Tue Nov 10 10:49:50 2015 - [warning] Got error on MySQL connect: 2013 (Lost connection to MySQL server at 'reading initial communication packet', system error: 111)
Tue Nov 10 10:49:50 2015 - [warning] Connection failed 3 time(s)..
Tue Nov 10 10:49:53 2015 - [warning] Got error on MySQL connect: 2013 (Lost connection to MySQL server at 'reading initial communication packet', system error: 111)
Tue Nov 10 10:49:53 2015 - [warning] Connection failed 4 time(s)..
Tue Nov 10 10:49:53 2015 - [warning] Master is not reachable from health checker!
Tue Nov 10 10:49:53 2015 - [warning] Master server7(server7:3307) is not reachable!
Tue Nov 10 10:49:53 2015 - [warning] SSH is reachable.
Tue Nov 10 10:49:53 2015 - [info] Connecting to a master server failed. Reading configuration file /etc/masterha_default.cnf and ./app1.conf again, and trying to connect to all servers to check server status..
Tue Nov 10 10:49:53 2015 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Tue Nov 10 10:49:53 2015 - [info] Reading application default configuration from ./app1.conf..
Tue Nov 10 10:49:53 2015 - [info] Reading server configuration from ./app1.conf..
Tue Nov 10 10:49:53 2015 - [info] GTID failover mode = 0
Tue Nov 10 10:49:53 2015 - [info] Dead Servers:
Tue Nov 10 10:49:53 2015 - [info] server7(server7:3307)
Tue Nov 10 10:49:53 2015 - [info] Alive Servers:
Tue Nov 10 10:49:53 2015 - [info] server6(server6:3307)
Tue Nov 10 10:49:53 2015 - [info] server8(server8:3307)
Tue Nov 10 10:49:53 2015 - [info] Alive Slaves:
Tue Nov 10 10:49:53 2015 - [info] server6(server6:3307) Version=5.6.27-log (oldest major version between slaves) log-bin:enabled
Tue Nov 10 10:49:53 2015 - [info] Replicating from server7(server7:3307)
Tue Nov 10 10:49:53 2015 - [info] server8(server8:3307) Version=5.6.27-log (oldest major version between slaves) log-bin:enabled
Tue Nov 10 10:49:53 2015 - [info] Replicating from server7(server7:3307)
Tue Nov 10 10:49:53 2015 - [info] Checking slave configurations..
Tue Nov 10 10:49:53 2015 - [info] read_only=1 is not set on slave server6(server6:3307).
Tue Nov 10 10:49:53 2015 - [warning] relay_log_purge=0 is not set on slave server6(server6:3307).
Tue Nov 10 10:49:53 2015 - [info] read_only=1 is not set on slave server8(server8:3307).
Tue Nov 10 10:49:53 2015 - [info] Checking replication filtering settings..
Tue Nov 10 10:49:53 2015 - [info] Replication filtering check ok.
Tue Nov 10 10:49:53 2015 - [info] Master is down!
Tue Nov 10 10:49:53 2015 - [info] Terminating monitoring script.
Tue Nov 10 10:49:53 2015 - [info] Got exit code 20 (Master dead).
Tue Nov 10 10:49:53 2015 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Tue Nov 10 10:49:53 2015 - [info] Reading application default configuration from ./app1.conf..
Tue Nov 10 10:49:53 2015 - [info] Reading server configuration from ./app1.conf..
Tue Nov 10 10:49:53 2015 - [info] MHA::MasterFailover version 0.56.
4、啟動master failover :
1) 階段1 ： 配置檢查：
Tue Nov 10 10:49:53 2015 - [info] Starting master failover.
Tue Nov 10 10:49:53 2015 - [info]
Tue Nov 10 10:49:53 2015 - [info] * Phase 1: Configuration Check Phase..
Tue Nov 10 10:49:53 2015 - [info]
Tue Nov 10 10:49:53 2015 - [info] GTID failover mode = 0
Tue Nov 10 10:49:53 2015 - [info] Dead Servers:
Tue Nov 10 10:49:53 2015 - [info] server7(server7:3307)
Tue Nov 10 10:49:53 2015 - [info] Checking master reachability via MySQL(double check)...
Tue Nov 10 10:49:53 2015 - [info] ok.
Tue Nov 10 10:49:53 2015 - [info] Alive Servers:
Tue Nov 10 10:49:53 2015 - [info] server6(server6:3307)
Tue Nov 10 10:49:53 2015 - [info] server8(server8:3307)
Tue Nov 10 10:49:53 2015 - [info] Alive Slaves:
Tue Nov 10 10:49:53 2015 - [info] server6(server6:3307) Version=5.6.27-log (oldest major version between slaves) log-bin:enabled
Tue Nov 10 10:49:53 2015 - [info] Replicating from server7(server7:3307)
Tue Nov 10 10:49:53 2015 - [info] server8(server8:3307) Version=5.6.27-log (oldest major version between slaves) log-bin:enabled
Tue Nov 10 10:49:53 2015 - [info] Replicating from server7(server7:3307)
Tue Nov 10 10:49:53 2015 - [info] Starting Non-GTID based failover.
Tue Nov 10 10:49:53 2015 - [info]
Tue Nov 10 10:49:53 2015 - [info] ** Phase 1: Configuration Check Phase completed.
Tue Nov 10 10:49:53 2015 - [info]
2） 階段2： 關閉dead master(master_ip_failover_script腳本沒有做 ， 忽略對deadmaster
的ip 失效，。);
Tue Nov 10 10:49:53 2015 - [info] * Phase 2: Dead Master Shutdown Phase..
Tue Nov 10 10:49:53 2015 - [info]
Tue Nov 10 10:49:53 2015 - [info] Forcing shutdown so that applications never connect to the current master..
Tue Nov 10 10:49:53 2015 - [warning] master_ip_failover_script is not set. Skipping invalidating dead master IP address.
Tue Nov 10 10:49:53 2015 - [warning] shutdown_script is not set. Skipping explicit shutting down of the dead master.
Tue Nov 10 10:49:53 2015 - [info] * Phase 2: Dead Master Shutdown Phase completed.
Tue Nov 10 10:49:53 2015 - [info]
3） 階段3 ： master恢復階段：
A:找lastest slave :
show slave status: 得到的Relay_Master_Log_File + Read_Master_Log_Pos 最大的就可以 。
因為server6和server8 的讀取master位點一樣，所以都是oldest和lastest ;
Tue Nov 10 10:49:53 2015 - [info] * Phase 3: Master Recovery Phase..
Tue Nov 10 10:49:53 2015 - [info]
Tue Nov 10 10:49:53 2015 - [info] * Phase 3.1: Getting Latest Slaves Phase..
Tue Nov 10 10:49:53 2015 - [info]
Tue Nov 10 10:49:53 2015 - [info] The latest binary log file/position on all slaves is master-bin.000004:120
Tue Nov 10 10:49:53 2015 - [info] Latest slaves (Slaves that received relay log files to the latest):
Tue Nov 10 10:49:53 2015 - [info] server6(server6:3307) Version=5.6.27-log (oldest major version between slaves) log-bin:enabled
Tue Nov 10 10:49:53 2015 - [info] Replicating from server7(server7:3307)
Tue Nov 10 10:49:53 2015 - [info] server8(server8:3307) Version=5.6.27-log (oldest major version between slaves) log-bin:enabled
Tue Nov 10 10:49:53 2015 - [info] Replicating from server7(server7:3307)
Tue Nov 10 10:49:53 2015 - [info] The oldest binary log file/position on all slaves is master-bin.000004:120
Tue Nov 10 10:49:53 2015 - [info] Oldest slaves:
Tue Nov 10 10:49:53 2015 - [info] server6(server6:3307) Version=5.6.27-log (oldest major version between slaves) log-bin:enabled
Tue Nov 10 10:49:53 2015 - [info] Replicating from server7(server7:3307)
Tue Nov 10 10:49:53 2015 - [info] server8(server8:3307) Version=5.6.27-log (oldest major version between slaves) log-bin:enabled
Tue Nov 10 10:49:53 2015 - [info] Replicating from server7(server7:3307)
Tue Nov 10 10:49:53 2015 - [info]
B:保存dead master binlog:
從lastest slave讀到的master binlog 的位點開始截取dead master binlog ，加上binlog的文件描述信息。
Tue Nov 10 10:49:53 2015 - [info] * Phase 3.2: Saving Dead Master's Binlog Phase..
Tue Nov 10 10:49:53 2015 - [info]
Tue Nov 10 10:49:53 2015 - [info] Fetching dead master's binary logs..
Tue Nov 10 10:49:53 2015 - [info] Executing command on the dead master server7(server7:3307): save_binary_logs --command=save --start_file=master-bin.000004 --start_pos=120 --binlog_dir=/var/lib/mysql,/var/log/mysql --output_file=/var/log/masterha/app1/saved_master_binlog_from_server7_3307_20151110104953.binlog --handle_raw_binlog=1 --disable_log_bin=0 --manager_version=0.56
Creating /var/log/masterha/app1 if not exists.. ok.
Concat binary/relay logs from master-bin.000004 pos 120 to master-bin.000004 EOF into /var/log/masterha/app1/saved_master_binlog_from_server7_3307_20151110104953.binlog ..
Binlog Checksum enabled
Dumping binlog format description event, from position 0 to 120.. ok.
No need to dump effective binlog data from /var/lib/mysql/master-bin.000004 (pos starts 120, filesize 120). Skipping.
Binlog Checksum enabled
/var/log/masterha/app1/saved_master_binlog_from_server7_3307_20151110104953.binlog has no effective data events.
Event not exists.
Tue Nov 10 10:49:54 2015 - [info] Additional events were not found from the orig master. No need to save.
C: 選舉新的master, 因為所有slave接收到的master位點信息是一樣的，所以他們不用再做同步了。
隨機找了一台server6 promote成為new master.
Tue Nov 10 10:49:54 2015 - [info]
Tue Nov 10 10:49:54 2015 - [info] * Phase 3.3: Determining New Master Phase..
Tue Nov 10 10:49:54 2015 - [info]
Tue Nov 10 10:49:54 2015 - [info] Finding the latest slave that has all relay logs for recovering other slaves..
Tue Nov 10 10:49:54 2015 - [info] All slaves received relay logs to the same position. No need to resync each other.
Tue Nov 10 10:49:54 2015 - [info] Searching new master from slaves..
Tue Nov 10 10:49:54 2015 - [info] Candidate masters from the configuration file:
Tue Nov 10 10:49:54 2015 - [info] Non-candidate masters:
Tue Nov 10 10:49:54 2015 - [info] New master is server6(server6:3307)
Tue Nov 10 10:49:54 2015 - [info] Starting master failover..
Tue Nov 10 10:49:54 2015 - [info]
From:
server7(server7:3307) (current master)
+--server6(server6:3307)
+--server8(server8:3307)
To:
server6(server6:3307) (new master)
+--server8(server8:3307)
D: 新的master的差異日志生成階段
new master(server6)已經包含了所有 relay log (就是說dead master 的binlog和new master的 relay log沒有差異了--即所謂的deadmaster binlog 和new master relaylog 的差異)
Tue Nov 10 10:49:54 2015 - [info]
Tue Nov 10 10:49:54 2015 - [info] * Phase 3.3: New Master Diff Log Generation Phase..
Tue Nov 10 10:49:54 2015 - [info]
Tue Nov 10 10:49:54 2015 - [info] This server has all relay logs. No need to generate diff files from the latest slave.
Tue Nov 10 10:49:54 2015 - [info]
E: 新的master log應用階段，即把 new master的relay log和dead master的binlog差異在new master做一次重放。
然后獲取到new master的binlog 文件和位點信息（master-bin.000003, pos=120)至此 ，dead master和new master已經同步了。
Tue Nov 10 10:49:54 2015 - [info] * Phase 3.4: Master Log Apply Phase..
Tue Nov 10 10:49:54 2015 - [info]
Tue Nov 10 10:49:54 2015 - [info] *NOTICE: If any error happens from this phase, manual recovery is needed.
Tue Nov 10 10:49:54 2015 - [info] Starting recovery on server6(server6:3307)..
Tue Nov 10 10:49:54 2015 - [info] This server has all relay logs. Waiting all logs to be applied..
Tue Nov 10 10:49:54 2015 - [info] done.
Tue Nov 10 10:49:54 2015 - [info] All relay logs were successfully applied.
Tue Nov 10 10:49:54 2015 - [info] Getting new master's binlog name and position..
Tue Nov 10 10:49:54 2015 - [info] master-bin.000003:120
Tue Nov 10 10:49:54 2015 - [info] All other slaves should start replication from here. Statement should be: CHANGE MASTER TO MASTER_HOST='server6 or server6', MASTER_PORT=3307, MASTER_LOG_FILE='master-bin.000003', MASTER_LOG_POS=120, MASTER_USER='repl_user', MASTER_PASSWORD='xxx';
Tue Nov 10 10:49:54 2015 - [warning] master_ip_failover_script is not set. Skipping taking over new master IP address.
Tue Nov 10 10:49:54 2015 - [info] ** Finished master recovery successfully.
Tue Nov 10 10:49:54 2015 - [info] * Phase 3: Master Recovery Phase completed.
Tue Nov 10 10:49:54 2015 - [info]
階段4： 並行操作，對每一個slave .比較它和new master的relay log的位點差異，
然后把這個差異在slave 補全， 最后做change master to new master;
啟動slave;
Tue Nov 10 10:49:54 2015 - [info] * Phase 4: Slaves Recovery Phase..
Tue Nov 10 10:49:54 2015 - [info]
Tue Nov 10 10:49:54 2015 - [info] * Phase 4.1: Starting Parallel Slave Diff Log Generation Phase..
Tue Nov 10 10:49:54 2015 - [info]
Tue Nov 10 10:49:54 2015 - [info] -- Slave diff file generation on host server8(server8:3307) started, pid: 34574. Check tmp log /var/log/masterha/app1/server8_3307_20151110104953.log if it takes time..
Tue Nov 10 10:49:54 2015 - [info]
Tue Nov 10 10:49:54 2015 - [info] Log messages from server8 ...
Tue Nov 10 10:49:54 2015 - [info]
Tue Nov 10 10:49:54 2015 - [info] This server has all relay logs. No need to generate diff files from the latest slave.
Tue Nov 10 10:49:54 2015 - [info] End of log messages from server8.
Tue Nov 10 10:49:54 2015 - [info] -- server8(server8:3307) has the latest relay log events.
Tue Nov 10 10:49:54 2015 - [info] Generating relay diff files from the latest slave succeeded.
Tue Nov 10 10:49:54 2015 - [info]
Tue Nov 10 10:49:54 2015 - [info] * Phase 4.2: Starting Parallel Slave Log Apply Phase..
Tue Nov 10 10:49:54 2015 - [info]
Tue Nov 10 10:49:54 2015 - [info] -- Slave recovery on host server8(server8:3307) started, pid: 34576. Check tmp log /var/log/masterha/app1/server8_3307_20151110104953.log if it takes time..
Tue Nov 10 10:49:54 2015 - [info]
Tue Nov 10 10:49:54 2015 - [info] Log messages from server8 ...
Tue Nov 10 10:49:54 2015 - [info]
Tue Nov 10 10:49:54 2015 - [info] Starting recovery on server8(server8:3307)..
Tue Nov 10 10:49:54 2015 - [info] This server has all relay logs. Waiting all logs to be applied..
Tue Nov 10 10:49:54 2015 - [info] done.
Tue Nov 10 10:49:54 2015 - [info] All relay logs were successfully applied.
Tue Nov 10 10:49:54 2015 - [info] Resetting slave server8(server8:3307) and starting replication from the new master server6(server6:3307)..
Tue Nov 10 10:49:54 2015 - [info] Executed CHANGE MASTER.
Tue Nov 10 10:49:54 2015 - [info] Slave started.
Tue Nov 10 10:49:54 2015 - [info] End of log messages from server8.
Tue Nov 10 10:49:54 2015 - [info] -- Slave recovery on host server8(server8:3307) succeeded.
Tue Nov 10 10:49:54 2015 - [info] All new slave servers recovered successfully.
Tue Nov 10 10:49:54 2015 - [info]
階段5： : 重置new master上的slave info .， 啟動new master;
Tue Nov 10 10:49:54 2015 - [info] * Phase 5: New master cleanup phase..
Tue Nov 10 10:49:54 2015 - [info]
Tue Nov 10 10:49:54 2015 - [info] Resetting slave info on the new master..
Tue Nov 10 10:49:54 2015 - [info] server6: Resetting slave info succeeded.
Tue Nov 10 10:49:54 2015 - [info] Master failover to server6(server6:3307) completed successfully.
Tue Nov 10 10:49:54 2015 - [info]
階段6： 故障轉移報告：
----- Failover Report -----
app1: MySQL Master failover server7(server7:3307) to server6(server6:3307) succeeded
Master server7(server7:3307) is down!
Check MHA Manager logs at server9 for details.
Started automated(non-interactive) failover.
The latest slave server6(server6:3307) has all relay logs for recovery.
Selected server6(server6:3307) as a new master.
server6(server6:3307): OK: Applying all logs succeeded.
server8(server8:3307): This host has the latest relay log events.
Generating relay diff files from the latest slave succeeded.
server8(server8:3307): OK: Applying all logs succeeded. Slave started, replicating from server6(server6:3307)
server6(server6:3307): Resetting slave info succeeded.
Master failover to server6(server6:3307) completed successfully.
簡而言之 ，mysql-master-ha的做法是：
1、檢測master 不可用。（ping 端口+ 用戶自己擴展腳本確認的方式）
2、從多個slaves中選擇relay log同步到最大master binlog 的做為new master , 即找到latest slave;
3、從dead master 找latest slave和它的master binlog 日志差異，然后在new master relay log 重放。 ----至此， new master和dead master一致。
4、對其他slave ，找它們和new master relay log的差異， 並重放 ,然后change master to new master -至此，new master和slaves一致。
至此， 完成mysql master的故障轉移。
但是mysql-master-ha的前提是old master物理機器的ssh連通性是可用的， 即物理節點本身可用。
而阿里的mysql ha 采用的做法如下：
1、 zk 節點探測mysql 可用性 。
2、如果dead master可以啟動， 則重新啟動， 不做master切換。-- 最理想的方式 。
3、如果dead master不可啟動了，則slaves利用zk的搶鎖機制（每一個mysql slave的zk agent 在/lock目錄下創建臨時順序zk 節點，並判斷自己是否是/lock下的最小節點，如果是， 則搶鎖成功，如果否 ，則監控/lock子節點，等待搶鎖） 成為new master ,然后dead master利用binlog回滾（對dead master和new master的日志差異做逆向重放） 和new master保持一致 。
4、如果在slave 的relay log 的exec 和read 位點有差異時， 處理方案： reset slave & change master to new master .
但是ppt有幾點疑問 ，第3步可以保證搶到zk鎖的是 latest slave么 ？？？？？？？第4步slave直接做reset slave 把relay log清理掉，重新和 new master做同步，效率是否會太低了????? 另外，從ppt來看，這種搶zk鎖的操作 ， 從邏輯上來看似乎也沒有問題，通過dead master的binlog回滾，保持old 和new master的一致 ，然后其他slaves和new master保持一致即可， 但是理解起來有點疑惑。
騰訊的做法：
1、利用半同步保證master的binlog 同步到slave並寫入relay log以后， master 才返回事務commit給client .
2、也是zk agent監控master 可用性，當master不可用時 ，數據庫proxy中的路由信息改為master 不可用。
3、 slave上報relay log中讀到的最新master binlog ,scheduler從中選擇一個latest slave(讀到old master binlog 最大位點的)作為new master, 並要求new master
重放完relay log （表示存儲引擎中已經有relay log所表示的數據庫變更了 ）
4、修改其他slave的master變成new master;
值得注意的是， 騰訊的TDSQL 文檔中沒有提到new master 和old master的日志差異，以及new master和其他slave的relay log 日志差異問題，
因為前者通過半同步得到了解決，后者通過其他slave的change master to new master以后，進行位點追趕。比如 new master已經到
master-bin.00002
135
而slave才到master-bin.00001的120 ,則直接change to new master master-bin.00001 ,120 進行位點追趕。
5、 修改new master的slave info .
6、修改代理的路由信息，master完成failover . 。
這里， 騰訊TDSQL 的做法理解起來更直觀一些。
總體比較來看，在old master 和new master的binlog同步上，都是一致的（即要保證old master和new master 的完全一致 ,mysql-master-ha用了perl腳本批量做new master的位點追趕， 騰訊用了半同步，阿里用了old master的日志回滾） 。而在new master和其他slave的slave log同步上， mysql-master-ha要求slave 和new master完全一致，騰訊和阿里則沒有要求（mysql-master-ha強調數據的一致性，阿里和騰訊強調的是系統的可用性， 因為本身這種master-slave的架構在使用上就要注意主從同步的延遲性問題）。
總體個人覺得，騰訊的TDSQL設計上更簡明些， 而mysql-master-ha在做完master failover以后，需要再通知數據庫代理層修改主備路由信息。這個是后話 （要自己實現TDSQL的功能 ） 。