一、MGR架構的介紹
1、簡介
MGR（MySQL Group Replication）是MySQL5.7.17版本引進來的一個數據庫高可用架構，解決了傳統異步復制和半同步復制的缺陷（主從數據一致性的問題），MGR依靠分布式一致性協議PAXOS，實現了主從數據庫的一致性。
PAXOS協議：是一種基於消息傳遞的一致性算法。MGR中由若干個節點共同組成一個組復制，一個事物的提交，必須經過組內大多數節點（N/2 + 1）投票並通過，才能提交。
2、MGR特點
* 高一致性：基於分布式PAXOS協議實現組復制，保證數據的一致性；
* 高容錯性：故障自動檢測機制，只要有半數節點存活MGR就可以繼續工作；
* 高擴展性：節點的增加與移除會自動更新組成員信息，新節點加入后，自動從其他節點同步增量數據，直到與其他節點數據一致；
* 高靈活性：提供單主模式和多主模式。單主模式在主庫宕機后能夠自動選主（根據MEMBER_ID的排序情況，最小的升級為主庫），所有寫操作都在主庫上進行，多主模式支持多節點寫入；
3、MGR故障檢測機制
MGR中有一個故障檢測機制，會提供某節點可能死掉的信息，某個server無響應時觸發猜測，組內其余成員進行協調以排除該節點，與它隔離開來，該節點無法同步其他節點傳送過來的binlog，也無法執行任何本地事物。
4、搭建MGR的前提條件
* 存儲引擎必須是Innodb（支持事物處理），並且每張表一定要有主鍵，用於解決write沖突；
* 必須打開GTID特性，binlog日志格式必須設置為ROW；
* 目前一個MGR集群組最多支持9個節點；
* 多主模式不支持SERIALIZABLE事物隔離級別；
二、MGR的搭建
1、IP配置
| 機器名稱 | IP | 服務器角色 | 備注 |
| node1 | 192.168.232.42 | MGR節點1 | 操作系統redhat7.5；MySQL5.7.22 |
| node2 | 192.168.232.43 | MGR節點2 | 操作系統redhat7.5；MySQL5.7.22 |
| node3 | 192.168.232.44 | MGR節點3 | 操作系統redhat7.5；MySQL5.7.22 |
2、配置IP映射
在各節點的/etc/hosts文件中配置如下內容：
192.168.232.42 node1 192.168.232.43 node2 192.168.232.44 node3
3、所有節點禁用SELinux
4、配置3個節點的my.cnf文件
[mysqld] datadir=/mysql/mysql5.7/data basedir=/mysql/mysql5.7 #socket=/var/lib/mysql/mysql.sock user=mysql port=3308 character-set-server=utf8 # skip-grant-tables # Disabling symbolic-links is recommended to prevent assorted security risks symbolic-links=0 # Settings user and group are ignored when systemd is used. # If you need to run mysqld under a different user or group, # customize your systemd unit file for mariadb according to the # instructions in http://fedoraproject.org/wiki/Systemd server_id=1 gtid_mode=ON enforce_gtid_consistency=ON master_info_repository=TABLE relay_log_info_repository=TABLE binlog_checksum=NONE log_slave_updates=ON log_bin=mysql-bin binlog_format=ROW # 這里binlog的模式必須是ROW transaction_write_set_extraction=XXHASH64 # 記錄事物的算法 loose-group_replication_group_name="b6cf6565-c415-4c4d-a4f0-49a596f98fca" # group的名稱，必須是一個有效的uuid，用來區分不同的group loose-group_replication_start_on_boot=off # 是否在啟動MySQL時啟動組復制，這里設置不自動啟動 loose-group_replication_local_address="192.168.232.42:33061" # 本地IP地址字符串，用於組內節點之間的連接 loose-group_replication_group-seeds="192.168.232.42:33061,192.168.232.43:33062,192.168.232.44:33063" # 種子服務器的配置 loose-group_relication_bootstrap_group=off # 配置是否自動引導組，第一次搭建MGR的時候使用 loose-group_replication_single_primary_mode=TRUE # 是否啟動單主模式 loose-group_replication_enforce_update_everywhere_checks=FALSE # 是否啟動多主模式 [mysqld_safe] log-error=/mysql/mysql5.7/log/mysqld.log #pid-file=/mysql/mysql5.7/run/mysqld.pid
注：
* 變量使用loose-前綴表示Server啟用時沒有加載復制插件也能繼續啟動
* 三個節點的loose-group_replication_group_name 必須一樣，要是有多個group的話，每個group應該不同
三個節點my.cnf配置如下：
5、安裝組復制的引擎
命令：
INSTALL PLUGIN group_replication SONAME 'group_replication.so';
查看是否安裝成功：
show plugins;
注：三個節點都安裝該插件。
6、創建同步的賬號
命令：
mysql> grant replication slave on *.* to 'repl_user1'@'192.168.232.%' identified by '$a123456'; Query OK, 0 rows affected, 1 warning (0.05 sec) mysql> flush privileges; Query OK, 0 rows affected (0.02 sec) mysql> select user,host from user; +---------------+---------------+ | user | host | +---------------+---------------+ | root | % | | repl_user1 | 192.168.232.% | | mysql.session | localhost | | mysql.sys | localhost | +---------------+---------------+ 4 rows in set (0.01 sec) mysql>
注：因為每個節點都有可能成為master，所以3個節點上都需要創建同步的用戶，我這里host配置為192.168.232.%意思是192.168.232.網段可以使用該用戶。
三、搭建MGR的單主模式
1、先把node1加入group組
注：
* 由於是第一個加入Group，需要啟動group_relication_bootstrap_group，加入成功后再關閉就行；
* 一個組Group是在節點參數為group_replication_bootstrap_group為on的條件下執行start group_replication產生的，如果要加入現有的Group，節點需要確保group_replication_bootstrap_group為off；
（1）開啟group_relication_bootstrap_group
mysql> show global variables like '%group_replication_bootstrap_group%'; +-----------------------------------+-------+ | Variable_name | Value | +-----------------------------------+-------+ | group_replication_bootstrap_group | OFF | +-----------------------------------+-------+ 1 row in set (0.00 sec) mysql> set global group_replication_bootstrap_group=on; Query OK, 0 rows affected (0.00 sec) mysql> show global variables like '%group_replication_bootstrap_group%'; +-----------------------------------+-------+ | Variable_name | Value | +-----------------------------------+-------+ | group_replication_bootstrap_group | ON | +-----------------------------------+-------+ 1 row in set (0.00 sec) mysql>
（2）配置MGR
mysql> CHANGE MASTER TO MASTER_USER='repl_user1', MASTER_PASSWORD='$a123456' FOR CHANNEL 'group_replication_recovery'; Query OK, 0 rows affected, 2 warnings (0.46 sec) mysql>
（3）啟動MGR
mysql> start group_replication; Query OK, 0 rows affected (2.33 sec) mysql> select * from performance_schema.replication_group_members ; +---------------------------+--------------------------------------+-------------+-------------+--------------+ | CHANNEL_NAME | MEMBER_ID | MEMBER_HOST | MEMBER_PORT | MEMBER_STATE | +---------------------------+--------------------------------------+-------------+-------------+--------------+ | group_replication_applier | 9b014152-37a6-11eb-89b5-000c29cddf72 | node1 | 3308 | ONLINE | +---------------------------+--------------------------------------+-------------+-------------+--------------+ 1 row in set (0.00 sec) mysql>
（4）關閉 group_replication_bootstrap_group
mysql> show global variables like '%group_replication_bootstrap_group%'; +-----------------------------------+-------+ | Variable_name | Value | +-----------------------------------+-------+ | group_replication_bootstrap_group | ON | +-----------------------------------+-------+ 1 row in set (0.00 sec) mysql> set global group_replication_bootstrap_group=off; Query OK, 0 rows affected (0.00 sec) mysql> show global variables like '%group_replication_bootstrap_group%'; +-----------------------------------+-------+ | Variable_name | Value | +-----------------------------------+-------+ | group_replication_bootstrap_group | OFF | +-----------------------------------+-------+ 1 row in set (0.01 sec) mysql>
注：啟動成功后，發現data目錄下面多了幾個_applier和_recovery文件
* _applier：保存同個組內其他Server的binlog信息；
* _recovery：用來存儲需要恢復的binlog信息，新加入的組成員是通過group_replication_recovery來恢復數據的，所需要的binlog日志都存放在_recovery系統文件中；
2、在node1上造數據
mysql> create database customer default character set utf8; Query OK, 1 row affected (0.01 sec) mysql> use customer; Database changed mysql> create table person (id int(10) primary key auto_increment not null,username varchar(20),age int(10)) ENGINE=InnoDB DEFAULT CHARSET=utf8; Query OK, 0 rows affected (0.04 sec) mysql> insert into person(username,age) values('jack',19); Query OK, 1 row affected (0.00 sec) mysql> insert into person(username,age) values('rose',20); Query OK, 1 row affected (0.00 sec) mysql> select * from person; +----+----------+------+ | id | username | age | +----+----------+------+ | 1 | jack | 19 | | 2 | rose | 20 | +----+----------+------+ 2 rows in set (0.00 sec) mysql>
3、把node2加入到mgr組中
mysql> CHANGE MASTER TO MASTER_USER='repl_user1', MASTER_PASSWORD='$a123456' FOR CHANNEL 'group_replication_recovery'; Query OK, 0 rows affected, 2 warnings (0.07 sec) mysql> start group_replication; Query OK, 0 rows affected (4.88 sec) mysql>
加入的時候，遇到的錯誤參考文章最后的相關錯誤
在node1上查看組內有幾個節點：
mysql> select * from performance_schema.replication_group_members ; +---------------------------+--------------------------------------+-------------+-------------+--------------+ | CHANNEL_NAME | MEMBER_ID | MEMBER_HOST | MEMBER_PORT | MEMBER_STATE | +---------------------------+--------------------------------------+-------------+-------------+--------------+ | group_replication_applier | 66720a6a-37a8-11eb-befc-000c29c918c1 | node2 | 3308 | RECOVERING | | group_replication_applier | 9b014152-37a6-11eb-89b5-000c29cddf72 | node1 | 3308 | ONLINE | +---------------------------+--------------------------------------+-------------+-------------+--------------+ 2 rows in set (0.00 sec) mysql>
4、把node3加入到MGR組中
mysql> CHANGE MASTER TO MASTER_USER='repl_user1', MASTER_PASSWORD='$a123456' FOR CHANNEL 'group_replication_recovery'; Query OK, 0 rows affected, 2 warnings (0.01 sec) mysql> start group_replication; Query OK, 0 rows affected (3.39 sec) mysql>
在node1上查看組內有幾個節點：
mysql> select * from performance_schema.replication_group_members ; +---------------------------+--------------------------------------+-------------+-------------+--------------+ | CHANNEL_NAME | MEMBER_ID | MEMBER_HOST | MEMBER_PORT | MEMBER_STATE | +---------------------------+--------------------------------------+-------------+-------------+--------------+ | group_replication_applier | 02c2cfed-37a9-11eb-b238-000c2927b3e8 | node3 | 3308 | ONLINE | | group_replication_applier | 66720a6a-37a8-11eb-befc-000c29c918c1 | node2 | 3308 | ONLINE | | group_replication_applier | 9b014152-37a6-11eb-89b5-000c29cddf72 | node1 | 3308 | ONLINE | +---------------------------+--------------------------------------+-------------+-------------+--------------+ 3 rows in set (0.00 sec) mysql>
至此，3個節點的單主模式已經搭建好了，node1是master節點，提供讀寫，node2和node3是只讀節點，提供只讀操作。
5、MGR單主模式災備切換
（1）模擬主庫宕機，因為我這里是虛擬機，直接shutdown即可
shutdown -h
（2）查看哪個節點升級為主庫
mysql> select * from performance_schema.replication_group_members ; +---------------------------+--------------------------------------+-------------+-------------+--------------+ | CHANNEL_NAME | MEMBER_ID | MEMBER_HOST | MEMBER_PORT | MEMBER_STATE | +---------------------------+--------------------------------------+-------------+-------------+--------------+ | group_replication_applier | 02c2cfed-37a9-11eb-b238-000c2927b3e8 | node3 | 3308 | ONLINE | | group_replication_applier | 66720a6a-37a8-11eb-befc-000c29c918c1 | node2 | 3308 | ONLINE | +---------------------------+--------------------------------------+-------------+-------------+--------------+ 2 rows in set (0.01 sec) mysql> select * from performance_schema.global_status where variable_name like '%group%'; +----------------------------------+--------------------------------------+ | VARIABLE_NAME | VARIABLE_VALUE | +----------------------------------+--------------------------------------+ | group_replication_primary_member | 02c2cfed-37a9-11eb-b238-000c2927b3e8 | +----------------------------------+--------------------------------------+ 1 row in set (0.02 sec) mysql>
注：performance_schema.replication_group_members的結果是按照MEMBER_ID升序排列的，主庫的升級的時候，MEMBER_ID最小的升級為主庫。
（3）查看node2和node3的讀寫狀態
--1、node2（只讀的） mysql> show global variables like '%super%'; +-----------------+-------+ | Variable_name | Value | +-----------------+-------+ | super_read_only | ON | +-----------------+-------+ 1 row in set (0.01 sec) --2、node3（讀寫的，說明node3是主） mysql> show global variables like '%super%'; +-----------------+-------+ | Variable_name | Value | +-----------------+-------+ | super_read_only | OFF | +-----------------+-------+ 1 row in set (0.02 sec)
（4）測試現在的主從數據是否可以同步
--node3上插入數據 mysql> use customer; Database changed mysql> select * from person; +----+----------+------+ | id | username | age | +----+----------+------+ | 1 | jack | 19 | | 2 | jack | 19 | | 3 | jack | 19 | | 4 | jack | 19 | | 5 | jack | 19 | | 6 | jack | 19 | | 7 | jack | 19 | +----+----------+------+ 7 rows in set (0.00 sec) mysql> insert into person(username,age) values('sdsd',20); Query OK, 1 row affected (0.02 sec) mysql> select * from person; +----+----------+------+ | id | username | age | +----+----------+------+ | 1 | jack | 19 | | 2 | jack | 19 | | 3 | jack | 19 | | 4 | jack | 19 | | 5 | jack | 19 | | 6 | jack | 19 | | 7 | jack | 19 | | 8 | sdsd | 20 | +----+----------+------+ 8 rows in set (0.00 sec) --node2上查看是否同步 mysql> use customer; Database changed mysql> select * from person; +----+----------+------+ | id | username | age | +----+----------+------+ | 1 | jack | 19 | | 2 | jack | 19 | | 3 | jack | 19 | | 4 | jack | 19 | | 5 | jack | 19 | | 6 | jack | 19 | | 7 | jack | 19 | | 8 | sdsd | 20 | +----+----------+------+ 8 rows in set (0.00 sec)
注：可以看到數據可以從新主同步數據到從節點
（5）node1重啟后，重新加入到MGR組中
mysql> CHANGE MASTER TO MASTER_USER='repl_user1', MASTER_PASSWORD='$a123456' FOR CHANNEL 'group_replication_recovery'; Query OK, 0 rows affected, 2 warnings (0.01 sec) mysql> start group_replication; Query OK, 0 rows affected (3.03 sec) mysql> select * from performance_schema.replication_group_members ; +---------------------------+--------------------------------------+-------------+-------------+--------------+ | CHANNEL_NAME | MEMBER_ID | MEMBER_HOST | MEMBER_PORT | MEMBER_STATE | +---------------------------+--------------------------------------+-------------+-------------+--------------+ | group_replication_applier | 02c2cfed-37a9-11eb-b238-000c2927b3e8 | node3 | 3308 | ONLINE | | group_replication_applier | 66720a6a-37a8-11eb-befc-000c29c918c1 | node2 | 3308 | ONLINE | | group_replication_applier | 9b014152-37a6-11eb-89b5-000c29cddf72 | node1 | 3308 | ONLINE | +---------------------------+--------------------------------------+-------------+-------------+--------------+ 3 rows in set (0.00 sec) mysql> use customer; Reading table information for completion of table and column names You can turn off this feature to get a quicker startup with -A Database changed mysql> select * from person; +----+----------+------+ | id | username | age | +----+----------+------+ | 1 | jack | 19 | | 2 | jack | 19 | | 3 | jack | 19 | | 4 | jack | 19 | | 5 | jack | 19 | | 6 | jack | 19 | | 7 | jack | 19 | | 8 | sdsd | 20 | +----+----------+------+ 8 rows in set (0.00 sec)
（6）查看Group的當前視圖
注：16079120577135158:12---- 表示發生了12次視圖切換
四、啟動MGR為多主模式
1、多主模式的搭建
如果從頭搭建多主模式的話，和單主模式的搭建步驟基本一致，只是把下面的參數改一下就行了
--動態配置 set global group_replication_single_primary_mode=FALSE set global group_replication_enforce_update_everywhere_checks=TRUE --配置文件修改 loose-group_replication_single_primary_mode=FALSE loose-group_replication_enforce_update_everywhere_checks=TRUE
2、單主模式修改為多主模式
已經是單主模式要想修改為多主模式，直接動態修改參數會報錯，如下：
mysql> set global group_replication_single_primary_mode=off; ERROR 3093 (HY000): Cannot change into or from single primary mode while Group Replication is running. mysql>
所以，需要停掉group_replication，才能從單主模式修改為多主。操作步驟如下
（1）組成員逐個退出group
--node3上退出group mysql> stop group_replication; Query OK, 0 rows affected (9.58 sec) --node2上退出group mysql> stop group_replication; Query OK, 0 rows affected (9.57 sec) --node1上退出group mysql> stop group_replication; Query OK, 0 rows affected (9.57 sec)
節點的狀態都是offline：
mysql> select * from performance_schema.replication_group_members ; +---------------------------+--------------------------------------+-------------+-------------+--------------+ | CHANNEL_NAME | MEMBER_ID | MEMBER_HOST | MEMBER_PORT | MEMBER_STATE | +---------------------------+--------------------------------------+-------------+-------------+--------------+ | group_replication_applier | 9b014152-37a6-11eb-89b5-000c29cddf72 | node1 | 3308 | OFFLINE | +---------------------------+--------------------------------------+-------------+-------------+--------------+ 1 row in set (0.00 sec)
（2）修改group_replication_single_primary_mode參數為off（node1、node2、node3上都要執行）
mysql> set global group_replication_single_primary_mode=FALSE; Query OK, 0 rows affected (0.01 sec) mysql> set global group_replication_enforce_update_everywhere_checks=TRUE; Query OK, 0 rows affected (0.00 sec)
（3）在node1上新增數據，模擬生成環境業務不停
mysql> use customer; Database changed mysql> create table tb2(id int primary key auto_increment not null,name varchar(100)); Query OK, 0 rows affected (0.46 sec) mysql> mysql> insert into tb2(name) values('wade'); Query OK, 1 row affected (0.00 sec) mysql> select * from tb2; +----+------+ | id | name | +----+------+ | 1 | wade | +----+------+ 1 row in set (0.00 sec)
（4）逐個啟動group
--node1加入group mysql> show global variables like '%group_replication_bootstrap_group%'; +-----------------------------------+-------+ | Variable_name | Value | +-----------------------------------+-------+ | group_replication_bootstrap_group | OFF | +-----------------------------------+-------+ 1 row in set (0.01 sec) mysql> set global group_replication_bootstrap_group=ON; Query OK, 0 rows affected (0.00 sec) mysql> CHANGE MASTER TO MASTER_USER='repl_user1', MASTER_PASSWORD='$a123456' FOR CHANNEL 'group_replication_recovery'; Query OK, 0 rows affected, 2 warnings (0.02 sec) mysql> start group_replication; Query OK, 0 rows affected (2.04 sec) mysql> set global group_replication_bootstrap_group=off; Query OK, 0 rows affected (0.00 sec) mysql> select * from performance_schema.replication_group_members ; +---------------------------+--------------------------------------+-------------+-------------+--------------+ | CHANNEL_NAME | MEMBER_ID | MEMBER_HOST | MEMBER_PORT | MEMBER_STATE | +---------------------------+--------------------------------------+-------------+-------------+--------------+ | group_replication_applier | 9b014152-37a6-11eb-89b5-000c29cddf72 | node1 | 3308 | ONLINE | +---------------------------+--------------------------------------+-------------+-------------+--------------+ 1 row in set (0.00 sec) mysql> --node2加入group mysql> CHANGE MASTER TO MASTER_USER='repl_user1', MASTER_PASSWORD='$a123456' FOR CHANNEL 'group_replication_recovery'; Query OK, 0 rows affected, 2 warnings (0.01 sec) mysql> start group_replication; Query OK, 0 rows affected (4.07 sec) --node3加入group mysql> CHANGE MASTER TO MASTER_USER='repl_user1', MASTER_PASSWORD='$a123456' FOR CHANNEL 'group_replication_recovery'; Query OK, 0 rows affected, 2 warnings (0.02 sec) mysql> start group_replication; Query OK, 0 rows affected (3.09 sec) --node1上查看組成員 mysql> select * from performance_schema.replication_group_members ; +---------------------------+--------------------------------------+-------------+-------------+--------------+ | CHANNEL_NAME | MEMBER_ID | MEMBER_HOST | MEMBER_PORT | MEMBER_STATE | +---------------------------+--------------------------------------+-------------+-------------+--------------+ | group_replication_applier | 02c2cfed-37a9-11eb-b238-000c2927b3e8 | node3 | 3308 | ONLINE | | group_replication_applier | 66720a6a-37a8-11eb-befc-000c29c918c1 | node2 | 3308 | ONLINE | | group_replication_applier | 9b014152-37a6-11eb-89b5-000c29cddf72 | node1 | 3308 | ONLINE | +---------------------------+--------------------------------------+-------------+-------------+--------------+ 3 rows in set (0.00 sec)
（5）查看數據是否同步
--node2上查看 mysql> select * from customer.tb2; +----+------+ | id | name | +----+------+ | 1 | wade | +----+------+ 1 row in set (0.00 sec) --node3上查看 mysql> select * from customer.tb2; +----+------+ | id | name | +----+------+ | 1 | wade | +----+------+ 1 row in set (0.00 sec)
注：可以看到數據已經同步過去了
（6）查看各個節點的讀寫狀態
mysql> show global variables like '%super_read_only%'; +-----------------+-------+ | Variable_name | Value | +-----------------+-------+ | super_read_only | OFF | +-----------------+-------+ 1 row in set (0.00 sec)
注：發現3個節點的super_read_only狀態都是off，都是讀寫的，這時候再查看global_status狀態發現為空，無法確定主庫是哪個，因為都是主庫
到這里，MySQL之MGR的多主模式已經搭建完成了。
五、運維常用的SQL
1、查看group成員
select * from performance_schema.replication_group_members;
2、查看當前SERVER在GROUP中的同步情況（查看applier通道的同步情況）
select * from performance_schema.replication_group_member_stats;
3、查看當前server中各個通道的使用情況
注：applier通道是一定有顯示，recovery通道看是否使用過，如果有則顯示，沒有則不顯示。
select * from performance_schema.replication_connection_status;
4、查看當前server中各個通道是否啟用
select * from performance_schema.replication_applier_status;
5、查看global參數的狀態
select * from performance_schema.global_status;
六、遇到的問題總結
1、加入MGR組時報的錯誤1
（1）錯誤如下：
[ERROR] Plugin group_replication reported: 'This member has more executed transactions than those present in the group. Local transactions: 02c2cfed-37a9-11eb-b238-000c2927b3e8:1-9 > Group transactions: b6cf6565-c415-4c4d-a4f0-49a596f98fca:1-14' 2020-12-14T02:35:17.114156Z 0 [ERROR] Plugin group_replication reported: 'The member contains transactions not present in the group. The member will now exit the group.'
（2）解決辦法
原因是node2節點包含了非group內的事務。查看node2的數據，確實有與其他節點不一樣的數據。應該是沒啟動GR服務時，被當做獨立庫操作了。
在node2節點上，重置master的信息，重新加入MGR，再次start解決了。
mysql> start group_replication; ERROR 3092 (HY000): The server is not configured properly to be an active member of the group. Please see more details on error log. mysql> reset master; Query OK, 0 rows affected (0.02 sec) mysql> CHANGE MASTER TO MASTER_USER='repl_user1', MASTER_PASSWORD='$a123456' FOR CHANNEL 'group_replication_recovery'; Query OK, 0 rows affected, 2 warnings (0.01 sec) mysql> start group_replication; Query OK, 0 rows affected (3.39 sec)
2、加入MGR組時報的錯誤2
（1）錯誤截圖如下
（2）解決辦法
出現這個問題的原因是節點1、2、3中的group_replication_group_name使用的是每台機器的uuid，但是這個配置需要3台機器保持一致。
修改三個節點的group_replication_group_name一致后，問題解決。