MariaDB Galera Cluster環境搭建及高可用測試

一、服務器概況
Galera Cluster需要至少三個節點，在此次實驗過程中，三個節點IP地址：
192.168.56.101
192.168.56.102
192.168.56.103
OS為centos 7.2
服務器配置：
4G 內存，2核CPU，20G數據磁盤空間。

關閉防火牆:

service iptables stop
systemctl stop firewalld
chkconfig iptables off

關閉SELinux:

/etc/selinux/config中的SELINUX=disabled

修改主機名

vi /etc/hostname MariaDB01 MariaDB02 MariaDB03 systemctl restart systemd-hostnamed

三台主機上加入/etc/hosts:

192.168.56.101 MariaDB01 192.168.56.102 MariaDB02 192.168.56.103 MariaDB03

 

二、MariaDB Galera Cluster軟件二進制安裝

  官網下載：https://downloads.mariadb.org/   10.0.29

  初始化節點1

su - mysql cd /app/mysql/ vi my.cnf

--my.cnf
[client]
port = 3306
socket = /tmp/mysql.sock
[mysqld]
port = 3306
socket = /tmp/mysql.sock
character-set-server=utf8
skip-name-resolve
basedir=/app/mysql
datadir=/data/mysqldata
back_log = 50
binlog_cache_size = 1M
max_heap_table_size = 64M
read_buffer_size = 2M
read_rnd_buffer_size = 16M
sort_buffer_size = 8M
join_buffer_size = 8M
thread_cache_size = 8
thread_concurrency = 8
ft_min_word_len = 4
binlog_format=ROW
default-storage-engine=innodb
innodb_autoinc_lock_mode=2
thread_stack = 240K
transaction_isolation = REPEATABLE-READ
tmp_table_size = 64M
log-bin=/data/binlog/mysql-bin
slow_query_log
long_query_time = 2
log_error=/data/logs/error.log
slow_query_log_file=/data/logs/slow.log
server-id = 2
key_buffer_size = 32M
bulk_insert_buffer_size = 64M
myisam_sort_buffer_size = 128M
myisam_max_sort_file_size = 10G
myisam_repair_threads = 1
myisam_recover
innodb_additional_mem_pool_size = 16M
innodb_buffer_pool_size = 1G
innodb_data_file_path = ibdata1:500M;ibdata2:16M:autoextend
innodb_file_per_table = 1
innodb_support_xa = on
sync_binlog = 0
table_definition_cache=65535
table_open_cache=4000
max_connections=3100
max_user_connections=3000
max_connect_errors=65536
max_allowed_packet=16M
innodb_strict_mode = 1
innodb_open_files = 4000
innodb_file_format = Barracuda
innodb_file_format_max = Barracuda
innodb_flush_method = O_DIRECT
expire_logs_days=3
innodb_data_home_dir = /data/mysqldata
innodb_write_io_threads = 8
innodb_read_io_threads = 8
innodb_thread_concurrency = 16
innodb_flush_log_at_trx_commit = 1
innodb_log_buffer_size = 128M
innodb_log_file_size = 256M
innodb_log_files_in_group = 3
innodb_log_group_home_dir = /data/redolog
innodb_max_dirty_pages_pct = 90
innodb_flush_method=O_DIRECT
innodb_lock_wait_timeout = 120

[galera]
bind-address=0.0.0.0
wsrep_provider=none
wsrep_node_address='192.168.56.101'
wsrep_provider=/app/mariadb-galera-10.0.29-linux/lib/galera/libgalera_smm.so
wsrep_provider_options='gcache.size=1999M'
wsrep_cluster_name='galera_cluster'
wsrep_cluster_address ='gcomm://192.168.56.101,192.168.56.102,192.168.56.103'
wsrep_node_name='MariaDB01'
wsrep_slave_threads=4
wsrep_certify_nonPK=1
wsrep_max_ws_rows=131072
wsrep_max_ws_size=1073741824
wsrep_debug=0
wsrep_convert_LOCK_to_trx=0
wsrep_retry_autocommit=1
wsrep_auto_increment_control=1
wsrep_replicate_myisam=1
wsrep_drupal_282555_workaround=0
wsrep_causal_reads=0
wsrep_sst_method=rsync
wsrep_sst_auth= sst_user:dbpass

[mysqldump]
quick
max_allowed_packet = 16M
[mysql]
no-auto-rehash
[myisamchk]
key_buffer_size = 512M
sort_buffer_size = 512M
read_buffer = 8M
write_buffer = 8M
[mysqlhotcopy]
interactive-timeout
[mysqld_safe]
open-files-limit = 8192

　　

./scripts/mysql_install_db --defaults-file=my.cnf ./bin/mysqld_safe --defaults-file=my.cnf --wsrep-new-cluster &

      初始化節點2、3

./scripts/mysql_install_db --defaults-file=my.cnf ./bin/mysqld_safe --defaults-file=my.cnf &

     為集群創建帳號

mysql> DELETE FROM mysql.user WHERE user=''; mysql> GRANT ALL ON *.* TO 'root'@'%' IDENTIFIED BY 'dbpass'; mysql> GRANT USAGE ON *.* to sst_user@'%' IDENTIFIED BY 'dbpass'; mysql> GRANT ALL PRIVILEGES on *.* to sst_user@'%'; mysql> FLUSH PRIVILEGES;

 

 三、日常維護

       1. 集群啟動
         啟動第一個節點：service mysql start --wsrep-new-cluster
         啟動其它節點：server mysql start
         關閉節點：server mysql stop
       2. 通過查看4567端口確認集群是否啟動
         netstat -plantu | grep mysqld
       3. 查詢 galera插件是否已啟用

SHOW status LIKE 'wsrep_ready'; SHOW VARIABLES LIKE 'wsrep_cluster_address';

       4. 檢查集群是否正常

          查看Galera集群狀態: mysql> show status like 'wsrep%';

(1)SHOW GLOBAL STATUS LIKE 'wsrep_cluster_state_uuid'; 集群中每個節點的值必須一致。 (2)SHOW GLOBAL STATUS LIKE 'wsrep_cluster_size'; 顯示集群的規模，此例應該顯示”3”。 (3)SHOW GLOBAL STATUS LIKE 'wsrep_cluster_status'; 顯示”Primary”為正常。 (4)SHOW GLOBAL STATUS LIKE 'wsrep_ready'; 顯示”ON”為正常。 (5) SHOW GLOBAL STATUS LIKE 'wsrep_connected'; 顯示”ON”為正常。 (6) SHOW GLOBAL STATUS LIKE 'wsrep_local_state_comment'; 顯示” Joined”為正常。

        cluster完整性檢測

        status 變量:
        wsrep_cluster_state_uuid: cluster中的所有node的該變量的值必須一樣, 不一樣表示node沒有連接到cluster。
        wsrep_cluster_conf_id: 次變量用來表示node是否在它相應的cluster中。cluster中所有node的該變量值應該一樣， 不一樣則表示nodes被分隔開了, node恢復的時候該變量也會恢復。
        wsrep_cluster_size: 表示cluster中有多少node節點， 等於預期的數量則表示所有node連接到了cluster。
        wsrep_cluster_status: 正常情況下值為Primary, 如果不為Primay,則該node當前不能操作(歸咎於多成員關系的改變和quorum的缺失), 同時也可能滿足split-brain的條件。
        如果cluster中沒有node 連接上(connected) PC(就是所有node屬於同一部分,但是node都是non-primary狀態),

 

        wsrep_sst_method的參數說明：

Method      Speed    Blocks Donor?    Available on Live Node    Type       Requires root access to database server? mysqldump slow yes yes logical  donor and joiner rsync fastest yes no physical none xtrabackup fast briefly no physical donor only

      rsync在數據同步（SST和IST)的時候，速度最快，但是會鎖住提供數據的節點，xtrabackup只會短暫的鎖住節點。

 

 四、高可用測試

     1、節點數據同步寫入

     2、故障節點修復后數據自行修復

附錄

MariaDB Galera Cluste安裝報錯處理

170525 23:19:48 [ERROR] WSREP: wsrep_load(): dlopen(): libssl.so.6: cannot open shared object file: No such file or directory
170525 23:19:48 [ERROR] WSREP: wsrep_load(/app/mariadb-galera-10.0.29-linux/lib/galera/libgalera_smm.so) failed: Invalid argument (22). Reverting to no provider.
cd /app/mysql/lib/galera/
ldd ./libgalera_smm.so
cd /usr/lib64
ln -s libcrypto.so.1.0.1e libcrypto.so.6
ln -s libssl.so.1.0.1e libssl.so.6

 

170526 0:04:07 [Warning] WSREP: Gap in state sequence. Need state transfer.
170526 0:04:07 [Note] WSREP: Running: 'wsrep_sst_rsync --role 'joiner' --address '192.168.56.103' --datadir '/data/mysqldata/' --defaults-file '/app/mariadb-galera-10.0.29-linux/my.cnf' --defaults-group-suffix '' --parent '2832' --binlog '/data/binlog/mysql-bin' '
which: no rsync in (/usr/sbin:/sbin:/app/mariadb-galera-10.0.29-linux//bin:/app/mysql/bin:/bin:/usr/bin:/usr/local/bin:/usr/sbin)
170526 0:04:07 [ERROR] WSREP: Failed to read 'ready <addr>' from: wsrep_sst_rsync --role 'joiner' --address '192.168.56.103' --datadir '/data/mysqldata/' --defaults-file '/app/mariadb-galera-10.0.29-linux/my.cnf' --defaults-group-suffix '' --parent '2832' --binlog '/data/binlog/mysql-bin'
Read: ''rsync' not found in PATH'
170526 0:04:07 [ERROR] WSREP: Process completed with error: wsrep_sst_rsync --role 'joiner' --address '192.168.56.103' --datadir '/data/mysqldata/' --defaults-file '/app/mariadb-galera-10.0.29-linux/my.cnf' --defaults-group-suffix '' --parent '2832' --binlog '/data/binlog/mysql-bin' : 2 (No such file or directory)
170526 0:04:07 [ERROR] WSREP: Failed to prepare for 'rsync' SST. Unrecoverable.
170526 0:04:07 [ERROR] Aborting
yum install rsync.x86_64

 

which: no lsof in (/usr/sbin:/sbin:/app/mariadb-galera-10.0.29-linux//bin:/app/mysql/bin:/bin:/usr/bin:/usr/local/bin:/usr/sbin:/app/xtrabackup/percona-xtrabackup/bin)
170526 10:13:28 [ERROR] WSREP: Failed to read 'ready <addr>' from: wsrep_sst_rsync --role 'joiner' --address '192.168.56.103' --datadir '/data/mysqldata/' --defaults-file '/app/mariadb-galera-10.0.29-linux/my.cnf' --defaults-group-suffix '' --parent '2821' --binlog '/data/binlog/mysql-bin'
Read: ''lsof' not found in PATH'
170526 10:13:28 [ERROR] WSREP: Process completed with error: wsrep_sst_rsync --role 'joiner' --address '192.168.56.103' --datadir '/data/mysqldata/' --defaults-file '/app/mariadb-galera-10.0.29-linux/my.cnf' --defaults-group-suffix '' --parent '2821' --binlog '/data/binlog/mysql-bin' : 2 (No such file or directory)
170526 10:13:28 [ERROR] WSREP: Failed to prepare for 'rsync' SST. Unrecoverable.
170526 10:13:28 [ERROR] Aborting
yum install lsof