在MariaDB 的高可用集群部署中，主流的實現方案還是Galera Cluster 與HAProxy和Pacemaker 的集成方案。
其中， Galera Cluster 負責數據庫節點之間的數據同步和一致性， Pacemaker 負責MariaDB 資源服務的啟停和運行監控等功能，HAProxy 主要負責數據庫對外服務的負載均衡。
MariaDB Galera Cluster 部署的第一步就是軟件安裝，在RHEL 和CentOS版本的Linux 系統中，建議通過Redhat 的RDO 源進行MariaDB Galera Cluster 的部署。與部署單機版的MariaDB 數據庫不同，在Galera 集群部署中，不再是安裝MariaDB-sever 軟件包，而是安裝MariaDB-Galera-server 和Galera 軟件包。在RDO 源配置完成后，將會看到yum 倉庫中新增了上述兩個軟件包。
Galera是一個MySQL(也支持MariaDB，Percona)的同步多主集群軟件，目前只支持InnoDB引擎。
主要功能:
同步復制
真正的multi-master，即所有節點可以同時讀寫數據庫
自動的節點成員控制，失效節點自動被清除
新節點加入數據自動復制
真正的並行復制，行級
用戶可以直接連接集群，使用感受上與MySQL完全一致
優勢:
因為是多主，所以不存在Slave lag(延遲)
不存在丟失交易的情況
同時具有讀和寫的擴展能力
更小的客戶端延遲
節點間數據是同步的,而Master/Slave模式是異步的,不同slave上的binlog可能是不同的
技術:
Galera集群的復制功能基於Galera library實現,為了讓MySQL與Galera library通訊，特別針對MySQL開發了wsrep API。
MariaDB介紹
MariaDB是MySQL的一個分支,由MySQL的創始人Michael Widenius主導開發,采用GPL授權許可。
開發這個分支的原因之一是Oracle公司收購了MySQL后,有將MySQL閉源的潛在風險,因此社區采用分支的方式來避開這個風險。
MariaDB的目的是完全兼容MySQL,包括API和命令行,使之能輕松成為MySQL的代替品。
在存儲引擎方面,使用XtraDB來代替MySQL的InnoDB。
方案總覽
haproxy作為MariaDB Galera Cluster的前端
2台haproxy用keepalived避免單點故障
3台MariaDB和一個garbd仲裁節點組成集群，仲裁節點上無數據
Galera的SST采用Percona提供的XtraBackup(防止鎖表，非阻塞)
環境准備
操作系統：CentOS 7.4版本
集群數量：3個節點
主機信息：
192.168.99.2 node1 selinux=disabled firewalld關閉
192.168.99.4 node2 selinux=disabled firewalld關閉
192.168.99.5 node3 selinux=disabled firewalld關閉
setenforce 0
sed -i "s/SELINUX=enforcing/SELINUX=disabled/g" /etc/selinux/config
systemctl disable firewalld.service
systemctl stop firewalld.service
--------------------------------------------------------------------------------
直接添加mariadb的專用源
cat >/etc/yum.repos.d/mariadb.repo<<EOF
[mariadb]
name = MariaDB
baseurl=http://yum.mariadb.org/10.1.40/centos7-amd64
gpgkey=https://yum.mariadb.org/RPM-GPG-KEY-MariaDB
gpgcheck=0
enable=1
EOF
yum clean all
yum install MariaDB-server -y
安裝的過程中會把MariaDB-client和galera一起安裝好
================================================================================
Package Arch Version Repository Size
================================================================================
Installing:
MariaDB-server x86_64 10.1.40-1.el7.centos mariadb 24 M
Installing for dependencies:
MariaDB-client x86_64 10.1.40-1.el7.centos mariadb 10 M
galera x86_64 25.3.26-1.rhel7.el7.centos mariadb 8.1 M
Transaction Summary
================================================================================
官方的mariadb galera cluster各個版本下載地址：
https://downloads.mariadb.org/mariadb-galera/+releases/
官方的MariaDB Galera集群入門：
https://mariadb.com/kb/en/library/getting-started-with-mariadb-galera-cluster/
內網可以考慮下載rpm包進行安裝
可以看到， yum 安裝命令會自動進行依賴包解析並進行依賴包的自動安裝，安裝完成之后，每個Galera 節點系統中將會增加許多與Galera 相關的目錄文件。
# rpm -ql galera
/usr/bin/garb-systemd
/usr/bin/garbd
/usr/lib/systemd/system/garb.service
/usr/lib64/galera
/usr/lib64/galera/libgalera_smm.so
/usr/share/doc/galera
/usr/share/doc/galera/COPYING
/usr/share/doc/galera/LICENSE.asio
/usr/share/doc/galera/LICENSE.chromium
/usr/share/doc/galera/LICENSE.crc32c
/usr/share/doc/galera/README
/usr/share/doc/galera/README-MySQL
/usr/share/man/man8/garbd.8.gz
# rpm -qf /bin/galera_new_cluster
MariaDB-server-10.1.14-1.el7.centos.x86_64
在mariadb-1服務器上運行（啟動集群后這些密碼的設置會同步到集群中的其他機器中）
啟動數據庫
systemctl start mariadb
設置密碼
mysqladmin -u root password 12345678
添加集群認證用戶（galera集群認證用戶---不是必須配置項）
MariaDB [(none)]> GRANT ALL PRIVILEGES ON *.* TO sst@'%' IDENTIFIED BY 'sstpass123' WITH GRANT OPTION;
MariaDB [(none)]> FLUSH PRIVILEGES;
關閉所有數據庫
systemctl stop mariadb
給三台測試的服務器都寫上這樣的配置，每台的配置相應的做一些調整
# cat /etc/my.cnf.d/galera.cnf | grep -v '^#\|^$'
[galera]
binlog_format=ROW
default-storage-engine=innodb
innodb_autoinc_lock_mode=2
bind-address=192.168.99.2
wsrep_on=ON
wsrep_provider=/usr/lib64/galera/libgalera_smm.so
wsrep_cluster_name="galera_cluster"
wsrep_cluster_address="gcomm://192.168.99.2:4567,192.168.99.4:4567,192.168.99.5:4567"
wsrep_node_name=node1
wsrep_node_address=192.168.99.2
wsrep_sst_auth=sst:sstpass123
# galera集群認證用戶：密碼---不是必須配置項
wsrep_sst_method=rsync
wsrep_causal_reads=ON
wsrep_slave_threads=1
wsrep_certify_nonPK=1
wsrep_max_ws_rows=131072
wsrep_max_ws_size=1073741824
wsrep_debug=0
wsrep_convert_LOCK_to_trx=0
wsrep_retry_autocommit=1
wsrep_auto_increment_control=1
wsrep_drupal_282555_workaround=0
wsrep_causal_reads=0
wsrep_notify_cmd=
[mariadb]
log-error=/var/log/mariadb/mariadb.log
注：使用galera-4包安裝的需要修改配置文件wsrep_provider=/usr/lib64/galera-4/libgalera_smm.so
將此文件復制到mariadb-2、mariadb-3，注意要把 wsrep_node_name 和 wsrep_node_address 改成相應節點的 hostname 和 ip。示范如下
[galera]
binlog_format=ROW
default-storage-engine=innodb
innodb_autoinc_lock_mode=2
bind-address=192.168.99.4
wsrep_on=ON
wsrep_provider=/usr/lib64/galera/libgalera_smm.so
wsrep_cluster_name="galera_cluster"
wsrep_cluster_address="gcomm://192.168.99.2:4567,192.168.99.4:4567,192.168.99.5:4567"
wsrep_node_name=node2
wsrep_node_address=192.168.99.4
wsrep_sst_auth=sst:sstpass123
wsrep_sst_method=rsync
wsrep_causal_reads=ON
wsrep_slave_threads=1
wsrep_certify_nonPK=1
wsrep_max_ws_rows=131072
wsrep_max_ws_size=1073741824
wsrep_debug=0
wsrep_convert_LOCK_to_trx=0
wsrep_retry_autocommit=1
wsrep_auto_increment_control=1
wsrep_drupal_282555_workaround=0
wsrep_causal_reads=0
wsrep_notify_cmd=
[mariadb]
log-error=/var/log/mariadb/mariadb.log
[galera]
query_cache_size=0 # 關閉查詢緩存
binlog_format=ROW # binlog文件格式：行
default_storage_engine=innodb # Mariadb 存儲引擎
innodb_autoinc_lock_mode=2 #主鍵自增模式修改為交叉模式
wsrep_provider=/usr/lib64/galera/libgalera_smm.so #galera 庫文件
wsrep_cluster_address="gcomm://192.168.99.2:4567,192.168.99.4:4567,192.168.99.5:4567" # galera集群url
wsrep_cluster_name='galera_cluster' # galera集群名稱
wsrep_node_address='192.168.99.2' # 該節點的地址
wsrep_node_name='node1' # 該節點的主機名
wsrep_sst_method=xtrabackup-v2 # 拷貝模式
wsrep_sst_auth=sst:sstpass123 # galera集群認證用戶：密碼---不是必須配置項
如果啟動其他節點數據庫有問題，建議改成rsync模式試試，但是最終我們建議使用xtrabackup-v2
啟動 MariaDB Galera Cluster 服務：
# /bin/galera_new_cluster
# systemctl status mariadb
● mariadb.service - MariaDB 10.3 database server
Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disabled)
Active: active (running) since Fri 2019-08-16 16:39:33 CST; 5min ago
Docs: man:mysqld(8)
https://mariadb.com/kb/en/library/systemd/
Process: 19118 ExecStartPost=/usr/libexec/mysql-check-upgrade (code=exited, status=0/SUCCESS)
Process: 19042 ExecStartPre=/usr/libexec/mysql-prepare-db-dir %n (code=exited, status=0/SUCCESS)
Process: 19017 ExecStartPre=/usr/libexec/mysql-check-socket (code=exited, status=0/SUCCESS)
Main PID: 19081 (mysqld)
Status: "Taking your SQL requests now..."
CGroup: /system.slice/mariadb.service
└─19081 /usr/libexec/mysqld --basedir=/usr --wsrep-new-cluster
Aug 16 16:39:33 45-113-32-200 systemd[1]: Starting MariaDB 10.3 database server...
Aug 16 16:39:33 45-113-32-200 mysql-prepare-db-dir[19042]: Database MariaDB is probably initialized in /var/lib/mysql already, nothing is done.
Aug 16 16:39:33 45-113-32-200 mysql-prepare-db-dir[19042]: If this is not the case, make sure the /var/lib/mysql is empty before running mysql-prepare-db-dir.
Aug 16 16:39:33 45-113-32-200 mysqld[19081]: 2019-08-16 16:39:33 0 [Warning] WSREP: option --wsrep-causal-reads is deprecated
Aug 16 16:39:33 45-113-32-200 mysqld[19081]: 2019-08-16 16:39:33 0 [Warning] WSREP: --wsrep-causal-reads=ON takes precedence over --wsrep-sync-wait=0. WSREP_SYNC_WAIT_BEFORE_READ is on
Aug 16 16:39:33 45-113-32-200 mysqld[19081]: 2019-08-16 16:39:33 0 [Warning] WSREP: --wsrep-sync-wait=1 takes precedence over --wsrep-causal-reads=OFF. WSREP_SYNC_WAIT_BEFORE_READ is on
Aug 16 16:39:33 45-113-32-200 mysqld[19081]: 2019-08-16 16:39:33 0 [Note] /usr/libexec/mysqld (mysqld 10.3.10-MariaDB) starting as process 19081 ...
Aug 16 16:39:33 45-113-32-200 mysqld[19081]: 2019-08-16 16:39:33 0 [Warning] Could not increase number of max_open_files to more than 1024 (request: 4182)
Aug 16 16:39:33 45-113-32-200 mysqld[19081]: 2019-08-16 16:39:33 0 [Warning] Changed limits: max_open_files: 1024 max_connections: 151 (was 151) table_cache: 421 (was 2000)
Aug 16 16:39:33 45-113-32-200 systemd[1]: Started MariaDB 10.3 database server.
剩余兩節點啟動方式為：
systemctl start mariadb
systemctl enable mariadb
查看集群狀態：（集群服務使用了4567和3306端口））
[root@node1 ~]# netstat -tnlp | grep -e 4567 -e 3306
tcp 0 0 0.0.0.0:4567 0.0.0.0:* LISTEN 17908/mysqld
tcp 0 0 192.168.99.2:3306 0.0.0.0:* LISTEN 17908/mysqld
擴展：剩余的mariadb啟動報錯
報錯：Database MariaDB is not initialized, but the directory /var/lib/mysql is not empty, so initialization cannot be done.
解決辦法： rm -rf /var/lib/mysql
mkdir -pv /var/lib/mysql
chown mysql:mysql -R /var/lib/mysql
啟動數據庫服務后，登錄mysql數據庫庫，通過wsrep_cluster_size參數來判斷啟動是否成功。
SHOW STATUS LIKE 'wsrep_cluster_size';
+--------------------+-------+
| Variable_name | Value |
+--------------------+-------+
| wsrep_cluster_size | 3 |
+--------------------+-------+
1 row in set (0.001 sec)
//查看Galera集群運行狀態
show status like '%wsrep%';
+------------------------------+-------------------------------------------------------+
| Variable_name | Value |
+------------------------------+-------------------------------------------------------+
| wsrep_apply_oooe | 0.000000 |
| wsrep_apply_oool | 0.000000 |
| wsrep_apply_window | 0.000000 |
| wsrep_causal_reads | 1 |
| wsrep_cert_deps_distance | 0.000000 |
| wsrep_cert_index_size | 0 |
| wsrep_cert_interval | 0.000000 |
| wsrep_cluster_conf_id | 3 |
| wsrep_cluster_size | 3 |
| wsrep_cluster_state_uuid | 6c8283d0-bff8-11e9-ad9e-8a56edae0f44 |
| wsrep_cluster_status | Primary |
| wsrep_commit_oooe | 0.000000 |
| wsrep_commit_oool | 0.000000 |
| wsrep_commit_window | 0.000000 |
| wsrep_connected | ON |
| wsrep_desync_count | 0 |
| wsrep_evs_delayed | |
| wsrep_evs_evict_list | |
| wsrep_evs_repl_latency | 0/0/0/0/0 |
| wsrep_evs_state | OPERATIONAL |
| wsrep_flow_control_paused | 0.000000 |
| wsrep_flow_control_paused_ns | 0 |
| wsrep_flow_control_recv | 0 |
| wsrep_flow_control_sent | 0 |
| wsrep_gcomm_uuid | 5efbadba-c001-11e9-9b8e-ab70e12329ca |
| wsrep_incoming_addresses | 192.168.99.2:3306,192.168.99.5:3306,192.168.99.4:3306 |
| wsrep_last_committed | 0 |
| wsrep_local_bf_aborts | 0 |
| wsrep_local_cached_downto | 18446744073709551615 |
| wsrep_local_cert_failures | 0 |
| wsrep_local_commits | 0 |
| wsrep_local_index | 0 |
| wsrep_local_recv_queue | 0 |
| wsrep_local_recv_queue_avg | 0.000000 |
| wsrep_local_recv_queue_max | 1 |
| wsrep_local_recv_queue_min | 0 |
| wsrep_local_replays | 0 |
| wsrep_local_send_queue | 0 |
| wsrep_local_send_queue_avg | 0.000000 |
| wsrep_local_send_queue_max | 1 |
| wsrep_local_send_queue_min | 0 |
| wsrep_local_state | 4 |
| wsrep_local_state_comment | Synced |
| wsrep_local_state_uuid | 6c8283d0-bff8-11e9-ad9e-8a56edae0f44 |
| wsrep_protocol_version | 8 |
| wsrep_provider_name | Galera |
| wsrep_provider_vendor | Codership Oy <info@codership.com> |
| wsrep_provider_version | 3.23(rXXXX) |
| wsrep_ready | ON |
| wsrep_received | 10 |
| wsrep_received_bytes | 742 |
| wsrep_repl_data_bytes | 0 |
| wsrep_repl_keys | 0 |
| wsrep_repl_keys_bytes | 0 |
| wsrep_repl_other_bytes | 0 |
| wsrep_replicated | 0 |
| wsrep_replicated_bytes | 0 |
| wsrep_thread_count | 2 |
+------------------------------+-------------------------------------------------------+
58 rows in set (0.001 sec)
運維常用命令
查看集群狀態
show status like 'wsrep%';
注釋：
wsrep_cluster_status為Primary，表示節點為主節點，正常讀寫。
wsrep_ready為ON，表示集群正常運行。
wsrep_connected: 如果該值為Off,且wsrep_ready的值也為Off,則說明該節點沒有連接到集群
wsrep_cluster_size為3，表示集群有三個節點。
wsrep_cluster_state_uuid:在集群所有節點的值應該是相同的,有不同值的節點,說明其沒有連接入集群。
wsrep_cluster_conf_id:正常情況下所有節點上該值是一樣的.如果值不同,說明該節點被臨時”分區”了.當節點之間網絡連接恢復的時候應該會恢復一樣的值。
wsrep_flow_control_paused:表示復制停止了多長時間.即表明集群因為Slave延遲而慢的程度.值為0~1,越靠近0越好,值為1表示復制完全停止.可優化wsrep_slave_threads的值來改善.
wsrep_flow_control_sent:表示該節點已經停止復制了多少次.
查看復制模式
show variables like "wsrep_sst_method";
wsrep_local_state_comment 狀態對照表
文件理解
grastate.dat
默認路徑為：/var/lib/mysql/
grastate.dat
可以通過該文件查看到該節點記錄的uuid和seqno，當節點正常退出Galera集群時，會將GTID的值更新到該文件中。
如果該節點數據庫服務正在運行，則seqno的值是-1的
在斷電的情況下，所有節點的seqno值可能都相同，此時需根 gvwstate.dat判斷啟動節點
gvwstate.dat
默認路徑為：/var/lib/mysql/gvwstate.dat
此文件保存了集群狀態信息，如果集群正常關閉的話，這個文件會自動刪除掉。
查看節點狀態
wsrep_local_state：
節點的狀態,取值1-6。
取值1：The node starts and establishes a connection to the Primary Component.
取值2：When the node succeeds with a state transfer request, it begins to cache write-sets.
取值3：The node receives a State Snapshot Transfer. It now has all cluster data and begins to apply the cached write-sets.
Here the node enables Flow Control to ensure an eventual decrease in the slave queue.
取值4：The node finishes catching up with the cluster. Its slave queue is now empty and it enables Flow Control to keep it empty.
The node sets the MySQL status variable wsrep_ready to the value 1. The node is now allowed to process transactions.
取值5：The node receives a state transfer request. Flow Control relaxes to DONOR. The node caches all write-sets it cannot apply.
取值6：The node completes the state transfer to joiner node.
問題
1.異常斷電
當機房突然停電，所有galera主機都非正常關機，來電后開機，會導致galera集群服務無法正常啟動。如何處理？
第1步：開啟galera集群的群主主機的mariadb服務。
第2步：開啟galera集群的成員主機的mariadb服務。
異常處理：galera集群的群主主機和成員主機的mysql服務無法啟動，如何處理？
解決方法一：第1步、刪除garlera群主主機的/var/lib/mysql/grastate.dat狀態文件
/bin/galera_new_cluster啟動服務。啟動正常。登錄並查看wsrep狀態。
第2步：刪除galera成員主機中的/var/lib/mysql/grastate.dat狀態文件
systemctl restart mariadb重啟服務。啟動正常。登錄並查看wsrep狀態。
解決方法二：第1步、修改garlera群主主機的/var/lib/mysql/grastate.dat狀態文件中的0為1
/bin/galera_new_cluster啟動服務。啟動正常。登錄並查看wsrep狀態。
第2步：修改galera成員主機中的/var/lib/mysql/grastate.dat狀態文件中的0為1
systemctl restart mariadb重啟服務。啟動正常。登錄並查看wsrep狀態。
注：如果被刪除了/var/lib/mysql/grastate.dat
，新建/var/lib/mysql/grastate.dat
要變更權限，執行chown mysql:mysql /var/lib/mysql/grastate.dat
2.腦裂
set global wsrep_provider_options="pc.bootstrap=true";
通過這個命令來強制恢復出現腦裂的節點。
擴展：
問題--->當gralera群主主機在機房突然斷電后，悲催的是群主主機起不來，只有幾台成員主機沒有問題....
解決辦法--->在某個成員主機中刪除/var/lib/mysql/grastate.dat文件，並執行/bin/galera_new_cluster啟動服務,於是這個成員主機就變成了群主主機。
然后在刪除其他galera成員主機中的/var/lib/mysql/grastate.dat狀態文件，systemctl restart mariadb重啟服務，問題發生了，其他的galera成員主機的mariadb服務啟動不了....
問題在於發生了腦裂，登錄新的群主主機執行set global wsrep_provider_options="pc.bootstrap=true";，再次在其他的galera成員主機重啟mariadb服務，啟動正常。OK，問題解決。
3.報錯/var/run/mariadb/mariadb.pid缺少權限
查看報錯日志/var/log/mariadb/mariadb.log
[ERROR] mysqld: Can't create/write to file '/var/run/mariadb/mariadb.pid' (Errcode: 13 "Permission denied")
[ERROR] Can't start server: can't create PID file: Permission denied:
解決辦法：mkdir -pv /var/run/mariadb/
chown -R mysql:mysql /var/run/mariadb
4.報錯WSREP: failed to open gcomm backend connection: 131: invalid UUID
[ERROR] WSREP: failed to open gcomm backend connection: 131: invalid UUID: 00000000 (FATAL)
at gcomm/src/pc.cpp:PC():271
[ERROR] WSREP: gcs/src/gcs_core.cpp:gcs_core_open():208: Failed to open backend connection: -131 (State not recoverable)
[ERROR] WSREP: gcs/src/gcs.cpp:gcs_open():1379: Failed to open channel 'wsrep_cluster' at 'gcomm://': -131 (State not recoverable)
[ERROR] WSREP: gcs connect failed: State not recoverable
[ERROR] WSREP: wsrep::connect(gcomm://) failed: 7
[ERROR] Aborting
解決辦法：進入該數據庫節點/var/lib/mysql/目錄，將文件gvwstate.dat移除(mv)掉。然后重新啟動mairbd即可
5.斷電后數據庫集群中的服務啟動了一段時間，發現集群的數據庫數據並沒有同步復制，數據一直往一台機器寫入，怎么辦？（生產環境實例...后果是操作失誤導致開啟了集群后數據被幾天沒有新數據的數據庫覆蓋了，也就是最近幾天的數據沒有了）
解決辦法：在備份數據后，把所有數據庫服務全部關閉，把一直有數據寫入數據庫作為主節點率先啟動執行/bin/galera_new_cluster
其他節點后啟動執行rm -f /var/lib/mysql/grastate.dat && systemctl start mariadb(有可能會發現由於同步數據時間過長，導致的無法啟動，如果有相關mysql進程可以殺死后多次啟動，直到數據同步完成能啟動為止，當然你也可以調整相關mysql啟動超時的參數---查看日志關於狀態變好可能是有幫助的)
一般節點的啟動狀態變好
[Note] WSREP: Shifting CLOSED -> OPEN (TO: 0)
[Note] WSREP: Restored state OPEN -> JOINED (0)
[Note] WSREP: Shifting JOINED -> SYNCED (TO: 0)
關於啟動后查看哪個節點才是主節點
]$ sudo grep -i node /var/log/mariadb/mariadb.log
2019-12-10 11:03:50 139673594214592 [Note] WSREP: Node e56ffd23 state prim
2019-12-10 11:03:50 139673593899776 [Note] WSREP: New cluster view: global state: dcf89fd8-1a5b-11ea-9e67-46b64f83070d:0, view# 4: Primary, number of nodes: 2, my index: 0, protocol version 3
2019-12-10 11:07:02 139752511609024 [Note] WSREP: Node 232e34f2 state prim
2019-12-10 11:07:02 139752511294208 [Note] WSREP: New cluster view: global state: dcf89fd8-1a5b-11ea-9e67-46b64f83070d:0, view# 1: Primary, number of nodes: 1, my index: 0, protocol version 3
2019-12-10 11:09:04 139752200206080 [Note]
WSREP: Node 232e34f2 state prim
2019-12-10 11:09:05 139752511294208 [Note] WSREP: New cluster view: global state: dcf89fd8-1a5b-11ea-9e67-46b64f83070d:0, view# 2: Primary, number of nodes: 2, my index: 0, protocol version 3
MariaDB [(none)]> show status like 'wsrep_gcomm_uuid';
+------------------+--------------------------------------+
| Variable_name | Value |
+------------------+--------------------------------------+
| wsrep_gcomm_uuid |
232e34f2-1afa-11ea-9759-6e2b142ae042 |
+------------------+--------------------------------------+
1 row in set (0.00 sec)
-------------------------------------------------------------------------------------------------------------------------------------------
生產環境的配置示范
/etc/my.cnf的配置
[client]
port=3306
socket=/var/lib/mysql/mysql.sock
[mysqld]
character-set-server=utf8
default-storage-engine=InnoDB
port=3306
socket=/var/lib/mysql/mysql.sock
skip-external-locking
key_buffer_size = 256M
max_allowed_packet = 1M
table_open_cache = 256
sort_buffer_size = 1M
read_buffer_size = 1M
read_rnd_buffer_size = 4M
myisam_sort_buffer_size = 64M
thread_cache_size = 8
query_cache_size= 16M
thread_concurrency = 8
max_connections = 800
log-bin=mysql-bin
binlog_format=mixed
server-id = 1
skip-name-resolve
[mysqldump]
quick
max_allowed_packet = 16M
[mysql]
no-auto-rehash
[myisamchk]
key_buffer_size = 128M
sort_buffer_size = 128M
read_buffer = 2M
write_buffer = 2M
[mysqlhotcopy]
interactive-timeout
!includedir /etc/my.cnf.d
高可用集群配置文件/etc/my.cnf.d/galera.cnf
[galera]
wsrep_on=ON
wsrep_provider=/usr/lib64/galera/libgalera_smm.so
wsrep_cluster_address="gcomm://192.168.30.126：4567,192.168.30.127:4567"
wsrep_cluster_name="wsrep_cluster"
wsrep_node_name=host-127
wsrep_node_address=192.168.30.127
binlog_format=row
default_storage_engine=InnoDB
innodb_autoinc_lock_mode=2
bind-address=192.168.30.127
wsrep_slave_threads=1
innodb_flush_log_at_trx_commit=0
[embedded]
[mariadb]
log-error=/var/log/mariadb/mariadb.log #添加日志文件方便分析和排查故障
pid-file=/var/run/mariadb/mariadb.pid
[mariadb-10.1]
生產環境由於在配置中添加了日志文件和pid-file導致服務起不來的問題
[root@node1 ~]# systemctl status mariadb.service
● mariadb.service - MariaDB 10.1.40 database server
Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disabled)
Drop-In: /etc/systemd/system/mariadb.service.d
└─migrated-from-my.cnf-settings.conf
Active: failed (Result: exit-code) since Fri 2019-12-27 16:16:03 CST; 7h left
Docs: man:mysqld(8)
https://mariadb.com/kb/en/library/systemd/
Process: 14595 ExecStart=/usr/sbin/mysqld $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION (code=exited, status=1/FAILURE)
Process: 14386 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= || VAR=`/usr/bin/galera_recovery`; [ $? -eq 0 ] && systemctl set-environment _WSREP_START_POSITION=$VAR || exit 1 (code=exited, status=0/SUCCESS)
Process: 14384 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
Main PID: 14595 (code=exited, status=1/FAILURE)
Dec 27 16:16:03 node1 mysqld[14595]: 2019-12-27 16:16:03 140593965365504 [Note] Plugin 'FEEDBACK' is disabled.
Dec 27 16:16:03 node1 mysqld[14595]: 2019-12-27 16:16:03 140592789059328 [Note] InnoDB: Dumping buffer pool(s) not yet started
Dec 27 16:16:03 node1 mysqld[14595]: 2019-12-27 16:16:03 140593648891648 [Note] WSREP: (1f745993, 'tcp://0.0.0.0:4567') connection established to 74e6b5d2 tcp://10.30.1.202:4567
Dec 27 16:16:03 node1 mysqld[14595]: 2019-12-27 16:16:03 140593648891648 [Note] WSREP: (1f745993, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers:
Dec 27 16:16:03 node1 mysqld[14595]: 2019-12-27 16:16:03 140593965365504 [Note] Server socket created on IP: '10.30.1.201'.
Dec 27 16:16:03 node1 mysqld[14595]: 2019-12-27 16:16:03 140593964320512 [ERROR] mysqld: Can't create/write to file '/var/run/mariadb/mariadb.pid' (Errcode: 2 "No such file or directory")
Dec 27 16:16:03 node1 systemd[1]: mariadb.service: main process exited, code=exited, status=1/FAILURE
Dec 27 16:16:03 node1 systemd[1]: Failed to start MariaDB 10.1.40 database server.
Dec 27 16:16:03 node1 systemd[1]: Unit mariadb.service entered failed state.
Dec 27 16:16:03 node1 systemd[1]: mariadb.service failed.
解決辦法
mkdir -pv /var/run/mariadb/
chown -R mysql:mysql /var/run/mariadb/
mkdir -pv /var/log/mariadb/
chown -R mysql:mysql /var/log/mariadb/
前同事關於高可用數據庫操作
啟動第一個節點
# service mysql bootstrap
或
#service mysql start --wsrep-new-cluster wsrep_cluster_address="gcomm://"
第一個節點一旦關閉后重啟，啟動時必須指定集群中其他任意一個節點的IP
# service mysql start --wsrep_cluster_address=gcomm://192.168.1.102
啟動后續節點
# service mysql start
如果異常關閉后不能啟動，執行以下命令后在嘗試
mysqld_safe --wsrep-recover
查看集群狀態
mysql>SHOW STATUS LIKE 'wsrep%';
啟動異常，報錯
WSREP: failed to open gcomm backend connection: 131: invalid UUID
進入該數據庫節點/var/lib/mysql/目錄，將文件gvwstate.dat移除(mv)掉。然后重新啟動mairbd即可
擴展：關於內網的mariadb和galera的安裝辦法
其實galera並不需要單獨安裝，當我們配置好mariadb的源時，在yum安裝mariadb-server包時會自動幫我們安裝一批軟件，其中就包括galera
下面是我們生產環境的安裝辦法
# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
10.88.66.5
yum.mariadb.org
# cat /etc/yum.repos.d/mariadb.repo
[mariadb]
name=MariaDB
gpgcheck=0
enable=1
Loaded plugins: fastestmirror
MariaDB-10.1.14-centos7-x86_64-server.rpm | 99 MB 00:01
Examining /var/tmp/yum-root-yhH8bv/MariaDB-10.1.14-centos7-x86_64-server.rpm: MariaDB-server-10.1.14-1.el7.centos.x86_64
Marking /var/tmp/yum-root-yhH8bv/MariaDB-10.1.14-centos7-x86_64-server.rpm to be installed
Resolving Dependencies
--> Running transaction check
---> Package MariaDB-server.x86_64 0:10.1.14-1.el7.centos will be installed
--> Processing Dependency: MariaDB-client for package: MariaDB-server-10.1.14-1.el7.centos.x86_64
Loading mirror speeds from cached hostfile
* base:
mirrors.tuna.tsinghua.edu.cn
* epel:
mirrors.tuna.tsinghua.edu.cn
* extras:
mirrors.tuna.tsinghua.edu.cn
* updates:
mirrors.neusoft.edu.cn
--> Processing Dependency: galera for package: MariaDB-server-10.1.14-1.el7.centos.x86_64
--> Processing Dependency: libjemalloc.so.1()(64bit) for package: MariaDB-server-10.1.14-1.el7.centos.x86_64
--> Running transaction check
---> Package MariaDB-client.x86_64 0:10.1.14-1.el7.centos will be installed
---> Package galera.x86_64 0:25.3.15-1.rhel7.el7.centos will be installed
---> Package jemalloc.x86_64 0:3.6.0-1.el7 will be installed
--> Finished Dependency Resolution
Dependencies Resolved
================================================================================
Package Arch Version Repository Size
================================================================================
Installing:
MariaDB-server x86_64 10.1.14-1.el7.centos /MariaDB-10.1.14-centos7-x86_64-server
423 M
Installing for dependencies:
MariaDB-client x86_64 10.1.14-1.el7.centos mariadb 39 M
galera x86_64 25.3.15-1.rhel7.el7.centos mariadb 7.7 M
jemalloc x86_64 3.6.0-1.el7 epel 105 k
Transaction Summary
================================================================================
Install 1 Package (+3 Dependent packages)
Total size: 469 M
Installed size: 624 M
Is this ok [y/d/N]: y
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Installing : MariaDB-client-10.1.14-1.el7.centos.x86_64 1/4
Installing : jemalloc-3.6.0-1.el7.x86_64 2/4
Installing : galera-25.3.15-1.rhel7.el7.centos.x86_64 3/4
Installing : MariaDB-server-10.1.14-1.el7.centos.x86_64 4/4
Verifying : MariaDB-server-10.1.14-1.el7.centos.x86_64 1/4
Verifying : galera-25.3.15-1.rhel7.el7.centos.x86_64 2/4
Verifying : jemalloc-3.6.0-1.el7.x86_64 3/4
Verifying : MariaDB-client-10.1.14-1.el7.centos.x86_64 4/4
Installed:
MariaDB-server.x86_64 0:10.1.14-1.el7.centos
Dependency Installed:
MariaDB-client.x86_64 0:10.1.14-1.el7.centos
galera.x86_64 0:25.3.15-1.rhel7.el7.centos
jemalloc.x86_64 0:3.6.0-1.el7
Complete!
作者：Dexter_Wang 工作崗位：某互聯網公司資深Linux架構師 聯系郵箱：993852246@qq.com