postgresql雙機熱備、高可用方案（采用pacemaker+corosync實現）

http://blog.csdn.net/qguanri/article/details/51151974

 

需求描述

我們有兩台centos7的數據庫主機A、B。要對A、B實現雙機熱備，A作為數據庫master主機，對外提供讀寫服務，B作為slave主機能實時同步A的數據。當A發生故障時，B從slave狀態切換到master狀態，以保證pg數據庫能正常對外提供服務。

經過調研，最終我們給出的解決方案是采用pacemaker和corosync組件來實現postgresql的高可用。

所有部署腳本可以下github中找到：https://github.com/qinguanri/pg-ha

環境說明

• 操作系統centos7：

[root@10 ~]# cat /etc/redhat-release CentOS Linux release 7.2.1511 (Core) [root@10 ~]# uname -a Linux 10.16.93.179 3.10.0-327.10.1.el7.x86_64 #1 SMP Tue Feb 16 17:03:50 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

postgresql版本：9.2.15

• 2台主機：

角色	IP
node1	10.16.93.179
node2	10.16.93.182

- 分配2個虛IP。我們的數據庫對外提供服務時，暴露的是虛IP：

vip-master: 10.16.93.180

vip-slave: 10.16.93.181

配置Linux集群環境

注意：請嚴格按照下文所給的順序進行配置

1.安裝Pacemaker和Corosync包

需要在所有節點執行：

$  yum install -y pacemaker pcs psmisc policycoreutils-python postgresql-server

2.禁用防火牆

在所有節點執行：

$ systemctl disable firewalld.service $ systemctl stop firewalld.service

3.啟用pcs

在所有節點執行：

$  systemctl start pcsd.service $ systemctl enable pcsd.service ln -s '/usr/lib/systemd/system/pcsd.service' '/etc/systemd/system/multi-user.target.wants/pcsd.service' $ echo hacluster | passwd hacluster --stdin Changing password for user hacluster. Changing password for user hacluster. passwd: all authentication tokens updated successfully.

4.集群認證

在任何一個節點上執行，這里選擇node1：

$  pcs cluster auth -u hacluster -p hacluster 10.16.93.179 10.16.93.182 10.16.93.179: Authorized 10.16.93.182: Authorized

5.同步配置

在node1上執行：

$  pcs cluster setup --last_man_standing=1 --name pgcluster 10.16.93.179 10.16.93.182 Shutting down pacemaker/corosync services... Redirecting to /bin/systemctl stop pacemaker.service Redirecting to /bin/systemctl stop corosync.service Killing any remaining services... Removing all cluster configuration files... 10.16.93.179: Succeeded 10.16.93.182: Succeeded

6.啟動集群

在node1上執行。（有時需要稍等十幾秒，才能看到命令的輸出結果）：

$ pcs cluster start --all 172.17.5.90: Starting Cluster... 172.17.5.91: Starting Cluster... 172.17.5.92: Starting Cluster...

7.檢驗

1）檢驗corosync

在node1上執行：

$ pcs status corosync
Membership information ----------------------  Nodeid Votes Name  1 1 10.16.93.179 (local)  2 1 10.16.93.182

2）檢驗pacemaker

root@10 data]# pcs status
Cluster name: pgcluster
WARNING: corosync and pacemaker node names do not match (IPs used in setup?) Last updated: Mon Apr 4 23:45:48 2016 Last change: Fri Apr 1 09:17:03 2016 by root via crm_attribute on tsung_test Stack: corosync Current DC: tsung_test (version 1.1.13-10.el7_2.2-44eb2dd) - partition with quorum 2 nodes and 5 resources configured Online: [ 10.16.93.179 tsung_test ] Full list of resources: Master/Slave Set: pgsql-cluster [pgsql] Masters: [ tsung_test ] Slaves: [ 10.16.93.179 ] Resource Group: master-group vip-master (ocf::heartbeat:IPaddr2): Started tsung_test Resource Group: slave-group vip-slave (ocf::heartbeat:IPaddr2): Started 10.16.93.179 PCSD Status: 10.16.93.179: Online tsung_test (10.16.93.182): Online Daemon Status: corosync: active/disabled pacemaker: active/disabled pcsd: active/enabled

注意：當vip-master和vip-slave都是Started狀態時，集群節點配置正確。否則，請檢查你的配置是否有誤

安裝和配置PostgreSQL

1.創建數據庫目錄

需要在所有節點上執行：

$  mkdir -p /data/postgresql/{data,xlog_archive} $ chown -R postgres:postgres /data/postgresql/ $ chmod 0700 /data/postgresql/data

2.初始化db

在node1上執行：

$  su - postgres
$ initdb -D /data/postgresql/data/
The files belonging to this database system will be owned by user "postgres". This user must also own the server process. The database cluster will be initialized with locale "en_US.UTF-8". The default database encoding has accordingly been set to "UTF8". The default text search configuration will be set to "english". fixing permissions on existing directory /data/postgresql/data ... ok creating subdirectories ... ok selecting default max_connections ... 100 selecting default shared_buffers ... 32MB creating configuration files ... ok creating template1 database in /data/postgresql/data/base/1 ... ok initializing pg_authid ... ok initializing dependencies ... ok creating system views ... ok loading system objects' descriptions ... ok creating collations ... ok creating conversions ... ok creating dictionaries ... ok setting privileges on built-in objects ... ok creating information schema ... ok loading PL/pgSQL server-side language ... ok vacuuming database template1 ... ok copying template1 to template0 ... ok copying template1 to postgres ... ok WARNING: enabling "trust" authentication for local connections You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb. Success. You can now start the database server using: postgres -D /data/postgresql/data or pg_ctl -D /data/postgresql/data -l logfile start

3.修改配置文件

在node1上執行：

$ vim /data/postgresql/data/postgresql.conf
listen_addresses = '*'
wal_level = hot_standby
synchronous_commit = on archive_mode = on archive_command = 'cp %p /data/postgresql/xlog_archive/%f' max_wal_senders=5 wal_keep_segments = 32 hot_standby = on restart_after_crash = off replication_timeout = 5000 wal_receiver_status_interval = 2 max_standby_streaming_delay = -1 max_standby_archive_delay = -1 synchronous_commit = on restart_after_crash = off hot_standby_feedback = on $ vim /data/postgresql/data/pg_hba.conf local all all trust host all all 127.0.0.1/32 trust host all all ::1/128 trust host all all 10.16.0.0/16 md5 host all all 10.18.0.0/16 md5 host replication all 10.16.0.0/16 md5

注意：pg_hba.conf文件用於配置postgresql的訪問控制，例如：host all all 10.16.0.0/16 md5,表示允許10.16.0.0/16網段的主機訪問所有用戶、所有數據庫，密碼使用md5密文的方式。你需要根據你的實際網段進行設置。當出現訪問權限錯誤時，請檢查你的pg_hba.conf文件

4.啟動

啟動數據庫，創建replicator用戶。將用戶postgres的密碼修改為postgres（pg初始安裝時，postgres用戶的登錄密碼默認為空）

在node1上執行：

$ pg_ctl -D /data/postgresql/data/ start server starting [ 2015-10-16 08:51:31.451 UTC 53158 5620ba93.cfa6 1 0]LOG: redirecting log output to logging collector process [ 2015-10-16 08:51:31.451 UTC 53158 5620ba93.cfa6 2 0]HINT: Future log output will appear in directory "pg_log". $ psql -U postgres psql (9.2.13) Type "help" for help. postgres=# create role replicator with login replication password '8d5e9531-3817-460d-a851-659d2e51ca99'; CREATE ROLE postgres=# alter user postgres with password 'postgres'; postgres=# \q

5.制作slave

在node2上執行：

$  su - postgres $ pg_basebackup -h 10.16.93.179 -U postgres -D /data/postgresql/data/ -X stream -P could not change directory to "/home/wenhang.pan" 20127/20127 kB (100%), 1/1 tablespace $ vim /data/postgresql/data/recovery.conf standby_mode = 'on' primary_conninfo = 'host=10.16.93.179 port=5432 user=replicator application_name=zhaopin-5-92 password=8d5e9531-3817-460d-a851-659d2e51ca99 keepalives_idle=60 keepalives_in terval=5 keepalives_count=5' restore_command = 'cp /data/postgresql/xlog_archive/%f %p' recovery_target_timeline = 'latest'

6.啟動slave

在node2上執行：

$ pg_ctl -D /data/postgresql/data/ start pg_ctl: another server might be running; trying to start server anyway server starting -bash-4.2$ LOG: database system was interrupted while in recovery at log time 2015-10-16 08:19:07 GMT HINT: If this has occurred more than once some data might be corrupted and you might need to choose an earlier recovery target. LOG: entering standby mode LOG: redo starts at 0/3000020 LOG: consistent recovery state reached at 0/30000E0 LOG: database system is ready to accept read only connections LOG: streaming replication successfully connected to primary

7.查看集群狀態

在node1上執行一下命令。這個時候，可看到node2是一個replication。同步狀態為sync

[root@10 ~]# psql -U postgres psql (9.2.15) Type "help" for help. postgres=# select * from pg_stat_replication ; pid | usesysid | usename | application_name | client_addr | client_hostname | client_port | backend_start | state | sent_location | write_location | flush_location | replay_location | sync_priority | sync_state -------+----------+------------+------------------+--------------+-----------------+-------------+----------------------- --------+-----------+---------------+----------------+----------------+-----------------+---------------+------------ 17349 | 16384 | replicator | 10.16.93.182 | 10.16.93.182 | | 22690 | 2016-04-14 14:07:25.18 1297+08 | streaming | 0/9014508 | 0/9014508 | 0/9014508 | 0/9014508 | 1 | sync (1 row) 

8.停止PostgreSQL服務

在node1、node2上執行：

$ pg_ctl -D /data/postgresql/data/ -mi stop waiting for server to shut down.... done server stopped

注意：在配置自動切換前，一定要先把node1、node2上的PostgreSQL服務停掉

配置自動切換

上面的配置完成了數據庫的初始化。下面進行自動切換相關的配置。當發生故障時，master/slave身份實現自動切換。

1.配置

在node1執行：

1）配置項較多，我們先將配置步驟寫到腳本里

[root@10 ~]# vim cluster_setup.sh

# 將cib配置保存到文件
pcs cluster cib pgsql_cfg                                                                   
# 在pacemaker級別忽略quorum
pcs -f pgsql_cfg property set no-quorum-policy="ignore" # 禁用STONITH pcs -f pgsql_cfg property set stonith-enabled="false" # 設置資源粘性，防止節點在故障恢復后發生遷移 pcs -f pgsql_cfg resource defaults resource-stickiness="INFINITY" # 設置多少次失敗后遷移 pcs -f pgsql_cfg resource defaults migration-threshold="3" # 設置master節點虛ip pcs -f pgsql_cfg resource create vip-master IPaddr2 ip="10.16.93.180" cidr_netmask="24" op start timeout="60s" interval="0s" on-fail="restart" op monitor timeout="60s" interval="10s" on-fail="restart" op stop timeout="60s" interval="0s" on-fail="block" # 設置slave節點虛ip pcs -f pgsql_cfg resource create vip-slave IPaddr2 ip="10.16.93.181" cidr_netmask="24" op start timeout="60s" interval="0s" on-fail="restart" op monitor timeout="60s" interval="10s" on-fail="restart" op stop timeout="60s" interval="0s" on-fail="block" # 設置pgsql集群資源 # pgctl、psql、pgdata和config等配置根據自己的環境修改,node list填寫節點的hostname，master_ip填寫虛master_ip pcs -f pgsql_cfg resource create pgsql pgsql pgctl="/usr/bin/pg_ctl" psql="/usr/bin/psql" pgdata="/data/postgresql/data/" config="/data/postgresql/data/postgresql.conf" rep_mode="sync" node_list="tsung_test 10.16.93.179" master_ip="10.16.93.180" repuser="replicator" primary_conninfo_opt="password=8d5e9531-3817-460d-a851-659d2e51ca99 keepalives_idle=60 keepalives_interval=5 keepalives_count=5" restore_command="cp /data/postgresql/xlog_archive/%f %p" restart_on_promote='true' op start timeout="60s" interval="0s" on-fail="restart" op monitor timeout="60s" interval="4s" on-fail="restart" op monitor timeout="60s" interval="3s" on-fail="restart" role="Master" op promote timeout="60s" interval="0s" on-fail="restart" op demote timeout="60s" interval="0s" on-fail="stop" op stop timeout="60s" interval="0s" on-fail="block" # 設置master/slave模式，clone-max=2，兩個節點 pcs -f pgsql_cfg resource master pgsql-cluster pgsql master-max=1 master-node-max=1 clone-max=2 clone-node-max=1 notify=true # 配置master ip組 pcs -f pgsql_cfg resource group add master-group vip-master # 配置slave ip組 pcs -f pgsql_cfg resource group add slave-group vip-slave # 配置master ip組綁定master節點 pcs -f pgsql_cfg constraint colocation add master-group with master pgsql-cluster INFINITY # 配置啟動master節點 pcs -f pgsql_cfg constraint order promote pgsql-cluster then start master-group symmetrical=false score=INFINITY # 配置停止master節點 pcs -f pgsql_cfg constraint order demote pgsql-cluster then stop master-group symmetrical=false score=0 # 配置slave ip組綁定slave節點 pcs -f pgsql_cfg constraint colocation add slave-group with slave pgsql-cluster INFINITY # 配置啟動slave節點 pcs -f pgsql_cfg constraint order promote pgsql-cluster then start slave-group symmetrical=false score=INFINITY # 配置停止slave節點 pcs -f pgsql_cfg constraint order demote pgsql-cluster then stop slave-group symmetrical=false score=0 # 把配置文件push到cib pcs cluster cib-push pgsql_cfg

2）執行操作文件

$ sh cluster_setup.sh

2.查看狀態

1）查看cluster狀態

在node1上執行：

[root@10 ~]# pcs status
Cluster name: pgcluster
WARNING: corosync and pacemaker node names do not match (IPs used in setup?) Last updated: Mon Apr 4 23:15:56 2016 Last change: Fri Apr 1 09:17:03 2016 by root via crm_attribute on tsung_test Stack: corosync Current DC: tsung_test (version 1.1.13-10.el7_2.2-44eb2dd) - partition with quorum 2 nodes and 5 resources configured Online: [ 10.16.93.179 tsung_test ] Full list of resources: Master/Slave Set: pgsql-cluster [pgsql] Masters: [ tsung_test ] Slaves: [ 10.16.93.179 ] Resource Group: master-group vip-master (ocf::heartbeat:IPaddr2): Started tsung_test Resource Group: slave-group vip-slave (ocf::heartbeat:IPaddr2): Started 10.16.93.179 PCSD Status: 10.16.93.179: Online tsung_test (10.16.93.182): Online Daemon Status: corosync: active/disabled pacemaker: active/disabled pcsd: active/enabled

2）查看PostgreSQL集群狀態

在node1上執行：

[root@tsung_test data]# psql -U postgres psql (9.2.15) Type "help" for help. postgres=# select * from pg_stat_replication ;  pid | usesysid | usename | application_name | client_addr | client_hostname | client_port | backend_start | state | sent_location | write_locatio n | flush_location | replay_location | sync_priority | sync_state -------+----------+------------+------------------+--------------+-----------------+-------------+-------------------------------+-----------+---------------+-------------- --+----------------+-----------------+---------------+------------  19415 | 16384 | replicator | 10.16.93.182 | 10.16.93.182 | | 18289 | 2016-04-01 09:17:29.037197-04 | streaming | 0/40F0ED8 | 0/40F0ED8  | 0/40F0ED8 | 0/40F0ED8 | 0 | async (1 row)

驗證自動切換

參考文獻

• http://clusterlabs.org/wiki/PgSQL_Replicated_Cluster
• http://m.oschina.net/blog/518928
• http://m.oschina.net/blog/519458?mType=Group