1概述
配置了四個Linux HA+ Oracle數據庫集群。其配置信息如下：
1.1店會系統&FC-arap系統集群
| 集群資源組名稱 |
ac_group |
fc_group |
| 集群資源文件掛載點 |
/ac-oradata |
/fc-oradata |
| 集群服務IP地址 |
fmpracora-vip/ 10.0.101.11 |
fmprfcora-vip/ 10.0.101.12 |
| 集群Listener |
cvs_lsn |
arap2_lsn |
| 集群數據庫 |
cvsdb |
arap2db |
| 優先運行的節點名稱 |
fmpracora-priv |
fmprfcora-priv |
| 優先運行的節點IP |
10.0.101.9 |
10.0.101.10 |
| 集群fence通訊IP |
10.0.202.15 |
10.0.202.19 |
1.2訂購-營收&主檔-預算檢討-基盤集群
| 集群資源組名稱 |
ma_group |
po_group |
| 集群資源文件掛載點 |
/ma-oradata |
/po-oradata |
| 集群服務IP地址 |
fmprmaora-vip/ 10.0.101.7 |
fmprpoora-vip/ 10.0.101.8 |
| 集群Listener |
arpdb_lsn |
yingshou_lsn |
| 集群數據庫 |
master_db, arpdb_db |
dingoudb, yingshou_db |
| 優先運行的節點名稱 |
fmprmaora-priv |
fmprpoora-priv |
| 優先運行的節點IP |
10.0.101.5 |
10.0.101.6 |
| 集群fence通訊IP |
10.0.202.16 |
10.0.202.20 |
1.3IDC-SCM系統集群
| 集群資源組名稱 |
idc_group |
| 集群資源文件掛載點 |
/idc-oradata |
| 集群服務IP地址 |
fmpridcora-vip/ 10.0.101.4 |
| 集群Listener |
arpdb_lsn |
| 集群數據庫 |
fmidcdb |
| 優先運行的節點名稱 |
fmpridcora-priv |
| 優先運行的節點IP |
10.0.101.4 |
| 集群fence通訊IP |
10.0.202.17 |
集群另外一個節點是10.0.101.3。
1.4JDE系統&JDE測試系統集群
| 集群資源組名稱 |
pr_group |
dr_group |
| 集群資源文件掛載點 |
/jdepr-oradata |
/jdedr-oradata |
| 集群服務IP地址 |
fmprjdeora-vip/ 10.0.101. 15 |
fmdrjdeora-vip/ 10.0.101. 16 |
| 集群Listener |
jde_lsn |
jdedbdr_lsn |
| 集群數據庫 |
jde_db |
jdedr_db |
| 優先運行的節點名稱 |
fmprjdeora-priv |
fmdrjdeora-priv |
| 優先運行的節點IP |
10.0.101.13 |
10.0.101.14 |
| 集群fence通訊IP |
10.0.202.18 |
10.0.202.22 |
2安裝配置手冊
RHEL 6.8 HA和RHEL 7.2 HA的安裝過程不相同，但是集群的配置過程相同。我們將軟件安裝過程分開，配置過程放在一起描述。
2.1 . RHEL 6.8 HA軟件安裝
將RHEL 6.8 ISO文件掛載到/software/rh68目錄。編輯/etc/yum.conf文件。
2.1.1 編輯/etc/yum.conf
[local]
name=local
baseurl=file:///software/rh68
enabled=1
gpgcheck=1
gpgkey=file:///software/rh68/RPM-GPG-KEY-redhat-release
[HighAvailability]
name=HighAvailability
baseurl=file:///software/rh68/HighAvailability
enabled=1
gpgcheck=0
2.1.2 安裝軟件
# yum install pcs pacemaker cman fence-agents
從RHEL 6.5以后，RHEL的HA通過pcs來管理，不再通過rgmanager管理。只有RHEL 6.8才有Oracle的Agent。
2.1.3 禁用和停止主機防火牆服務
禁用防火牆
#chkconfig iptables off
#chkconfig ip6tables off
停止防火牆服務
#service iptables stop
#service ip6tables off
For TCP: Ports 2224, 3121, 21064
For UDP: Ports, 5405
2.1.4 啟動和配置服務pcsd
# chkconfig pcsd on
# service pcsd start
2.1.5 重置hacluster密碼
hacluster是pcs 管理員賬號，建議集群中的服務器設置相同的密碼。
# passwd hacluster (hacluster1234)
更改用戶 hacluster 的密碼 。
新的 密碼：
無效的密碼： 密碼包含用戶名在某些地方
重新輸入新的 密碼：
passwd：所有的身份驗證令牌已經成功更新。
2.2RHEL 7.2 HA軟件安裝
2.2.1 /etc/yum.conf
[HighAvailability]
name=HighAvailability
baseurl=file:///software/rh72/addons/HighAvailability
enabled=1
gpgcheck=0
[local]
name=local
baseurl=file:///software/rh72/
enabled=1
gpgcheck=1
gpgkey=file:///software/rh72/RPM-GPG-KEY-redhat-release
2.2.2 安裝軟件
# yum install pcs fence-agents-all
只有RHEL 7.2才有Oracle的HA Agent, RHEL 7.0和7.1都沒有Oracle的HA Agent。在RHEL 7.2,將不再使用cman服務.
2.2.3 啟動和配置服務pcsd
[root@fmprfcdb ~]# systemctl start pcsd.service
[root@fmprfcdb ~]# systemctl enable pcsd.service
ln -s '/usr/lib/systemd/system/pcsd.service' '/etc/systemd/system/multi-user.target.wants/pcsd.service'
檢查pcsd服務能隨機自動重新啟動
[root@fmpracora dinxin]# systemctl list-unit-files |grep pcs
pcsd.service enabled
rpcsvcgssd.service static
2.2.4 禁用和停止主機防火牆服務
#systemctl stop firewalld
2.2.5 重置hacluster密碼
hacluster是pcs 管理員賬號，建議集群中的服務器設置相同的密碼。
# passwd hacluster (hacluster1234)
更改用戶 hacluster 的密碼 。
新的 密碼：
無效的密碼： 密碼包含用戶名在某些地方
重新輸入新的 密碼：
passwd：所有的身份驗證令牌已經成功更新。
2.3配置集群
RHEL 6.8和RHEL 7.2配置集群的步驟相同。將在以下介紹。
2.3.1 編輯/etc/hosts
[root@fmprpoora oracle]# cat /etc/hosts
127.0.0.1localhost localhost.localdomain localhost4 localhost4.localdomain4
::1localhost localhost.localdomain localhost6 localhost6.localdomain6
10.0.101.6 fmprmaora
10.0.101.5 fmprpoora
10.0.200.16 fmprpoora-priv
10.0.200.20 fmprmaora-priv
10.0.101.7fmprpoora-vip
10.0.101.8fmprmaora-vip
其中fmprpoora-priv，fmprmaora-priv為心跳IP地址對應的主機名，將用作集群間的通訊。
2.3.2 授權hacluster 訪問集群中的節點
# pcs cluster auth fmprfcdb-priv fmpracdb-priv
Username: hacluster
Password:
fmprfcdb-priv: Authorized
fmpracdb-priv: Authorized
授權的令牌保存在以下目錄中。
[root@fmprpoora oracle]# cat /var/lib/pcsd/tokens
{
"format_version": 2,
"data_version": 3,
"tokens": {
"fmprmaora-priv": "a2a19af3-584e-4f88-873b-c3bee27d137c",
"fmprpoora-priv": "33169580-5ba5-4cb3-9a1c-a4a87fd3d226"
}
2.3.3 創建cluster
[root@fmprfcora pcsd]# pcs cluster setup --start --name fmprfcac_cluster fmprfcora-priv fmpracora-priv
Shutting down pacemaker/corosync services...
Redirecting to /bin/systemctl stop pacemaker.service
Redirecting to /bin/systemctl stop corosync.service
Killing any remaining services...
Removing all cluster configuration files...
fmprfcora-priv: Succeeded
fmpracora-priv: Succeeded
Starting cluster on nodes: fmprfcora-priv, fmpracora-priv...
fmpracora-priv: Starting Cluster...
fmprfcora-priv: Starting Cluster...
Synchronizing pcsd certificates on nodes fmprfcora-priv, fmpracora-priv...
fmprfcora-priv: Success
fmpracora-priv: Success
Restaring pcsd on the nodes in order to reload the certificates...
fmprfcora-priv: Success
fmpracora-priv: Success
[root@fmprfcora pcsd]#
2.3.4 配置集群服務隨主機啟動而自動啟動
[root@fmprfcora pcsd]# pcs cluster enable --all
fmprfcora-priv: Cluster Enabled
fmpracora-priv: Cluster Enabled
2.3.5 檢查cluster服務狀態
[root@fmpracdb pcsd]# pcs cluster status
Cluster Status:
Last updated: Sun Aug 7 11:07:59 2016 Last change: Sun Aug 7 11:06:44 2016 by hacluster via crmd on fmpracora-priv
Stack: corosync
Current DC: fmpracora-priv (version 1.1.13-10.el7-44eb2dd) - partition with quorum
2 nodes and 0 resources configured
Online: [ fmpracora-priv fmprfcora-priv ]
PCSD Status:
fmprfcora-priv: Online
fmpracora-priv: Online
2.3.6 配置集群的fence設備
#pcs stonith create imm-fmprjdeora fence_ipmilan action=reboot auth=password ipaddr=10.0.202.18 lanplus=1 login=USERID passwd=PASSW0RD delay=120cipher=1 pcmk_host_list=fmprjdeora-priv pcmk_host_check=static-list power_timeout=60
#pcs stonith create imm-fmdrjdeora fence_ipmilan action=reboot auth=password ipaddr=10.0.202.22 lanplus=1 login=USERID passwd=PASSW0RD cipher=1 pcmk_host_list=fmdrjdeora-priv pcmk_host_check=static-list power_timeout=60
查看當前fence設備配置的命令如下：
[root@fmprpoora oracle]# pcs stonith show --full
Resource: imm-fmprmaora (class=stonith type=fence_ipmilan)
Attributes: action=reboot auth=password ipaddr=10.0.202.20 lanplus=1 login=USERID passwd=PASSW0RD cipher=1 pcmk_host_list=fmprmaora-priv pcmk_host_check=static-list power_timeout=60
Operations: monitor interval=60s (imm-fmprmaora-monitor-interval-60s)
Resource: imm-fmprpoora (class=stonith type=fence_ipmilan)
Attributes: action=reboot auth=password ipaddr=10.0.202.16 lanplus=1 login=USERID passwd=PASSW0RD delay=5 cipher=1 pcmk_host_list=fmprpoora-priv pcmk_host_check=static-list power_timeout=60 debug=/tmp/fmprpoora.log
Operations: monitor interval=60s (imm-fmprpoora-monitor-interval-60s)
[root@fmprpoora oracle]#
2.3.7 測試fence
檢查能否使用fence_ipmilan命令查看fence設備的狀態
[root@fmpridcora oracle]# fence_ipmilan -a 10.0.202.22 -P -l USERID -p PASSW0RD -o status -v
Executing: /usr/bin/ipmitool -I lanplus -H 10.0.202.22 -U USERID -P PASSW0RD -p 623 -L ADMINISTRATOR chassis power status
0 Chassis Power is on
Status: ON
測試能否使用命令行將節點fence，fence之后的設備將自動重新啟動
root@fmprpoora ~]# pcs stonith fence fmprmaora-priv
Node: fmprmaora-priv fenced
[root@fmprmaora ~]# pcs stonith fence fmprpoora-priv
Node: fmprpoora-priv fenced
2.3.8 創建浮動IP
創建浮動IP地址資源
[root@fmprfcora pcsd]# pcs resource create fmprfcora-vip ocf:heartbeat:IPaddr2 ip=10.0.101.12 cidr_netmask=24 op monitor interval=30s
[root@fmprfcora pcsd]# pcs resource create fmpracora-vip ocf:heartbeat:IPaddr2 ip=10.0.101.11 cidr_netmask=24 op monitor interval=30s
[root@fmprfcora pcsd]# pcs resource show --full
Resource: fmprfcora-vip (class=ocf provider=heartbeat type=IPaddr2)
Attributes: ip=10.0.101.12 cidr_netmask=24
Operations: start interval=0s timeout=20s (fmprfcora-vip-start-interval-0s)
stop interval=0s timeout=20s (fmprfcora-vip-stop-interval-0s)
monitor interval=30s (fmprfcora-vip-monitor-interval-30s)
Resource: fmpracora-vip (class=ocf provider=heartbeat type=IPaddr2)
Attributes: ip=10.0.101.11 cidr_netmask=24
Operations: start interval=0s timeout=20s (fmpracora-vip-start-interval-0s)
stop interval=0s timeout=20s (fmpracora-vip-stop-interval-0s)
monitor interval=30s (fmpracora-vip-monitor-interval-30s)
2.3.9 創建文件系統資源
文件系統和掛載點需要已經創建好，並在集群中的各節點依次掛載完成。接下來在集群的文件系統資源。
[root@fmpracdb pcsd]# pcs resource create ac_fs Filesystem \
>device="/dev/mapper/acvg-aclv" directory="/ac-oradata" fstype="ext4"
[root@fmpracdb pcsd]# pcs resource create fc_fs Filesystem \
>device="/dev/mapper/fcvg-fclv" directory="/fc-oradata" fstype="ext4"
顯示文件系統資源的配置
[root@fmpracora dinxin]# pcs resource show ac_fs
Resource: ac_fs (class=ocf provider=heartbeat type=Filesystem)
Attributes: device=/dev/mapper/acvg-aclv directory=/ac-oradata fstype=ext4
Operations: start interval=0s timeout=60 (ac_fs-start-interval-0s)
stop interval=0s timeout=60 (ac_fs-stop-interval-0s)
monitor interval=20 timeout=40 (ac_fs-monitor-interval-20)
[root@fmpracora dinxin]# pcs resource show fc_fs
Resource: fc_fs (class=ocf provider=heartbeat type=Filesystem)
Attributes: device=/dev/mapper/fcvg-fclv directory=/fc-oradata fstype=ext4
Operations: start interval=0s timeout=60 (fc_fs-start-interval-0s)
stop interval=0s timeout=60 (fc_fs-stop-interval-0s)
monitor interval=20 timeout=40 (fc_fs-monitor-interval-20)
2.3.10 創建數據庫lsnr和db資源
確認當前pcs版本支持Oracle數據庫。以下表示支持oracle數據庫
[root@fmpracora dinxin]# pcs resource list |grep ora
ocf:heartbeat:oracle - Manages an Oracle Database instance
ocf:heartbeat:oralsnr - Manages an Oracle TNS listener
確認oracle數據庫和listener已經創建好，並且手工能夠在集群中的節點正常啟動到open狀態。
確保能解析數據庫sid,
[oracle@fmpracora admin]$ tnsping cvs
TNS Ping Utility for Linux: Version 11.2.0.4.0 - Production on 23-AUG-2016 13:51:06
Copyright (c) 1997, 2013, Oracle. All rights reserved.
Used parameter files:
Used TNSNAMES adapter to resolve the alias
Attempting to contact (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = fmpracora-vip)(PORT = 1526)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = cvs)))
OK (0 msec)
[oracle@fmpracora admin]$ tnsping arap2
TNS Ping Utility for Linux: Version 11.2.0.4.0 - Production on 23-AUG-2016 13:51:10
Copyright (c) 1997, 2013, Oracle. All rights reserved.
Used parameter files:
Used TNSNAMES adapter to resolve the alias
Attempting to contact (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = fmprfcora-vip)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = arap2)))
OK (0 msec)
當前的oracle listener.ora和tnsnames.ora文件如下：
CVS =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = fmpracora-vip)(PORT = 1526))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = cvs)
)
)
ARAP2 =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = fmprfcora-vip)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = arap2)
)
)
[oracle@fmpracora admin]$ cat listener.ora
LSNR_arap2=
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = TCP)(HOST = fmprfcora-vip)(PORT = 1521))
)
SECURE_REGISTER_LSNR_arap2 = (TCP)
LSNR_cvs=
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = TCP)(HOST = fmpracora-vip)(PORT = 1526))
)
SECURE_REGISTER_LSNR_cvs = (TCP)
使用以下命令創建oracle數據庫資源和listnener資源
[root@fmpracdb pcsd]# pcs resource create arap2db oracle sid=arap2 home=/home/app/11.2/db_1 user=oracle shutdown_method=immediate
[root@fmpracdb pcsd]# pcs resource create arap2_lsn oralsnr sid=arap2 home=/home/app/11.2/db_1 user=oracle listener=LSNR_arap2
[root@fmpracdb pcsd]# pcs resource create cvsdb oracle sid=cvs home=/home/app/11.2/db_1 user=oracle shutdown_method=immediate
[root@fmpracdb pcsd]# pcs resource create cvs_lsn oralsnr sid=cvs home=/home/app/11.2/db_1 user=oracle listener=LSNR_cvs2
使用以下命令查看當前數據庫資源和listener資源配置
[root@fmpracora dinxin]# pcs resource show cvs_lsn
Resource: cvs_lsn (class=ocf provider=heartbeat type=oralsnr)
Attributes: sid=cvs home=/home/app/11.2/db_1 user=oracle listener=LSNR_cvs
Operations: start interval=0s timeout=120 (cvs_lsn-start-interval-0s)
stop interval=0s timeout=120 (cvs_lsn-stop-interval-0s)
monitor interval=10 timeout=30 (cvs_lsn-monitor-interval-10)
[root@fmpracora dinxin]# pcs resource show cvsdb
Resource: cvsdb (class=ocf provider=heartbeat type=oracle)
Attributes: sid=cvs home=/home/app/11.2/db_1 user=oracle shutdown_method=immediate
Operations: start interval=0s timeout=120 (cvsdb-start-interval-0s)
stop interval=0s timeout=120 (cvsdb-stop-interval-0s)
monitor interval=120 timeout=30 (cvsdb-monitor-interval-120)
[root@fmpracora dinxin]# pcs resource show arap2db
Resource: arap2db (class=ocf provider=heartbeat type=oracle)
Attributes: sid=arap2 home=/home/app/11.2/db_1 user=oracle shutdown_method=immediate
Operations: start interval=0s timeout=120 (arap2db-start-interval-0s)
stop interval=0s timeout=120 (arap2db-stop-interval-0s)
monitor interval=120 timeout=30 (arap2db-monitor-interval-120)
[root@fmpracora dinxin]# pcs resource show arap2_lsn
Resource: arap2_lsn (class=ocf provider=heartbeat type=oralsnr)
Attributes: sid=arap2 home=/home/app/11.2/db_1 user=oracle listener=LSNR_arap2
Operations: start interval=0s timeout=120 (arap2_lsn-start-interval-0s)
stop interval=0s timeout=120 (arap2_lsn-stop-interval-0s)
monitor interval=10 timeout=30 (arap2_lsn-monitor-interval-10)
[root@fmpracora dinxin]#
2.3.11 創建集群組
創建集群組，在組中添加浮動IP地址資源，文件系統資源，oracle listener資源，以及Oracle數據庫資源，並設置資源間的依賴關系。組設置成功后，將作為一個整體進行節點間的切換。
# pcs resource group add ac_group fmpracora-vip
#pcs resource group add ac_group ac_fs --after fmpracora-vip
#pcs resource group add ac_group cvs_lsn --after ac_fs
#pcs resource group add ac_group cvsdb --after cvs_lsn
2.3.12 創建集群資源優先啟動節點
集群節點將在優先節點啟動。當優先節點因故障關機后，自動切換到其它節點。在優先節點開機重新啟動后，將跟隨cman,pcs服務啟動而自動回切到優先節點。
#pcs constraint location fc_fs prefers fmprfcora-priv
#pcs constraint location fmprfcora-vip prefers fmprfcora-priv
#pcs constraint location cvs_lsn prefers fmprfcora-priv
#pcs constraint location cvsdb prefers fmprfcora-priv
#pcs constraint location fc_group prefers fmprfcora-priv
#pcs constraint location ac_fs prefers fmpracora-priv
#pcs constraint location fmpracora-vip prefers fmpracora-priv
#pcs constraint location apar2_lsn prefers fmpracora-priv
#pcs constraint location apar2db prefers fmpracora-priv
#pcs constraint location ac_group prefers fmpracora-priv
查看優先節點：
[root@fmpracora dinxin]# pcs constraint
Location Constraints:
Resource: ac_group
Enabled on: fmpracora-priv (score:INFINITY)
Resource: fc_group
Enabled on: fmprfcora-priv (score:INFINITY)
Resource: fmpracora-vip
Enabled on: fmpracora-priv (score:INFINITY)
Resource: fmprfcora-vip
Enabled on: fmprfcora-priv (score:INFINITY)
3RHEL 6.8 & 7.2 HA維護手冊
3.1移動資源組到其它節點
[root@fmdrjdeora ~]# pcs resource move dr_group
Warning: Creating location constraint cli-ban-dr_group-on-fmdrjdeora-priv with a score of -INFINITY for resource dr_group on node fmdrjdeora-priv.
This will prevent dr_group from running on fmdrjdeora-priv until the constraint is removed. This will be the case even if fmdrjdeora-priv is the last node in the cluster.
服務遷移到其它節點。
[root@fmdrjdeora ~]# pcs resource
Resource Group: dr_group
fmdrjdeora-vip(ocf::heartbeat:IPaddr2):Started fmprjdeora-priv
dr_fs(ocf::heartbeat:Filesystem):Started fmprjdeora-priv
jdedbdr_lsn(ocf::heartbeat:oralsnr):Started fmprjdeora-priv
jdedr_db(ocf::heartbeat:oracle):Started fmprjdeora-priv
Resource Group: pr_group
pr_fs(ocf::heartbeat:Filesystem):Started fmprjdeora-priv
fmprjdeora-vip(ocf::heartbeat:IPaddr2):Started fmprjdeora-priv
jdedb(ocf::heartbeat:oracle):Started fmprjdeora-priv
jde_lsn(ocf::heartbeat:oralsnr):Started fmprjdeora-priv
且生成一個在節點fmdrjdeora-priv禁用
[root@fmdrjdeora oracle]# pcs constraint
Location Constraints:
Resource: dr_fs
Enabled on: fmdrjdeora-priv (score:INFINITY)
Resource: dr_group
Enabled on: fmdrjdeora-priv (score:300)
Enabled on: fmprjdeora-priv (score:100)
Disabled on: fmdrjdeora-priv (score:-INFINITY) (role: Started)
3.2回切該資源回到原優先節點
需要清除限制，否則該資源不會fmdrjdeora-priv運行。
[root@fmprjdeora ~]# pcs resource clear dr_group
清除資源后，constraint恢復正常。
[root@fmprjdeora ~]# pcs constraint
Location Constraints:
Resource: dr_fs
Enabled on: fmdrjdeora-priv (score:INFINITY)
Resource: dr_group
Enabled on: fmdrjdeora-priv (score:300)
Enabled on: fmprjdeora-priv (score:100)
Resource: fmdrjdeora-vip
Enabled on: fmdrjdeora-priv (score:300)
Resource: fmprjdeora-vip
Enabled on: fmprjdeora-priv (score:INFINITY)
Resource: jde_lsn
Enabled on: fmprjdeora-priv (score:INFINITY)
Resource: jdedb
Enabled on: fmprjdeora-priv (score:INFINITY)
Resource: pr_fs
Enabled on: fmprjdeora-priv (score:INFINITY)
Resource: pr_group
Enabled on: fmprjdeora-priv (score:INFINITY)
Ordering Constraints:
Resource Sets:
set fmprjdeora-vip pr_fs jde_lsn jdedb
set fmdrjdeora-vip dr_fs
Colocation Constraints:
資源就會自動回切到原始服務器。
[root@fmdrjdeora ~]# pcs resource clear dr_group
[root@fmdrjdeora ~]# pcs resource
Resource Group: dr_group
fmdrjdeora-vip(ocf::heartbeat:IPaddr2):Started fmdrjdeora-priv
dr_fs(ocf::heartbeat:Filesystem):Stopped
jdedbdr_lsn(ocf::heartbeat:oralsnr):Stopped
jdedr_db(ocf::heartbeat:oracle): Stopped
Resource Group: pr_group
pr_fs(ocf::heartbeat:Filesystem):Started fmprjdeora-priv
fmprjdeora-vip(ocf::heartbeat:IPaddr2):Started fmprjdeora-priv
jdedb(ocf::heartbeat:oracle):Started fmprjdeora-priv
jde_lsn(ocf::heartbeat:oralsnr):Started fmprjdeora-priv
[root@fmdrjdeora ~]# pcs resource
Resource Group: dr_group
fmdrjdeora-vip(ocf::heartbeat:IPaddr2):Started fmdrjdeora-priv
dr_fs(ocf::heartbeat:Filesystem):Started fmdrjdeora-priv
jdedbdr_lsn(ocf::heartbeat:oralsnr):Started fmdrjdeora-priv
jdedr_db(ocf::heartbeat:oracle):Started fmdrjdeora-priv
Resource Group: pr_group
pr_fs(ocf::heartbeat:Filesystem): Started fmprjdeora-priv
fmprjdeora-vip(ocf::heartbeat:IPaddr2):Started fmprjdeora-priv
jdedb(ocf::heartbeat:oracle):Started fmprjdeora-priv
jde_lsn(ocf::heartbeat:oralsnr):Started fmprjdeora-priv
3.3查看資源運行狀態
[root@fmpracora dinxin]# pcs resource
Resource Group: ac_group
fmpracora-vip(ocf::heartbeat:IPaddr2):Started fmpracora-priv
ac_fs(ocf::heartbeat:Filesystem):Started fmpracora-priv
cvs_lsn(ocf::heartbeat:oralsnr): Started fmpracora-priv
cvsdb(ocf::heartbeat:oracle):Started fmpracora-priv
Resource Group: fc_group
fmprfcora-vip(ocf::heartbeat:IPaddr2):Started fmprfcora-priv
fc_fs(ocf::heartbeat:Filesystem):Started fmprfcora-priv
arap2_lsn(ocf::heartbeat:oralsnr):Started fmprfcora-priv
arap2db(ocf::heartbeat:oracle):Started fmprfcora-priv
[root@fmpracora dinxin]#
3.4查看資源運行狀態
[root@fmpracora dinxin]# pcs status
Cluster name: fmprfcac_cluster
Last updated: Tue Aug 23 15:29:07 2016 Last change: Tue Aug 23 15:24:26 2016 by root via cibadmin on fmpracora-priv
Stack: corosync
Current DC: fmpracora-priv (version 1.1.13-10.el7-44eb2dd) - partition with quorum
2 nodes and 10 resources configured
Online: [ fmpracora-priv fmprfcora-priv ]
Full list of resources:
imm-fmprfcora(stonith:fence_ipmilan):Started fmpracora-priv
imm-fmpracora(stonith:fence_ipmilan):Started fmprfcora-priv
Resource Group: ac_group
fmpracora-vip(ocf::heartbeat:IPaddr2):Started fmpracora-priv
ac_fs(ocf::heartbeat:Filesystem):Started fmpracora-priv
cvs_lsn(ocf::heartbeat:oralsnr):Started fmpracora-priv
cvsdb(ocf::heartbeat:oracle): Started fmpracora-priv
Resource Group: fc_group
fmprfcora-vip(ocf::heartbeat:IPaddr2):Started fmprfcora-priv
fc_fs(ocf::heartbeat:Filesystem):Started fmprfcora-priv
arap2_lsn(ocf::heartbeat:oralsnr):Started fmprfcora-priv
arap2db(ocf::heartbeat:oracle):Started fmprfcora-priv
Failed Actions:
* cvsdb_monitor_120000 on fmpracora-priv 'not running' (7): call=245, status=complete, exitreason='none',
last-rc-change='Mon Aug 22 14:05:39 2016', queued=0ms, exec=0ms
* cvs_lsn_monitor_10000 on fmpracora-priv 'not running' (7): call=243, status=complete, exitreason='none',
last-rc-change='Tue Aug 23 10:45:13 2016', queued=0ms, exec=0ms
PCSD Status:
fmprfcora-priv: Online
fmpracora-priv: Online
Daemon Status:
corosync: active/enabled
pacemaker: active/enabled
pcsd: active/enabled
3.5standby節點
設置節點為standby，該節點的所有集群資源將自動切換到集群中的其它節點
[root@fmprpoora ~]# pcs cluster standby fmprpoora-priv
取消standby的命令是unstandby，該節點的所有集群資源將自動回切到本節點。
[root@fmprpoora ~]# pcs cluster unstandby fmprpoora-priv
3.6清除資源錯誤計數計
[root@fmpracora dinxin]# pcs resource cleanup
Waiting for 1 replies from the CRMd. OK
3.7維護節點
節點處於維護狀態時，該節點上的資源將不會自動啟動。
3.7.1 RHEL 6.8維護節點
[root@fmpridcora oracle]#pcs node maintenance fmpridcora-priv
[root@fmpridcora oracle]# pcs status
Cluster name: fmpridc-clu
Last updated: Tue Aug 23 14:56:21 2016 Last change: Tue Aug 23 14:55:59 2016 by root via crm_attribute on fmpridcora-priv
Stack: cman
Current DC: fmprbiora-priv (version 1.1.14-8.el6-70404b0) - partition with quoru m
2 nodes and 6 resources configured
Node fmpridcora-priv: maintenance
Online: [ fmprbiora-priv ]
Full list of resources:
imm-fmpridcora (stonith:fence_ipmilan): Started fmprbiora-priv
imm-fmprbiora(stonith:fence_ipmilan):Started fmpridcora-priv (unmanag ed)
Resource Group: idc_group
fmpridcora-vip (ocf::heartbeat:IPaddr2): Started fmpridcora-priv (unmanaged)
idc_fs(ocf::heartbeat:Filesystem): Started fmpridcora-priv (unmanaged)
fmidc_lsn(ocf::heartbeat:oralsnr):Started fmpridcora-priv (unmanaged)
fmidcdb(ocf::heartbeat:oracle): Started fmpridcora-priv (unmanaged)
PCSD Status:
fmpridcora-priv: Online
fmprbiora-priv: Online
清除節點維護狀態：
[root@fmpridcora oracle]# pcs node unmaintenance fmpridcora-priv
[root@fmpridcora oracle]# pcs status
Cluster name: fmpridc-clu
Last updated: Tue Aug 23 15:32:28 2016 Last change: Tue Aug 23 15:31:19 2016 by root via crm_attribute on fmpridcora-priv
Stack: cman
Current DC: fmprbiora-priv (version 1.1.14-8.el6-70404b0) - partition with quorum
2 nodes and 6 resources configured
Online: [ fmprbiora-priv fmpridcora-priv ]
Full list of resources:
imm-fmpridcora (stonith:fence_ipmilan): Started fmprbiora-priv
imm-fmprbiora(stonith:fence_ipmilan):Started fmpridcora-priv
Resource Group: idc_group
fmpridcora-vip(ocf::heartbeat:IPaddr2):Started fmpridcora-priv
idc_fs(ocf::heartbeat:Filesystem):Started fmpridcora-priv
fmidc_lsn(ocf::heartbeat:oralsnr):Started fmpridcora-priv
fmidcdb(ocf::heartbeat:oracle):Started fmpridcora-priv
PCSD Status:
fmpridcora-priv: Online
fmprbiora-priv: Online
3.7.2 RHEL 7.2
[root@fmpracora dinxin]# pcs property set maintenance-mode=true
[root@fmpracora dinxin]# pcs property show maintenance-mode
Cluster Properties:
maintenance-mode: true
[root@fmpracora dinxin]# pcs status
Cluster name: fmprfcac_cluster
Last updated: Tue Aug 23 15:11:46 2016 Last change: Tue Aug 23 15:09:15 2016 by root via cibadmin on fmpracora-priv
Stack: corosync
Current DC: fmpracora-priv (version 1.1.13-10.el7-44eb2dd) - partition with quorum
2 nodes and 10 resources configured
Online: [ fmpracora-priv fmprfcora-priv ]
Full list of resources:
imm-fmprfcora(stonith:fence_ipmilan):Started fmpracora-priv (unmanaged)
imm-fmpracora(stonith:fence_ipmilan):Started fmprfcora-priv (unmanaged)
Resource Group: ac_group
fmpracora-vip(ocf::heartbeat:IPaddr2):Started fmpracora-priv (unmanaged)
ac_fs(ocf::heartbeat:Filesystem):Started fmpracora-priv (unmanaged)
cvs_lsn(ocf::heartbeat:oralsnr):Stopped (unmanaged)
cvsdb (ocf::heartbeat:oracle): Started fmpracora-priv (unmanaged)
Resource Group: fc_group
fmprfcora-vip(ocf::heartbeat:IPaddr2):Started fmprfcora-priv (unmanaged)
fc_fs(ocf::heartbeat:Filesystem):Started fmprfcora-priv (unmanaged)
arap2_lsn(ocf::heartbeat:oralsnr):Started fmprfcora-priv (unmanaged)
arap2db(ocf::heartbeat:oracle):Started fmprfcora-priv (unmanaged)
PCSD Status:
fmprfcora-priv: Online
fmpracora-priv: Online
Daemon Status:
corosync: active/enabled
pacemaker: active/enabled
pcsd: active/enabled
清除維護模式
[root@fmpracora dinxin]# pcs property set maintenance-mode=false
清除維護模式后，系統狀態恢復正常。
[root@fmpracora dinxin]# pcs resource
Resource Group: ac_group
fmpracora-vip (ocf::heartbeat:IPaddr2): Started fmpracora-priv
ac_fs(ocf::heartbeat:Filesystem):Started fmpracora-priv
cvs_lsn(ocf::heartbeat:oralsnr):Started fmpracora-priv
cvsdb(ocf::heartbeat:oracle):Started fmpracora-priv
Resource Group: fc_group
fmprfcora-vip(ocf::heartbeat:IPaddr2):Started fmprfcora-priv
fc_fs(ocf::heartbeat:Filesystem):Started fmprfcora-priv
arap2_lsn(ocf::heartbeat:oralsnr):Started fmprfcora-priv
arap2db(ocf::heartbeat:oracle):Started fmprfcora-priv
3.8手工啟動資源
當系統處於維護模式時，可以手工啟動資源。啟動資源加上--full參數，可以看到詳細的輸出。
[root@fmprpoora ~]# pcs resource debug-start fmprpoora-vip
Operation start for fmprpoora-vip (ocf:heartbeat:IPaddr2) returned 0
>stderr: INFO: Adding inet address 10.0.101.7/24 with broadcast address 10.0.101.255 to device bond0
>stderr: INFO: Bringing device bond0 up
>stderr: INFO: /usr/libexec/heartbeat/send_arp -i 200 -r 5 -p /var/run/resource-agents/send_arp-10.0.101.7 bond0 10.0.101.7 auto not_used not_used
3.9手工停止資源
當系統處於維護模式時，可以停止資源。
[root@fmprmaora ~]# pcs resource debug-stop po_fs
Operation stop for po_fs (ocf:heartbeat:Filesystem) returned 0
>stderr: INFO: Running stop for /dev/mapper/povg-polv on /po-oradata
>stderr: INFO: Trying to unmount /po-oradata
>stderr: INFO: unmounted /po-oradata successfully
[root@fmprmaora ~]# pcs resource debug-stop fmprpoora-vip
Operation stop for fmprpoora-vip (ocf:heartbeat:IPaddr2) returned 0
>stderr: INFO: IP status = ok, IP_CIP=
[root@fmprmaora ~]#
3.10刪除cluster配置
[root@fmprpoora ~]# pcs cluster destroy
Shutting down pacemaker/corosync services...
Pacemaker Cluster Manager is already stopped [確定]
Stopping cluster:
Leaving fence domain... [確定]
Stopping gfs_controld... [確定]
Stopping dlm_controld... [確定]
Stopping fenced... [確定]
Stopping cman... [確定]
Unloading kernel modules... [確定]
Unmounting configfs... [確定]
Killing any remaining services...
Removing all cluster configuration files...
[root@fmprpoora ~]#
pcs resource disable<resource-group-name>