OS：RedHat EL6.0
Oracle： Oracle 11gR2
在Oracle 11gR2，構建RAC時可以通過ASM創建asm disk，但是需要安裝asmlib相關軟件；對於RedHat EL6.0，Oracle的asm不能很好的支持，所以可以通過udev來創建asm disk，就不需要安裝asmlib相關軟件。
一、查看系統環境
內核版本：
[root@rac01 rules.d]# uname -a
Linux rac01 2.6.32-642.6.2.el6.x86_64 #1 SMP Wed Oct 26 06:52:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
系統用戶：
[root@rac01 rules.d]# id grid
uid=1100(grid) gid=1000(oinstall) groups=1000(oinstall),1300(dba),1200(asmadmin),1201(asmdba),1202(asmoper)
二、創建共享磁盤
在Windows server 2012R2 服務器上啟用iSCSI服務，並新建iSCSI共享磁盤
選擇有足夠容量的卷
給虛擬磁盤命名並划分大小，然后配置iSCSI目標
命名訪問目標，添加訪問程序，選擇"IP地址類型"，輸入RAC 節點IP（將兩個節點的IP都添加上）
繼續下一步，不啟用驗證服務，確認創建，創建完成后會在管理頁面顯示出已經創建好的共享磁盤
三、配置共享存儲
在節點1和節點2上分別安裝iSCSI Initiator，以節點一為例，主要涉及以下命令
yum -y install iscsi-initiator-utils
chkconfig iscsid on
chkconfig iscsi on
最后發現並且掛載iscsi target
[root@rac01 rules.d]# iscsiadm -m discovery -t sendtargets -p 192.168.2.70
192.168.2.70:3260,1 iqn.1991-05.com.microsoft:win-ct207djdp08-max-002-target
192.168.2.70:3260,1 iqn.1991-05.com.microsoft:win-ct207djdp08-test1-target
[root@rac01 rules.d]# iscsiadm -m node -T iqn.1991-05.com.microsoft:win-ct207djdp08-max-002-target --login
成功后，用fdisk -l命令可以看到多出的共享磁盤，進行分區，分區結果如下圖
四、建立UDEV規則文件
查看磁盤UUID：
[root@rac01 rules.d]# /sbin/scsi_id -g -u -d /dev/sdb
360003ff44dc75adca44127e408e97955
創建UDEV配置文件：
[root@rac01 rules.d]# cd /etc/udev/rules.d/
[root@rac01 rules.d]# vim 99-oracle-asmdisk.rules
KERNEL=="sdb5", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", RESULT=="360003ff44dc75adca44127e408e97955", NAME="asm_disk1", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sdb6", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", RESULT=="360003ff44dc75adca44127e408e97955", NAME="asm_disk2", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sdb7", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", RESULT=="360003ff44dc75adca44127e408e97955", NAME="asm_disk3", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sdb8", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", RESULT=="360003ff44dc75adca44127e408e97955", NAME="asm_disk4", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sdb9", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", RESULT=="360003ff44dc75adca44127e408e97955", NAME="asm_disk5", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sdb10", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", RESULT=="360003ff44dc75adca44127e408e97955", NAME="asm_disk6", OWNER="grid", GROUP="asmadmin", MODE="0660"
五、啟動UDEV service
載入配置文件：
[root@rac01 rules.d]# udevadm control --reload-rules
啟動UDEV service:
[root@rac01 rules.d]# start_udev
查看asm disks:
[root@rac01 rules.d]# ls -l /dev/asm*
brw-rw---- 1 grid asmadmin 8, 21 Nov 25 11:22 /dev/asm_disk1
brw-rw---- 1 grid asmadmin 8, 22 Nov 25 11:22 /dev/asm_disk2
brw-rw---- 1 grid asmadmin 8, 23 Nov 25 11:22 /dev/asm_disk3
brw-rw---- 1 grid asmadmin 8, 24 Nov 25 11:22 /dev/asm_disk4
brw-rw---- 1 grid asmadmin 8, 25 Nov 25 11:22 /dev/asm_disk5
brw-rw---- 1 grid asmadmin 8, 26 Nov 25 11:22 /dev/asm_disk6
六、添加額外ASM磁盤
安裝完RAC環境后，突然覺得之前配置的ASM磁盤不夠用，於是打算再添加一塊大容量的ASM磁盤；前面操作都無差異，這里補充后續操作。
查看asm disks:
[root@rac01 rules.d]# ls -l /dev/asm*
[root@rac01 rules.d]# ls -l /dev/asm*
brw-rw---- 1 grid asmadmin 8, 21 Nov 25 11:22 /dev/asm_disk1
brw-rw---- 1 grid asmadmin 8, 22 Nov 25 11:22 /dev/asm_disk2
brw-rw---- 1 grid asmadmin 8, 23 Nov 25 11:22 /dev/asm_disk3
brw-rw---- 1 grid asmadmin 8, 24 Nov 25 11:22 /dev/asm_disk4
brw-rw---- 1 grid asmadmin 8, 25 Nov 25 11:22 /dev/asm_disk5
brw-rw---- 1 grid asmadmin 8, 26 Nov 25 11:22 /dev/asm_disk6
brw-rw---- 1 grid asmadmin 8, 33 Nov 25 09:42 /dev/asm_disk7
在節點1中，使用grid用戶運行asmca:
成功彈出ASM設置界面，選擇Create
輸入磁盤組名，選擇External（外部冗余）（因為已經存在常規冗余磁盤組，為了省事兒這里選擇外部冗余），選擇磁盤：
點擊OK
點擊OK回到初始界面，點擊右下角Exit，然后選擇Yes保留操作並退出
使用oracle 賬戶登錄數據庫並查詢ASM磁盤組信息：
[root@rac01 ~]# su - oracle
[oracle@rac01 ~]$ sqlplus / as sysdba
SQL*Plus: Release 11.2.0.4.0 Production on Fri Nov 25 13:09:54 2016
Copyright (c) 1982, 2013, Oracle. All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options
SQL> select name,total_mb from v$asm_diskgroup;
NAME TOTAL_MB
------------------------------ ----------
DATA_DB 16378
DATA 6144
CY 102398
七、額外問題
完成上面的操作后，發現節點2出現了一些問題
[oracle@rac02 admin]$ sqlplus / as sysdba
SQL*Plus: Release 11.2.0.4.0 Production on Fri Nov 25 13:13:45 2016
Copyright (c) 1982, 2013, Oracle. All rights reserved.
Connected to an idle instance.
SQL> select name,total_mb from v$asm_diskgroup;
select name,total_mb from v$asm_diskgroup
*
ERROR at line 1:
ORA-01034: ORACLE not available
Process ID: 0
Session ID: 0 Serial number: 0
實例似乎被關閉了，重啟實例后，發現新加的ASM磁盤組掛載異常
SQL> startup
ORACLE instance started.
Total System Global Area 1603411968 bytes
Fixed Size 2253664 bytes
Variable Size 1056967840 bytes
Database Buffers 536870912 bytes
Redo Buffers 7319552 bytes
Database mounted.
Database opened.
SQL> select name,total_mb from v$asm_diskgroup;
NAME TOTAL_MB
------------------------------ ----------
DATA_DB 16378
DATA 6144
CY 0
[grid@rac02 admin]$ ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 3
Total space (kbytes) : 262120
Used space (kbytes) : 3072
Available space (kbytes) : 259048
ID : 277519239
Device/File Name : +DATA
Device/File integrity check succeeded
Device/File not configured
Device/File not configured
Device/File not configured
Device/File not configured
Cluster registry integrity check succeeded
Logical corruption check bypassed due to non-privileged user
手動掛載：
[grid@rac02 admin]$ sqlplus / as sysasm
SQL*Plus: Release 11.2.0.4.0 Production on Fri Nov 25 13:30:30 2016
Copyright (c) 1982, 2013, Oracle. All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options
SQL> alter diskgroup CY mount;
Diskgroup altered.
SQL> select GROUP_NUMBER,NAME,TYPE,ALLOCATION_UNIT_SIZE,STATE from v$asm_diskgroup;
GROUP_NUMBER NAME TYPE ALLOCATION_UNIT_SIZE STATE
------------ ------------------------------ ------ --------------------
1 DATA_DB NORMAL 1048576 MOUNTED
2 DATA NORMAL 1048576 MOUNTED
3 CY EXTERN 1048576 MOUNTED
再次查詢ASM磁盤組信息：
[oracle@rac02 admin]$ sqlplus / as sysdba
SQL*Plus: Release 11.2.0.4.0 Production on Fri Nov 25 13:40:10 2016
Copyright (c) 1982, 2013, Oracle. All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options
SQL> select name,total_mb from v$asm_diskgroup;
NAME TOTAL_MB
------------------------------ ----------
DATA_DB 16378
DATA 6144
CY 102398
完成后發現CRS服務異常：
[grid@rac02 admin]$ crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4529: Cluster Synchronization Services is online
CRS-4534: Cannot communicate with Event Manager
排查后發現是硬盤名稱發生了變化，修改完相關配置文件（vim /etc/udev/rules.d/99-oracle-asmdisk.rules），CRS服務正常：
[grid@rac02 ~]$ crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
然后發現因為之前的異常操作，導致監聽異常：
[grid@rac02 admin]$ lsnrctl status
LSNRCTL for Linux: Version 11.2.0.4.0 - Production on 25-NOV-2016 15:19:25
Copyright (c) 1991, 2013, Oracle. All rights reserved.
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER)))
STATUS of the LISTENER
------------------------
Alias LISTENER
Version TNSLSNR for Linux: Version 11.2.0.4.0 - Production
Start Date 25-NOV-2016 15:11:22
Uptime 0 days 0 hr. 8 min. 2 sec
Trace Level off
Security ON: Local OS Authentication
SNMP OFF
Listener Parameter File /u01/app/11.2.0/grid/network/admin/listener.ora
Listener Log File /u01/app/11.2.0/grid/log/diag/tnslsnr/rac02/listener/alert/log.xml
Listening Endpoints Summary...
(DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER)))
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=127.0.0.1)(PORT=1521)))
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.2.209)(PORT=1521)))
Services Summary...
Service "+ASM" has 1 instance(s).
Instance "+ASM2", status READY, has 1 handler(s) for this service...
Service "rac" has 1 instance(s).
Instance "rac2", status READY, has 1 handler(s) for this service...
Service "racXDB" has 1 instance(s).
Instance "rac2", status READY, has 1 handler(s) for this service...
The command completed successfully
以節點1為例
修改兩個節點的監聽文件：
[root@rac01 ~]# su - grid
[grid@rac01 ~]$ cd /u01/app/11.2.0/grid/network/admin
[grid@rac01 admin]$ ls
endpoints_listener.ora listener.ora shrept.lst
endpoints_listener.ora.bak.rac01 listener.ora.bak.rac01 sqlnet.ora
listener16112111AM0504.bak samples
[grid@rac01 admin]$ vim endpoints_listener.ora
#將HOST參數改為本機IP
LISTENER_RAC01=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=rac01-vip)(PORT=1521))(ADDRESS=(PROTOCOL=TCP)(HOST=192.168.2.206)(PORT=1521)(IP=FIRST))))
修改TNS配置文件：
[oracle@rac01 ~]$ cd /u01/app/oracle/product/11.2.0/db_1/network/admin
[oracle@rac01 admin]$ ls
samples shrept.lst tnsnames.ora
[oracle@rac01 admin]$ vim tnsnames.ora
RAC =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = max-cluster)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = rac)
)
)
RAC1 =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = max-cluster)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = rac)
)
)
修改hosts文件：
[root@rac01 ~]# vim /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost.localdomain localhost6 localhost6.localdomain6
#eth0 public ip
192.168.2.206 rac01
192.168.2.207 rac02
#eth1 private ip
192.168.3.206 rac01-priv
192.168.3.207 rac02-priv
#virtual ip
192.168.2.208 rac01-vip
192.168.2.209 rac02-vip
#scan ip
192.168.2.210 max-cluster
修改網卡信息：
[root@rac01 ~]# vim /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE="eth0"
NM_CONTROLLED="yes"
ONBOOT=yes
HWADDR=00:0C:29:8F:C2:C6
BOOTPROTO=static
IPADDR=192.168.2.206
NETMASK=255.255.255.0
GATEWAY=192.168.2.1
DNS1=192.168.2.1
[root@rac01 ~]# vim /etc/sysconfig/network-scripts/ifcfg-eth1
DEVICE="eth1"
NM_CONTROLLED="yes"
ONBOOT=yes
HWADDR=00:0C:29:8F:C2:D0
BOOTPROTO=static
IPADDR=192.168.3.206
NETMASK=255.255.255.0
---------------------------------------------------------------------------------------------------------------