###############################################
在cs6.5,cs7.2上安裝grid11.2.0.1 和database11.2.0.1還是有問題
1.i386與i686的問題
2.cs7.2中的semmni內核參數問題 已經設置了kernel.sem = 250 32000 100 128這四個值，但安裝界面不認識，只認識semmni參數名
所以選擇cs5.5老版本
CRS-4664: Node oracle successfully pinned.
Adding daemon to inittab
CRS-4123: Oracle High Availability Services has been started.
ohasd is starting
ADVM/ACFS is not supported on centos-release-7-2.1511.el7.centos.2.10.x86_64
http://www.cnblogs.com/taowang2016/p/3316307.html ADVM/ACFS is not supported on centos-release-5-8.el5.centos
從Oracle11.2開始，ASM不僅是一個磁盤組，他還提供了了一個卷管理器，稱為ADVM（asm dynamic volume manager）。ADVM向用戶提供卷管理服務，並提供標准的磁盤設備驅動程序。利用ADVM，可以在卷組中創建一個或多個卷，每個卷對應操作系統中的 一個設備文件，這些卷是可以動態擴展的，就像操作系統中的卷或者利用第三方軟件創建的卷一樣，應用程序也可以對ADVM卷中的數據進行讀寫操作。
在 ADVM卷中可以創建ACFS文件系統，ACFS是一種跨平台的、可擴展的集群文件系統，多個節點可以同時訪問asfs中的文件。ACFS不僅可以作為 oracle數據庫軟件的安裝路徑，還可以用來存儲數據庫中的警告文件和跟蹤文件，還可以存儲諸如視頻、聲音、圖像、文本等類型的文件。
但是在CentOS5.8上面安裝rac的時候，卻發現 ADVM/ACFS is not supported on centos-release-5-8.el5.centos(同樣適用於單機環境的asm安裝，在安裝grid的最后執行root.sh的時候出現 的。)
ADVM/ACFS is not supported on centos-release-5-5.el5.centos 解決方法
centOS 5.5 安裝 ORACLE 11G RAC 問題匯總
########################################################
oracle grid infrastrucure GI
提供了自動重啟功能和asm功能
1、下載linux.x64_11gR2_grid.zip
http://www.oracle.com/technetwork/database/enterprise-edition/downloads/112010-linx8664soft-100572.html
2、上傳到目標機上的獨立目錄
mkdir /mnt
chmod -R 777 oracle
cd /mnt;ls
cksum linux.x64_11gR2_grid.zip
unzip linux.x64_11gR2_grid.zip
3、在虛擬機里添加兩塊20g硬盤做asm,centos5.8,2G內存
4、環境配置
vi /etc/security/limits.conf
#for oracle
oracle soft nproc 2048
oracle hard nproc 16384
oracle soft nofile 2048
oracle hard nofile 65536
oracle soft stack 10240
oracle hard stack 32768
#for grid
grid soft nproc 2048
grid hard nproc 16384
grid soft nofile 2048
grid hard nofile 65536
grid soft stack 10240
grid hard stack 32768
vi /etc/security/limits.d/90-nproc.conf
* soft nproc 1024
將上面改為
* - nproc 16384
vi /etc/hosts
#public ip
192.168.30.31 rac01
192.168.30.41 rac02
#private ip
10.10.10.31 rac01prv
10.10.10.32 rac02prv
#vip
192.168.30.32 rac01vip
192.168.30.42 rac01vip
#scanip
192.168.30.50 scanip
vi /etc/sysctl.conf
fs.aio-max-nr = 1048576
fs.file-max = 6815744
kernel.shmmni = 4096
kernel.sem = 250 32000 100 128
net.ipv4.ip_local_port_range = 9000 65500
net.core.rmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 1048576
下面兩個參數默認是有的，在cs6.8desktop中，所以不需要設置了
kernel.shmall = 2097152
kernel.shmmax = 4294967295
sysctl -p
創建用戶和組
oracle
grid
oracle restart + oracle asm
(grid軟件的兩個功能1 ASM 2 restart）
grid軟件就是相當於一個單機版的群集，當你的系統重啟的時候數據庫會自動跟着啟動。這個軟件在oracle里面必須要裝。
首先做2個用戶：oracle和grid
建立5個組：
1數據庫軟件組dba（管理數據庫）和oper
2grid軟件組asmadmin（管理ASM），asmdba（可以訪問ASM）和asmoper
在oracle組中的用戶所擁有的權限各不相同，並且非常嚴格。
用戶及組處理
groupadd -g 501 oinstall
groupadd -g 502 dba
groupadd -g 503 oper
groupadd -g 504 asmadmin
groupadd -g 505 asmdba
groupadd -g 506 asmoper
useradd -g oinstall -G asmadmin,asmdba,asmoper,oper,dba grid
useradd -g oinstall -G dba,asmdba,oper oracle
或
usermod -g oinstall -G dba,asmdba,oper oracle
passwd oracle
passwd grid
id oracle
id grid
創建安裝目錄
mkdir -p /u01/app/oracle/product/11.2.0/dbhome_1
chown -R oracle:oinstall /u01
mkdir -p /u01/app/oracle/product/11.2.0/grid
chown -R grid:oinstall /u01/app/oracle/product/11.2.0/grid
chmod -R 775 /u01
下面的這種處理方法
mkdir -p /oracle/app/grid
mkdir -p /oracle/app/11.2.0/grid
chown -R grid:oinstall /oracle
mkdir -p /oracle/app/oraInventory
chown -R grid:oinstall /oracle/app/oraInventory
mkdir -p /oracle/app/oracle
mkdir -p /oracle/app/oracle/product/11.2.0/db_1
chown -R oracle:oinstall /oracle/app/oracle/
chmod -R 775 /oracle/
目錄處理
這里需說明一點，很多時候，Grid Infrastructure安裝中出現問題，都和 grid與oracle的目錄以及用戶組混亂造成。
為了避免混亂，對grid用戶的ORACLE_BASE目錄，設置為/u02/app/grid
mkdir -p /u02/app/oracle/product/11.2.0/grid
chown -R grid:oinstall /u02
chmod -R 775 /u02
mkdir -p /u01/app/oracle/product/11.2.0/dbhome_1
chown -R oracle:oinstall /u01
chmod -R 775 /u01
設置用戶環境變量
oracle 和grid 兩個用戶都要配置。
vi /home/oracle/.bash_profile
加入：
export ORACLE_BASE=/u01/app/oracle
export ORACLE_SID=orcl (數據庫實例的名字)
export ORACLE_HOME=$ORACLE_BASE/product/11.2.0/dbhome_1 (oracle軟件的安裝目錄)
export PATH=$PATH:$ORACLE_HOME/bin
vi /home/grid/.bash_profile
加入：
export ORACLE_BASE=/u01/app/oracle
export ORACLE_SID=+ASM (asm實例名)
export ORACLE_HOME=$ORACLE_BASE/product/11.2.0/grid (grid安裝目錄)
export PATH=$PATH:$ORACLE_HOME/bin
5、安裝asm
下載包並安裝：本來有三個包，在cs6下官網只有兩個包，另外一個包叫kmod-oracleasm
yum install kmod-oracleasm
rpm -ivh oracleasm*
===================================== rpm -ivh http://mirror.symnds.com/distributions/CentOS-vault/6.7/os/x86_64/Packages/compat-libcap1-1.10-1.x86_64.rpm yum install kmod-oracleasm -y rpm -ivh oracleasm* fdisk /dev/sdb n,p,1,w /etc/init.d/oracleasm configure grid,asmadmin,y,y /etc/init.d/oracleasm createdisk crs /dev/sdb1 /etc/init.d/oracleasm createdisk data1 /dev/sdc1 /etc/init.d/oracleasm createdisk data2 /dev/sdd1 /etc/init.d/oracleasm createdisk fra /dev/sde1 oracleasm listdisks mkfs.ext4 /dev/sdd1 ===================================
分區之前添加的兩塊盤，然后初始化
/etc/init.d/oracleasm configure
grid
oinstall
y
y
oracle user login
unzip p13390677_112040_Linux-x86-64_1of7.zip
unzip p13390677_112040_Linux-x86-64_2of7.zip
cd database/
./runInstaller
su -
2g到16g內存的話，swap空間等於內存大小，16G以上，等於內存大小
grep SwapTotal /proc/meminfo
dd if=/dev/zero of=swapfile bs=1M count=1024
mkswap swapfile
swapon swapfile
free
vi /etc/fstab
/root/swapfile swap swap defaults 0 0
yum install gcc
yum install libaio-devel
yum install elfutils-libelf-devel
yum install gcc-c++
yum install compat-libstdc++-33
rpm -qa|grep ksh
rpm -ivh http://mirror.symnds.com/distributions/CentOS-vault/5.8/os/x86_64/CentOS/pdksh-5.2.14-37.el5.x86_64.rpm
rpm -ivh http://download.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
yum install rlwrap
rpm -ivh http://download.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm yum install rlwrap 有時會報下面這個錯， [Errno -1] repomd.xml does not match metalink for epel Trying other mirror. 這時有兩個解決辦法， 一是直接下載下面的文件，但有可能在linux下用wget與curl報404錯誤，這時，在圖形界面下的瀏覽器直接可以下載，然后傳到服務器上，再rpm -ivh 就可以了 http://mirrors.sohu.com/fedora-epel/6Server/x86_64/rlwrap-0.42-1.el6.x86_64.rpm 二是修改epel.repo文件，將mirrors行注釋，啟用baseurl行，也能下 2016年10月29日 星期六 18時45分45秒 才解決，花了大約一個小時解決這個小問題，唉，太坑了
grid need to install follow packages,and oracle might not need to install it.
rpm -ivh http://mirror.symnds.com/distributions/CentOS-vault/6.7/os/x86_64/Packages/compat-libcap1-1.10-1.x86_64.rpm
yum install kmod-oracleasm
rpm -ivh oracleasm*
mount /dev/sr0 /media
rpm -ivh /media/Packages/compat-libcap1-1.10-1.x86_64.rpm
rpm -ivh http://mirror.symnds.com/distributions/CentOS-vault/5.8/os/x86_64/CentOS/pdksh-5.2.14-37.el5.x86_64.rpm
yum install gcc
yum install libaio-devel
yum install elfutils-libelf-devel
yum install gcc-c++
yum install compat-libstdc++-33
或者下面一條命令
yum install gcc libaio-devel elfutils-libelf-devel gcc-c++ compat-libstdc++-33
編輯文件 vi /etc/pam.d/login，添加如下行(11gR1有要求，R2沒有要求)：
session required pam_limits.so
編輯 vi /etc/profile 文件，添加如下行(11gR1有要求，R2沒有要求)：
if [ $USER = "oracle" ]; then
if [ $SHELL = "/bin/ksh" ]; then
ulimit -p 16384
ulimit -n 65536
else
ulimit -u 16384 -n 65536
fi
fi
該配置在用戶oracle登錄時會立即生效，如果當前 oracle 用戶已經登錄可退出后重新登錄使之生效。
/etc/init.d/oracleasm configure
grid
asmadmin
y
y
只在一個節點上創建就可以了，另一個掃描一下就可以了
/etc/init.d/oracleasm createdisk rac /dev/mapper/mpathbp1
/etc/init.d/oracleasm deletedisk rac
oracleasm scandisks
oracleasm listdisks
6、用grid用戶安裝
用grid登錄
./runInstaller
出現界面選第二個，單機
可能不出現asm磁盤，首先前面的環境設置先做好，其次可以更改路徑/dev/oracleasm/disks.
high 有三份相同的數據
normal 有兩份相同的數據
external 沒有冗余
硬盤30G,swap10G,不然這里檢測條件時會出問題。
安裝依賴包，下面這個包一定要安裝，如果不提前裝好，會出現下面的問題
[root@oracle Packages]# rpm -ivh compat-libcap1-1.10-1.x86_64.rpm
Preparing... ########################################### [100%]
1:compat-libcap1 ########################################### [100%]
[root@oracle Packages]# pwd
/media/cdrom/Packages
過程為，之前沒有裝包---》報錯后根據報錯安裝這個包---》再次運行腳本會報下面的錯---》根據提示還是報錯（加-deconfig -force參數）---》所以最好提前裝好
[root@rac1 11.2.0]# /g01/oraInventory/orainstRoot.sh
Changing permissions of /g01/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.
Changing groupname of /g01/oraInventory to oinstall.
The execution of the script is complete.
[root@rac1 11.2.0]# /g01/app/11.2.0/grid/root.sh
Running Oracle 11g root.sh script...
The following environment variables are set as:
ORACLE_OWNER= grid
ORACLE_HOME= /g01/app/11.2.0/grid
Enter the full pathname of the local bin directory: [/usr/local/bin]:
Copying dbhome to /usr/local/bin ...
Copying oraenv to /usr/local/bin ...
Copying coraenv to /usr/local/bin ...
Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root.sh script.
Now product-specific root actions will be performed.
2013-10-10 03:41:35: Parsing the host name
2013-10-10 03:41:35: Checking for super user privileges
2013-10-10 03:41:35: User has super user privileges
Using configuration parameter file: /g01/app/11.2.0/grid/crs/install/crsconfig_params
Creating trace directory
/g01/app/11.2.0/grid/bin/clscfg.bin: error while loading shared libraries: libcap.so.1: cannot open shared object file: No such file or directory
Failed to create keys in the OLR, rc = 127, 32512
OLR configuration failed
第二次執行root.sh時，出現下面的報錯。
/u01/app/11.2.0/grid/crs/install/roothas.pl -deconfig -force -verbose
[root@oracle Packages]# /u02/app/grid/product/11.2.0/grid/root.sh
Running Oracle 11g root.sh script...
The following environment variables are set as:
ORACLE_OWNER= grid
ORACLE_HOME= /u02/app/grid/product/11.2.0/grid
Enter the full pathname of the local bin directory: [/usr/local/bin]:
The file "dbhome" already exists in /usr/local/bin. Overwrite it? (y/n)
[n]: y
Copying dbhome to /usr/local/bin ...
The file "oraenv" already exists in /usr/local/bin. Overwrite it? (y/n)
[n]: y
Copying oraenv to /usr/local/bin ...
The file "coraenv" already exists in /usr/local/bin. Overwrite it? (y/n)
[n]: y
Copying coraenv to /usr/local/bin ...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root.sh script.
Now product-specific root actions will be performed.
2016-09-02 11:39:34: Checking for super user privileges
2016-09-02 11:39:34: User has super user privileges
2016-09-02 11:39:34: Parsing the host name
Using configuration parameter file: /u02/app/grid/product/11.2.0/grid/crs/install/crsconfig_params
Improper Oracle Clusterware configuration found on this host
Deconfigure the existing cluster configuration before starting
to configure a new Clusterware
run '/u02/app/grid/product/11.2.0/grid/crs/install/rootcrs.pl -deconfig'
to configure existing failed configuration and then rerun root.sh
[root@oracle Packages]#
可以忽略i386的包
root用戶下
1、以文件系統增加(以下是以增加512M的文件系統為例)
dd if=/dev/zero of=/home/oracle/myswaps/swapfile1 bs=1M count=512
注：of后面的路徑一定不能是/dev下，否則在激活swap文件時報參數無效
2、創建swap文件
mkswap /home/oracle/myswaps/swapfile1
3、激活swap文件
swapon /home/oracle/myswaps/swapfile1
4、此時用free -m就可以看到swap空間已經增加了，但是重啟后就不會存在了
5、添加重啟后還會生效，vi /etc/fstab添加如下內容：
/home/oracle/myswaps/swapfile1 swap swap defaults 0 0
執行兩個腳本，並安裝完成后
注意：如果之前安裝過gi，在執行root.sh的時候需要刪除之前的配置信息：命令如下：
/u01/app/11.2.0/grid/crs/install/roothas.pl -deconfig -force -verbose
CRS-4124: Oracle High Availability Services startup failed.
CRS-4000: Command Start failed, or completed with errors.
ohasd failed to start: Inappropriate ioctl for device
ohasd failed to start at/u01/app/11.2.0/grid/crs/install/rootcrs.pl line 443.
第一次安裝11gR2 RAC的時候就遇到了這個11.0.2.1的經典問題，上網一查才知道這是個oracle bug，
具體可以參考：https://forums.oracle.com/thread/2352285
解決辦法也很簡單，
就是在執行root.sh之前執行以下命令
/bin/dd if=/var/tmp/.oracle/npohasd of=/dev/null bs=1024 count=1
如果出現
/bin/dd: opening`/var/tmp/.oracle/npohasd': No such file or directory
的時候文件還沒生成就繼續執行，直到能執行為止，一般出現Adding daemon to inittab這條信息的時候執行dd命令。
另外還有一種解決方法就是更改文件權限
chown root:oinstall /var/tmp/.oracle/npohasd
重新執行root.sh之前別忘了刪除配置：/u01/app/11.2.0/grid/crs/install/roothas.pl -deconfig -force -verbose
接下來驗證一下
ps -ef|grep asm
su - grid
sqlplus / as sysasm
select status from v$instance
由於在安裝GI的時候只能創建一個磁盤組，如果需要增加磁盤組的話，在GI安裝完成后，使用ASMCA命令增加磁盤組。
grid安裝完成后，重啟操作系統，啟動ASM實例報錯如下：
[grid@oragrid ~]$ sqlplus / as sysasm SQL*Plus: Release 11.2.0.4.0 Production on Sun Jun 7 16:21:31 2015 Copyright (c) 1982, 2013, Oracle. All rights reserved. Connected to an idle instance. SQL> startup ORA-01078: failure in processing system parameters ORA-29701: unable to connect to Cluster Synchronization Service [grid@oragrid ~]$ crsctl check css CRS-4639: Could not contact Oracle High Availability Services CRS-4000: Command Check failed, or completed with errors. [grid@oragrid ~]$ crsctl check has CRS-4639: Could not contact Oracle High Availability Services 第一種方法：
切換到root用戶執行啟動init.ohasd [root@oragrid ~]# /etc/init.d/init.ohasd run & 再切換到grid用戶check，可能有點慢，等一會就會online [grid@oragrid ~]$ crsctl check has CRS-4638: Oracle High Availability Services is online [grid@oragrid ~]$ crsctl check css CRS-4529: Cluster Synchronization Services is online [grid@oragrid ~]$ crsctl stat resource -t -init -------------------------------------------------------------------------------- NAME TARGET STATE SERVER STATE_DETAILS -------------------------------------------------------------------------------- Local Resources -------------------------------------------------------------------------------- ora.DATA.dg ONLINE OFFLINE oragrid ora.LISTENER.lsnr ONLINE OFFLINE oragrid STARTING ora.asm ONLINE OFFLINE oragrid Instance Shutdown ora.ons OFFLINE OFFLINE oragrid -------------------------------------------------------------------------------- Cluster Resources -------------------------------------------------------------------------------- ora.cssd 1 ONLINE OFFLINE ora.diskmon 1 OFFLINE OFFLINE ora.evmd 1 ONLINE OFFLINE STARTING 原來是開機啟動的init.ohasd沒有起作用，重新啟動一次就好了。 直接殺掉該進程，會自動重新生成一個。 [root@oragrid ~]# ps -ef | grep init.ohasd root 972 1 0 22:01 ? 00:00:00 /bin/sh /etc/init.d/init.ohasd run root 2474 2450 0 22:04 pts/0 00:00:00 grep init.ohasd [root@oragrid ~]# kill -9 972 [root@oragrid ~]# ps -ef | grep init.ohasd root 2538 1 0 22:05 ? 00:00:00 /bin/sh /etc/init.d/init.ohasd run root 2638 2450 0 22:05 pts/0 00:00:00 grep init.ohasd [root@oragrid ~]# su - grid [grid@oragrid ~]$ crsctl check has CRS-4638: Oracle High Availability Services is online [grid@oragrid ~]$ crsctl check css CRS-4529: Cluster Synchronization Services is online [grid@oragrid ~]$ crs_stat -t Name Type Target State Host ------------------------------------------------------------ ora.DATA.dg ora....up.type ONLINE ONLINE oragrid ora....ER.lsnr ora....er.type ONLINE ONLINE oragrid ora.asm ora.asm.type ONLINE ONLINE oragrid ora.cssd ora.cssd.type ONLINE ONLINE oragrid ora.diskmon ora....on.type OFFLINE OFFLINE ora.evmd ora.evm.type ONLINE ONLINE oragrid ora.ons ora.ons.type OFFLINE OFFLINE 再進入sqlplus就可以看到ASM實例已經啟動了。但是每次操作系統重啟都會出現這個問題，上述方法治標不治本，且沒有找到最終的原因，請教高手解決。 第二種方法： 在網上找到這篇文章，http://blog.csdn.net/tianlesoftware/article/details/8207629，說這個問題是11.2.0.1的bug，但是我現在安裝的版本是11.2.0.4，使用這篇文章的解決辦法，在root用戶下執行下面的命令： [root@oragrid ~]# /bin/dd if=/var/tmp/.oracle/npohasd of=/dev/null bs=1024 count=1 ^C0+0 records in 0+0 records out 0 bytes (0 B) copied, 13.6369 s, 0.0 kB/s 然后在grid用戶下查看has和css狀態，可以看到已經啟動了 [grid@oragrid ~]$ crsctl check has CRS-4638: Oracle High Availability Services is online [grid@oragrid ~]$ crsctl check css CRS-4529: Cluster Synchronization Services is online [grid@oragrid ~]$ crs_stat -t Name Type Target State Host ------------------------------------------------------------ ora.DATA.dg ora....up.type ONLINE ONLINE oragrid ora.FRA.dg ora....up.type ONLINE ONLINE oragrid ora....ER.lsnr ora....er.type ONLINE ONLINE oragrid ora.asm ora.asm.type ONLINE ONLINE oragrid ora.cssd ora.cssd.type ONLINE ONLINE oragrid ora.diskmon ora....on.type OFFLINE OFFLINE ora.evmd ora.evm.type ONLINE ONLINE oragrid ora.ons ora.ons.type OFFLINE OFFLINE ora.stone.db ora....se.type OFFLINE OFFLINE 居然這個方法管用，但是系統重啟后問題依舊，難道這個bug在11.2.0.4還存在嗎？
以grid用戶安裝grid完成后。
接下來，以oracle用戶的身份，安裝oracle數據庫。