由於測試環境打19c補丁失敗，導致節點2 d2 crs無法啟動！打補丁沒有有效備份，親人兩行淚
先刪再加節點。
一、刪除節點
1.1 刪除實例
參考 https://www.cnblogs.com/binliubiao/p/11996092.html Deleting a Cluster Node on Linux and UNIX Systems https://docs.oracle.com/en/database/oracle/oracle-database/19/cwadd/
adding-and-deleting-cluster-nodes.html#GUID-8ADA9667-EC27-4EF9-9F34-C8F65A757F2A 刪除可以參考上述博客,添加節點建議參考MOS文檔 Oracle E-Business Suite Release 12.2: Adding or Deleting 11gR2 or 12c1 Oracle RAC Nodes (Doc ID 1570554.1) Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 測試環境RAC2節點,打補丁2節點安裝失敗,因此對節點2進行刪除后,再添加。 刪除實例，1.1都在正常的節點1執行操作 【由於節點2集群crs無法啟動,因此刪除操作在節點1執行] srvctl stop instance -db db_unique_name -node node_name srvctl relocate server -servers "server_name_list" -serverpool Free srvctl config database -db db_unique_name oracle$ srvctl config database -db orcl dbca -silent -deleteInstance -nodeList d2 -gdbName orcl -instanceName orcl2 -sysDBAUserName sysdba -sysDBAPassword oracle --刪除實例需要本地實例已啟動！ [FATAL] [DBT-11502] The instance (orcl1) is not running on the local node. CAUSE: A locally running instance is required to perform the selected operation. ACTION: Specify a locally running database, or execute DBCA on a node where the database instance is running. $ srvctl start instance -d orcl -instance orcl1 $ dbca -silent -deleteInstance -nodeList d2 -gdbName orcl -instanceName orcl2 Prepare for db operation 40% complete Deleting instance 48% complete 52% complete 56% complete 60% complete 64% complete 68% complete 72% complete 76% complete 80% complete Completing instance management. 100% complete $ srvctl stop database -d orcl 本次測試環境3套DB 均為啟動存活的節點1實例,進行刪除實例操作。 SQL> alter system set enable_ddl_logging=true; $ dbca -silent -deleteInstance -nodeList d2 -gdbName abc -instanceName abc2 --觀察DB Alert日志,並沒有ddl操作,而是上次了實例2的redo and undo，並且RAC相關參數設置了一下！ 2020-04-15T06:57:54.304305+08:00 Deleted Oracle managed file +DATA/ABC/ONLINELOG/group_3.309.1036713505 Deleted Oracle managed file +DATA/ABC/ONLINELOG/group_4.310.1036713537 Deleted Oracle managed file +DATA/ABC/DATAFILE/undotbs2.308.1036712911 2020-04-15T06:57:55.090943+08:00 ALTER SYSTEM RESET undo_tablespace SCOPE=SPFILE SID='abc2'; 2020-04-15T06:57:55.098168+08:00 ALTER SYSTEM RESET instance_number SCOPE=SPFILE SID='abc2'; 2020-04-15T06:57:55.107884+08:00 ALTER SYSTEM RESET thread SCOPE=SPFILE SID='abc2'; -- $ srvctl stop database -d abc 繼續刪除第三個實例 $ srvctl start instance -d nocdd -instance nocdd1 $ dbca -silent -deleteInstance -nodeList d2 -gdbName nocdd -instanceName nocdd2
1.2刪除Oracle軟件
檢測交換分區,及產品軟件清單目錄 【節點2執行，刪除的節點】
oracle$$ORACLE_HOME/oui/bin/runInstaller -updateNodeList ORACLE_HOME=/u03/app/oracle/product/19.0.0/db_1 "CLUSTER_NODES={d2}" -local Starting Oracle Universal Installer...
Checking swap space: must be greater than 500 MB. Actual 3072 MB Passed
The inventory pointer is located at /etc/oraInst.loc
[oracle@d2:/home/oracle]$ cat /etc/oraInst.loc
inventory_loc=/u01/app/oraInventory
inst_group=oinstall
oracle$cd $ORACLE_HOME/deinstall
oracle$./deinstall -local
······
Specify the list of database names that are configured locally on this node for this Oracle home. Local
configurations of the discovered databases will be removed []: 回車，默認讀取當前oracle_home目錄 ······
The cluster node(s) on which the Oracle home deinstallation will be performed are:d2
Oracle Home selected for deinstall is: /u03/app/oracle/product/19.0.0/db_1
Inventory Location where the Oracle home registered is: /u01/app/oraInventory
Do you want to continue (y - yes, n - no)? [n]: y 繼續,yes ······
Successfully detached Oracle home '/u03/app/oracle/product/19.0.0/db_1' from the central inventory on the local node.
Successfully deleted directory '/u03/app/oracle/product/19.0.0/db_1' on the local node.
Successfully deleted directory '/u03/app/oracle' on the local node.
Oracle Universal Installer cleanup was successful.
Oracle deinstall tool successfully cleaned up temporary directories.
更新inventory # 在保留任一節點執行命令 su - oracle cd $ORACLE_HOME/oui/bin ./runInstaller -updateNodeList ORACLE_HOME=/u03/app/oracle/product/19.0.0/db_1 "CLUSTER_NODES={d1}"
1.3刪除Grid軟件
檢查集群節點是否被鎖定,只需要在刪除的節點操作即可，是檢測是否被鎖定
[grid@d1:/home/grid]$ olsnodes -s -t d1 Active Unpinned
[grid@d2:/home/grid]$ olsnodes -s -t PRCO-19: Failure retrieving list of nodes in the cluster
PRCO-2: Unable to communicate with the clusterware
# 如果節點被鎖定，運行crsctl unpin css命令后繼續下一步
crsctl unpin css -n delete_node
在不刪除二進制文件的情況下取消配置Oracle Clusterware https://docs.oracle.com/en/database/oracle/oracle-database/19/cwlin/unconfiguring-oracle-clusterware-without-removing-binaries.html#
GUID-41C5E8EE-68F9-4849-B4B8-BD82D4A17A57
在刪除的節點上,以root用戶執行
su - root
cd /u01/app/19.0.0/grid/crs/install
# /u01/app/19.0.0/grid/perl/bin/perl ./rootcrs.pl -deconfig -force
直接執行./rootcrs.pl提示./rootcrs.pl: line 88: =head1: command not found,perl無法執行腳本。
2020/04/15 17:03:56 CLSRSC-336: Successfully deconfigured Oracle Clusterware stack on this node
#若刪除整個集群，在最后一個節點執行
#./rootcrs.pl -deconfig -force -lastnode<br> 在刪除的節點上執行刪除軟件命令操作d2，只刪除本地節點2 su - grid cd $ORACLE_HOME/deinstall ./deinstall -local
ASM configuration was not detected in this Oracle home. Was ASM configured in this Oracle home (y|n) [n]: y
【asm 與grid安裝在一起共享一個oracle_home]
ASM Diagnostic Destination : /u01/app/grid
ASM Diskgroups :
ASM diskstring : /dev/*_1*
Diskgroups will not be dropped
If you want to retain the existing diskgroups or if any of the information detected is incorrect, you can modify by entering 'y'.
Do you want to modify above information (y|n) [n]: n 【不修改上述對磁盤組的操作,不刪除磁盤組] Oracle Grid Infrastructure Home is:
Oracle Home selected for deinstall is: /u01/app/19.0.0/grid
Inventory Location where the Oracle home registered is: /u01/app/oraInventory
Following Oracle Restart enabled listener(s) will be de-configured: ASMNET1LSNR_ASM,LISTENER,LISTENER_SCAN1
ASM instance will be de-configured from this Oracle home
Do you want to continue (y - yes, n - no)? [n]: y [確認清理上述資源】
####################### DEINSTALL CLEAN OPERATION SUMMARY #######################
Successfully detached Oracle home '/u01/app/19.0.0/grid' from the central inventory on the local node.
Failed to delete directory '/u01/app/19.0.0/grid' on the local node due to error : Either user has no permission
to delete or file is in use.
Review the permissions and manually delete '/u01/app/19.0.0/grid' on local node.
Failed to delete directory '/u01/app/oraInventory' on the local node due to error : Either user has no permission
to delete or file is in use.
Review the permissions and manually delete '/u01/app/oraInventory' on local node.
Oracle Universal Installer cleanup completed with errors.
Run 'rm -r /etc/oraInst.loc' as root on node(s) 'd2' at the end of the session. Run 'rm -r /opt/ORCLfmap' as root on node(s) 'd2' at the end of the session. Run 'rm -r /etc/oratab' as root on node(s) 'd2' at the end of the session. Review the permissions and contents of '/u01/app/grid' on nodes(s) 'd2'.
If there are no Oracle home(s) associated with '/u01/app/grid', manually delete '/u01/app/grid' and its contents.
Oracle deinstall tool successfully cleaned up temporary directories.
root# 刪除上述目錄! 四個目錄均遞歸刪除！
刪除CRS配置【存活的節點執行,已刪除的crs都沒了】 su - root cd /u01/app/19.0.0/grid/bin ./crsctl delete node -n d2
[grid@d1:/u01/app/19.0.0/grid/crs/install]$ cluvfy stage -post nodedel -n 2 -verbose
Verifying Node Removal ...
Verifying CRS Integrity ...PASSED
Verifying Clusterware Version Consistency ...PASSED
Verifying Node Removal ...PASSED
Post-check for node removal was successful.
CVU operation performed: stage -post nodedel
Date: Apr 15, 2020 2:28:00 AM
CVU home: /u01/app/19.0.0/grid/
User: grid
發現存在一個vip 未被刪除FAILED OVER,STABLE # /u01/app/19.0.0/grid/bin/srvctl remove vip -vip d2 -force
二、添加節點
2.1添加Grid集群節點
前期准備工作，需要與安裝RAC工作一樣，本次環境都是好的【新增節點不需要再需要安裝介質】
$ mkdir -p /u01/app/19.0.0/grid $ ssh d1 date;ssh d2 date; ssh d1-priv date;ssh d2-priv date; 運行CVU程序 su - grid cd ${ORACLE_HOME}/bin cluvfy comp peer [-refnode <new node name>] -n node_list [-orainv orainventory_group] [-osdba osdba_group] [-verbose] CVU用作檢查與其他節點的兼容性的參考的節點。如果未指定此選項，則CVU報告節點列表中所有節點的值。 grid_node1$cluvfy comp peer -refnode d1 -n d2 -verbose 加節點（GI） grid_node1$/u01/app/19.0.0/grid/addnode/addnode.sh -silent -ignoreSysPrereqs -ignorePrereqFailure "CLUSTER_NEW_NODES={d2}"
"CLUSTER_NEW_PRIVATE_NODE_NAMES={d2-priv}" "CLUSTER_NEW_VIRTUAL_HOSTNAMES={d2-vip}" As a root user, execute the following script(s): 1. /u01/app/oraInventory/orainstRoot.sh 2. /u01/app/19.0.0/grid/root.sh Execute /u01/app/oraInventory/orainstRoot.sh on the following nodes: [d2] Execute /u01/app/19.0.0/grid/root.sh on the following nodes: [d2]
······
[root@d1 ~]# /u01/app/oraInventory/orainstRoot.sh Changing permissions of /u01/app/oraInventory. Adding read,write permissions for group. Removing read,write,execute permissions for world. Changing groupname of /u01/app/oraInventory to oinstall. The execution of the script is complete.
驗證
$ cluvfy stage -post nodeadd -n d2
2.2添加Oracle軟件
加節點（DB) oracle_node1$/u03/app/oracle/product/19.0.0/db_1/addnode/addnode.sh -silent -ignoreSysPrereqs -ignorePrereqFailure "CLUSTER_NEW_NODES={d2}" As a root user, execute the following script(s): 1. /u03/app/oracle/product/19.0.0/db_1/root.sh Execute /u03/app/oracle/product/19.0.0/db_1/root.sh on the following nodes: [d2]
2.2添加DB
-- Start the Instance on the New Node The prerequisite steps to take prior to adding an instance using DBCA are as follows:. Ensure that all the existing instances have been started using an spfile from the shared location. Verify that all the instances are listening on the default grid listener. The following command displays all the existing instances: $ $CRS_HOME/bin/lsnrctl status Run $ORACLE_HOME/bin/dbca from any of the existing RAC instance Oracle homes. Select "Oracle Real Application Clusters" database and click Next. Select "Instance Management" and click Next. Select "Add an Instance" and click Next. Click Next to accept the default instance name (or change it, if necessary). Check the summary window.
測試環境3套DB,啟動單個實例后，靜默方式添加節點2實例 $ srvctl start instance -d abc -instance abc1 $dbca -ignorePreReqs -ignorePrereqFailure -silent -addInstance -nodeName d2 -gdbName abc -instanceName abc2 -sysDBAUserName sys
-sysDBAPassword oracle $ srvctl start database -d nocdd $dbca -ignorePreReqs -ignorePrereqFailure -silent -addInstance -nodeName d2 -gdbName nocdd -instanceName nocdd2 -sysDBAUserName sys
-sysDBAPassword oracle $ srvctl start database -d orcl $dbca -ignorePreReqs -ignorePrereqFailure -silent -addInstance -nodeName d2 -gdbName orcl -instanceName orcl2 -sysDBAUserName sys
-sysDBAPassword oracle
DB_ALERT
2020-04-16T00:50:34.958060+08:00
CREATE SMALLFILE UNDO TABLESPACE "UNDOTBS2" DATAFILE SIZE 290M AUTOEXTEND ON NEXT 5120K MAXSIZE 32767M BLOCKSIZE 8192
2020-04-16T00:50:36.603592+08:00
Completed: CREATE SMALLFILE UNDO TABLESPACE "UNDOTBS2" DATAFILE SIZE 290M AUTOEXTEND ON NEXT 5120K MAXSIZE 32767M BLOCKSIZE 8192
ALTER DATABASE ADD LOGFILE THREAD 2 GROUP 3 SIZE 200M,
GROUP 4 SIZE 200M
2020-04-16T00:50:39.079100+08:00
Completed: ALTER DATABASE ADD LOGFILE THREAD 2 GROUP 3 SIZE 200M,
GROUP 4 SIZE 200M
ALTER DATABASE ENABLE PUBLIC THREAD 2
Completed: ALTER DATABASE ENABLE PUBLIC THREAD 2
2020-04-16T00:50:39.185364+08:00
ALTER SYSTEM SET instance_number=2 SCOPE=SPFILE SID='nocdd2';
2020-04-16T00:50:39.213198+08:00
ALTER SYSTEM SET thread=2 SCOPE=SPFILE SID='nocdd2';
2020-04-16T00:50:39.243485+08:00
ALTER SYSTEM SET undo_tablespace='UNDOTBS2' SCOPE=SPFILE SID='nocdd2';
2020-04-16T00:51:29.154327+08:00
補丁信息發現,添加節點補丁自動打上了！！！ [grid@d1:/u01/app/19.0.0/grid/OPatch]$ ./opatch lsinventory oracle ······ SQL> select ACTION,NAMESPACE,VERSION,BUNDLE_SERIES,COMMENTS FROM dba_registry_history;
ACTION NAMESPACE VERSION BUND COMMENTS
---------- ---------- -------------------- ---- ----------------------------------------
BOOTSTRAP DATAPATCH 19 RDBMS_19.5.0.0.0DBRUR_LINUX.X64_191213
RU_APPLY SERVER 19.0.0.0.0 Patch applied on 19.3.0.0.0: Release_Update - 190410122720