ORA-16401: archivelog rejected by RFS
無線出單系統郵件告警10.111.20.1
1. 報錯
SYS > ! oerr ora 16041
16041, 00000, "Remote File Server fatal error"
// *Cause: The Remote File Server (RFS) process at a redo transport
// destination encountered a fatal error and was no longer
// available. Further redo transmission to this destination may not be
// possible.
// *Action: Refer to the appropriate RFS trace file located at the redo
// transport destination for details regarding the error encountered
// and take corrective action.
遠程傳輸錯誤，RFS服務器，與參數fal_server,fal_client參數有關
2. 搜索百度
通常此問題是由於主庫的fal_client參數與備庫的log_archive_dest_n中的service參數不匹配造成的。
3. 查詢參數
Primary
--查詢DG成員
SQL> show parameter config
log_archive_config rac,dgwless,wxcdrac
--減少自己本身，可能存在兩個DG成員
--查詢主庫db_unique_name
SQL> show parameter db_unique_name
db_unique_name wxcdrac
--查詢主庫歸檔參數
SQL> show parameter archive
NAME VALUE
---------------------------------------------------
log_archive_dest_2 service=dgwless LGWR ASYNC va
lid_for=(online_logfiles,prima
ry_role) db_unique_name=dgwless
log_archive_dest_3 SERVICE=RAC LGWR ASYNC VALID_F
OR=(ONLINE_LOGFILES,PRIMARY_RO
LE) db_unique_name=rac
--查詢歸檔線程參數狀態，以上歸檔線程3對應的參數可以過濾，因為此DG可能已被廢棄
--除非是人為或者誤操作置為defer，否則基本判斷問題，排除此DG
log_archive_dest_state_2 ENABLE
log_archive_dest_state_3 DEFER
--查詢主庫fal參數
SQL> show parameter fal
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
fal_client string wxcdrac2
fal_server string dgwless
--通過主庫參數，查詢備庫IP地址
SQL> host tnsping dgwless
Attempting to contact (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = 10.135.101.23)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = dgwless)))
OK (90 msec)
--連接備庫后，查詢驗證主備關系
SQL> select dbid,name, database_role,open_mode,FLASHBACK_ON from v$database;
DBID NAME DATABASE_ROL OPEN_MODE FLASHBACK_ON
----------- --------------- ------------ ---------- ------------------
2283108100 RAC PRIMARY READ WRITE NO
SQL> select dbid,name, database_role,open_mode,FLASHBACK_ON from v$database;
DBID NAME DATABASE_ROL OPEN_MODE FLASHBACK_ON
----------- --------------- ------------ ---------- ------------------
2283108100 RAC PHYSICAL STANDBY MOUNTED NO
--standby
SQL> show parameter fal
NAME VALUE
------------- ------------------------------
fal_client dgwless
fal_server wxcdrac --發現與主庫fal_client參數不同
--查詢備庫歸檔參數
log_archive_dest_1 location=/picclife/app/oracle/archdg/
log_archive_dest_2 SERVICE=WXCDRAC LGWR ASYNC VAL --發現tns也與主庫fal_client不同
ID_FOR=(ONLINE_LOGFILES,PRIMAR
Y_ROLE) db_unique_name=wxcdrac
--主庫fal_client 在11g已被棄用，按照道理將主庫fal_client與備庫log_archvie_service不同，無異常，查詢版本，
當前10G
Enterprise Edition Release 10.2.0.3.0
--查詢主庫集群狀態
wxcdrac2:/home/oracle$ export ORACLE_SID=+ASM1
wxcdrac2:/home/oracle$ cat .profile
export ORACLE_HOME=$ORACLE_BASE/product/10.2.0/db_1
export ORA_CRS_HOME=$ORACLE_BASE/product/10.2.0/crs_1
wxcdrac2:/home/oracle$ cd $ORA_CRS_HOME/bin
wxcdrac2:/picclife/app/oracle/product/10.2.0/crs_1/bin$
wxcdrac2:/picclife/app/oracle/product/10.2.0/crs_1/bin$ crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora.wxcdrac.db application ONLINE ONLINE wxcdrac1
ora....c1.inst application ONLINE ONLINE wxcdrac1
ora....c2.inst application ONLINE ONLINE wxcdrac2
ora....SM1.asm application ONLINE ONLINE wxcdrac1
ora....C1.lsnr application ONLINE ONLINE wxcdrac1
ora....ac1.gsd application ONLINE ONLINE wxcdrac1
ora....ac1.ons application ONLINE ONLINE wxcdrac1
ora....ac1.vip application ONLINE ONLINE wxcdrac1
ora....SM2.asm application ONLINE ONLINE wxcdrac2
ora....C2.lsnr application ONLINE ONLINE wxcdrac2
ora....ac2.gsd application ONLINE ONLINE wxcdrac2
ora....ac2.ons application ONLINE ONLINE wxcdrac2
ora....ac2.vip application ONLINE ONLINE wxcdrac2
4. 查詢日志
--備庫日志
Wed Aug 8 01:59:19 2018
Redo Shipping Client Connected as PUBLIC
-- Connected User is Valid
RFS[2586]: Assigned to RFS process 29818962
RFS[2586]: Identified database type as 'physical standby'
Primary database is in MAXIMUM PERFORMANCE mode
Primary database is in MAXIMUM PERFORMANCE mode
RFS[2586]: Successfully opened standby log 15: '/picclife/app/oracle/oradata/dgwless/std02_1.log'
Wed Aug 8 02:04:42 2018
RFS[2579]: Archivelog thread 2 sequence 3661 cannot be reused
Wed Aug 8 02:04:42 2018
Errors in file /picclife/app/oracle/admin/dgwless/udump/dgwless_rfs_13238468.trc:
ORA-16401: archivelog rejected by RFS
Redo Shipping Client Connected as PUBLIC
-- Connected User is Valid
--查詢trace文件
KCRRSNPS
Identified standby redo log 12 for implicit mid-log reconnect
*** 2018-08-08 02:04:42.324
*** 2018-08-08 02:04:42.324 61287 kcrr.c
RFS[2579]: Archivelog thread 2 sequence 3661 cannot be reused
This problem can occur as the result of multiple primary and/or
standby database(s) trying to archive to this standby database.
This is not allowed, since successfully completed archivelogs
would be overwritten.
ORA-16401: archivelog rejected by RFS
*** 2018-08-08 02:13:43.562
Dispatching RFS shutdown notification
以上備庫查詢得到的信息是，歸檔接受出現網絡問題，但不明確
查詢主庫Alert日志
--查詢郵件告警所在的數據庫Alert日志
SQL> show parameter background_dump_dest
NAME VALUE
--------------------- ------------------------------
background_dump_dest /picclife/app/oracle/admin/wxcdrac/bdump
Wed Aug 8 01:02:57 2018
LNS: Standby redo logfile selected for thread 1 sequence 3993 for destination LOG_ARCHIVE_DEST_2
Wed Aug 8 01:03:32 2018
Starting control autobackup
Control autobackup written to SBT_TAPE device
comment 'API Version 2.0,MMS Version 10.0.0.116',
media 'V_89405_737884'
handle 'c-2283108100-20180808-01'
Wed Aug 8 01:08:19 2018
Errors in file /picclife/app/oracle/admin/wxcdrac/bdump/wxcdrac1_arc2_10682458.trc:
ORA-16401: archivelog rejected by RFS
Wed Aug 8 01:11:58 2018
ARC0: Attempting destination LOG_ARCHIVE_DEST_2 network reconnect (3135)
ARC0: Destination LOG_ARCHIVE_DEST_2 network reconnect abandoned
Wed Aug 8 01:11:58 2018
Errors in file /picclife/app/oracle/admin/wxcdrac/bdump/wxcdrac1_arc0_11337924.trc:
ORA-03135: connection lost contact
FAL[server, ARC0]: Error 3135 creating remote archivelog file 'dgwless'
FAL[server, ARC0]: FAL archive failed, see trace file.
Wed Aug 8 01:11:58 2018
Errors in file /picclife/app/oracle/admin/wxcdrac/bdump/wxcdrac1_arc0_11337924.trc:
ORA-16055: FAL request rejected
ARCH: FAL archive failed. Archiver continuing
Wed Aug 8 01:11:58 2018
ORACLE Instance wxcdrac1 - Archival Error. Archiver continuing.
*** 2018-08-08 01:08:19.693
Error 16401 creating standby archive log file at host 'dgwless'
ORA-16401: archivelog rejected by RFS
--從以上的信息還是未明確發現由於參數問題導致的報錯
5. 搜索MOS
1183143.1
ORA-16401: ARCHIVELOG REJECTED BY RFS error occurs when incorrect naming convention for fal_client parameter is set.
Version 10.2.0.4 to 11.1.0.7 [Release 10.2 to 11.1]
配置主備不一致的fal_client時會發生此錯誤
***** Primary *****
log_archive_config DG_CONFIG=(p1ncm1,p1ncm2)
log_archive_dest_1 location=/u99/ORACLE/p1ncm1/archive valid_for=(ALL_LOGFILES,ALL_ROLES) db_unique_name=p1ncm1
log_archive_dest_2 service=p1ncm2_dgfal.csm.fub.com optional reopen=60 valid_for=(ONLINE_LOGFILES,PRIMARY_ROLE) db_unique_name=p1ncm2
fal_client p1ncm1_fc.csm.fub.com
fal_server p1ncm2_fs.csm.fub.com
********* Standby ********
log_archive_config DG_CONFIG=(p1ncm1,p1ncm2)
log_archive_dest_1 location=/u99/ORACLE/p1ncm2/archive valid_for=(ALL_LOGFILES,ALL_ROLES) db_unique_name=p1ncm2
log_archive_dest_2 service=p1ncm1_dgfal.csm.fub.com optional reopen=60 valid_for=(ONLINE_LOGFILES,PRIMARY_ROLE) db_unique_name=p1ncm1
fal_client p1ncm2_fc.csm.fub.com
fal_server p1ncm1_fs.csm.fub.com
SOLUTION
standby
alter system set fal_client=p1ncm2_dgfal.csm.fub.com scope=both sid='*';
6. 處理
-修改主庫的fal_client參數，與備庫保持一致
--修改前需要進行驗證
SQL> host tnsping wxcdrac2
SQL> host tnsping wxcdrac
SQL> alter system set fal_client=wxcdrac;
--第二次，在其它客戶遇到同樣報錯，問題是tnsping 主庫fal_client不通，端口錯誤。
7. 補充說明
MOS ID 288074.1
Versions 11.2, 12.1 對於11g12c而言，很可能此消息是一個提示作用
Error: ORA-16401 archive log rejected by Remote File Server (RFS)
---------------------------------------------------------------------------
Cause: An attempt was made to re-archive an existing archive log.
Action: See alert log and trace file for more details. No action is necessary.
This is an informational statement provided to record the event for
diagnostic purposes.
Versions 9.2, 10.1, 10.2, 11.1
Error: ORA-16401 archivelog rejected by RFS 對於10g而言，此報錯，很可能為fal_client參數配置錯誤
---------------------------------------------------------------------------
Cause: An attempt was made to re-archive an existing archivelog. This usually
happens because either a multiple primary database or standby
database(s) or both are trying to archive to this standby database.
Action: See alert log and trace file for more details. No action is necessary;
this is an informational statement provided to record the event for
diagnostic purposes.