節后第一天，開發說一個開發庫（11g rac+dg ）不能登錄了。報錯
ORA-09925: Unable to create audit trail file
Linux-x86_64 Error: 28: No space left on device
Additional information: 9925
ORA-01075: you are currently logged on
[oracle@rac01 ~]$ sqlplus / as sysdba SQL*Plus: Release 11.2.0.4.0 Production on Mon Oct 12 09:38:19 2020 Copyright (c) 1982, 2013, Oracle. All rights reserved. ERROR: ORA-09925: Unable to create audit trail file Linux-x86_64 Error: 28: No space left on device Additional information: 9925 ORA-01075: you are currently logged on
初步懷疑是歸檔滿了，空間不夠，（開發庫的原因，沒有處理）
用rman登錄，報錯
[oracle@rac01 ~]$ rman target / Recovery Manager: Release 11.2.0.4.0 - Production on Mon Oct 12 09:40:46 2020 Copyright (c) 1982, 2011, Oracle and/or its affiliates. All rights reserved. RMAN-00571: =========================================================== RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS =============== RMAN-00571: =========================================================== RMAN-00554: initialization of internal recovery manager package failed RMAN-04005: error from target database: ORA-09925: Unable to create audit trail file Linux-x86_64 Error: 28: No space left on device Additional information: 9925
這里都提示空間不足：
發生上面錯誤的原因 有以下三種可能： 1磁盤空間已滿，導致無法創建審核文件，使用df -h查詢磁盤空間； 2磁盤inodes已經占用完了，導致無法打開新的文件，使用df -i查看inode使用情況； 3路徑權限不足，無法創建審核文件，查看路徑權限情況ls -l
----查看磁盤 [root@rac02 bin]# ./crsctl start cluster -all CRS-4639: Could not contact Oracle High Availability Services CRS-4000: Command Start failed, or completed with errors. [root@rac02 bin]# ll /dev/asm-disk* brw-rw---- 1 grid asmadmin 8, 16 Oct 12 09:57 /dev/asm-diskb brw-rw---- 1 grid asmadmin 8, 32 Oct 12 09:51 /dev/asm-diskc brw-rw---- 1 grid asmadmin 8, 48 Oct 12 09:46 /dev/asm-diskd brw-rw---- 1 grid asmadmin 8, 64 Oct 12 09:57 /dev/asm-diske brw-rw---- 1 grid asmadmin 8, 80 Oct 12 09:57 /dev/asm-diskf brw-rw---- 1 grid asmadmin 8, 96 Oct 12 09:52 /dev/asm-diskg brw-rw---- 1 grid asmadmin 8, 112 Oct 12 09:51 /dev/asm-diskh [root@rac02 bin]# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sr0 11:0 1 1024M 0 rom sda 8:0 0 80G 0 disk ├─sda1 8:1 0 500M 0 part /boot └─sda2 8:2 0 79.5G 0 part ├─VolGroup-lv_root (dm-0) 253:0 0 50G 0 lvm / ├─VolGroup-lv_swap (dm-1) 253:1 0 8G 0 lvm [SWAP] └─VolGroup-lv_home (dm-2) 253:2 0 21.5G 0 lvm /home sdc 8:32 0 40G 0 disk sdf 8:80 0 5G 0 disk sdg 8:96 0 5G 0 disk sdb 8:16 0 40G 0 disk sdd 8:48 0 40G 0 disk sde 8:64 0 50G 0 disk sdh 8:112 0 5G 0 disk [root@rac02 bin]# ./crs_stat -t -v CRS-0184: Cannot communicate with the CRS daemon. This command is deprecated and has been replaced by 'crsctl status resource' This command remains for backward compatibility only [root@rac02 bin]# find / -name crsd.log /u01/app/11.2.0/grid/log/rac02/crsd/crsd.log [root@rac02 bin]# tail -n 100 /u01/app/11.2.0/grid/log/rac02/crsd/crsd.log 2020-10-12 10:06:28.576: [ CRSMAIN][677160736] Checking the OCR device 2020-10-12 10:06:28.577: [ CRSMAIN][677160736] Sync-up with OCR 2020-10-12 10:06:28.577: [ CRSMAIN][677160736] Connecting to the CSS Daemon 2020-10-12 10:06:28.577: [ CRSMAIN][677160736] Getting local node number 2020-10-12 10:06:28.577: [ CRSMAIN][670709504] Policy Engine is not initialized yet! 2020-10-12 10:06:28.578: [ CRSMAIN][677160736] Initializing OCR [ CLWAL][677160736]clsw_Initialize: OLR initlevel [70000] 2020-10-12 10:06:34.048: [ OCRASM][677160736]proprasmo: Error in open/create file in dg [OCR] [ OCRASM][677160736]SLOS : SLOS: cat=7, opn=kgfoAl06, dep=9925, loc=kgfokge 2020-10-12 10:06:34.048: [ OCRASM][677160736]ASM Error Stack : ORA-09925: Unable to create audit trail file Linux-x86_64 Error: 28: No space left on device Additional information: 9925
使用 df -h 查看 空間正常
查看asm磁盤是否正常掛載
[root@rac02 ~]# su - grid [grid@rac02 ~]$ sqlplus / as sysasm SQL*Plus: Release 11.2.0.4.0 Production on Mon Oct 12 10:08:44 2020 Copyright (c) 1982, 2013, Oracle. All rights reserved. ERROR: ORA-09925: Unable to create audit trail file Linux-x86_64 Error: 28: No space left on device Additional information: 9925 ORA-09925: Unable to create audit trail file Linux-x86_64 Error: 28: No space left on device Additional information: 9925
有點奇怪。。於是google了一下。發現一篇相似
https://blog.csdn.net/HLandHB/article/details/104705626/
檢查
[root@rac02 bin]# df -i Filesystem Inodes IUsed IFree IUse% Mounted on /dev/mapper/VolGroup-lv_root 3276800 3276800 0 100% / tmpfs 4110420 276 4110144 1% /dev/shm /dev/sda1 128016 39 127977 1% /boot /dev/mapper/VolGroup-lv_home 1411680 144 1411536 1% /home
居然是的，磁盤的inode使用完了。
解決：
cd /u01/app/oracle/admin/orcl/adump/ ls | xargs -n 10 rm -fr ls
刪除這個目錄的多個文件。正常情況下是不會有很多個文件的。查看該文件
[oracle@rac01 adump]$ more ORCL1_ora_7401_20201012114037123698143795.aud Audit file /u01/app/oracle/admin/orcl/adump/ORCL1_ora_7401_20201012114037123698143795.aud Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP, Data Mining and Real Application Testing options ORACLE_HOME = /u01/app/oracle/product/11.2.0/db_1 System name: Linux Node name: rac01 Release: 2.6.32-754.el6.x86_64 Version: #1 SMP Tue Jun 19 21:26:04 UTC 2018 Machine: x86_64 VM name: VMWare Version: 6 Instance name: ORCL1 Redo thread mounted by this instance: 1 Oracle process number: 42 Unix process pid: 7401, image: oracle@rac01 (TNS V1-V3) Mon Oct 12 11:40:37 2020 +08:00 LENGTH : '160' ACTION :[7] 'CONNECT' DATABASE USER:[1] '/' PRIVILEGE :[6] 'SYSDBA' CLIENT USER:[6] 'oracle' CLIENT TERMINAL:[5] 'pts/0' STATUS:[1] '0' DBID:[10] '1640682678'
才想起，之前這個庫為了查一個東西，開啟了errorstack
SQL> show parameter aud NAME TYPE VALUE ------------------------------------ ----------- ------------------------------ audit_file_dest string /u01/app/oracle/admin/ORCL/adum p audit_sys_operations boolean FALSE audit_syslog_level string audit_trail string DB
關閉errorstack
conn / as sysdba
alter system set events '1438 trace name errorstack off';
貼一下alert log
[oracle@rac01 trace]$ tail -n 100 alert_ORCL1.log ORA-16014: log 1 sequence# 1289 not archived, no available destinations ORA-00312: online log 1 thread 1: '+DATA/orcl/onlinelog/group_1.269.1032629563' ORA-00312: online log 1 thread 1: '+FRA/orcl/onlinelog/group_1.262.1032629563' Mon Oct 12 09:26:48 2020 DIA0 detected that there is an archiving problem on the cluster. Several processes including LGWR and at least one ARC process are hung. Instance termination is not allowed. External intervention is required. DIA0 detected that there is an archiving problem on the cluster. Several processes including LGWR and at least one ARC process are hung. Instance termination is not allowed. External intervention is required. Non critical error ORA-48180 caught while writing to trace file "/u01/app/oracle/diag/rdbms/cad/ORCL1/trace/ORCL1_dia0_13133_1923.trc" Error message: Linux-x86_64 Error: 28: No space left on device Additional information: 1 Writing to the above trace file is disabled for now on... Mon Oct 12 09:27:52 2020 Suspending MMON action 'undo usage' for 82800 seconds Mon Oct 12 09:29:19 2020 ARCH: Archival stopped, error occurred. Will continue retrying ORACLE Instance ORCL1 - Archival Error ORA-16038: log 1 sequence# 1289 cannot be archived ORA-19504: failed to create file "" ORA-00312: online log 1 thread 1: '+DATA/orcl/onlinelog/group_1.269.1032629563' ORA-00312: online log 1 thread 1: '+FRA/orcl/onlinelog/group_1.262.1032629563' Mon Oct 12 09:29:19 2020 ARCH: Archival stopped, error occurred. Will continue retrying ORACLE Instance ORCL1 - Archival Error ORA-16014: log 1 sequence# 1289 not archived, no available destinations ORA-00312: online log 1 thread 1: '+DATA/orcl/onlinelog/group_1.269.1032629563' ORA-00312: online log 1 thread 1: '+FRA/orcl/onlinelog/group_1.262.1032629563' Mon Oct 12 09:32:58 2020 Some DDE async actions failed or were cancelled Non critical error ORA-48180 caught while writing to trace file "/u01/app/oracle/diag/rdbms/orcl/ORCL1/trace/ORCL1_m002_26767.trc" Error message: Linux-x86_64 Error: 28: No space left on device Additional information: 1 Writing to the above trace file is disabled for now on... Mon Oct 12 09:34:19 2020 ARCH: Archival stopped, error occurred. Will continue retrying ORACLE Instance ORCL1 - Archival Error ORA-16038: log 1 sequence# 1289 cannot be archived ORA-19504: failed to create file "" ORA-00312: online log 1 thread 1: '+DATA/orcl/onlinelog/group_1.269.1032629563' ORA-00312: online log 1 thread 1: '+FRA/orcl/onlinelog/group_1.262.1032629563' ARCH: Archival stopped, error occurred. Will continue retrying ORACLE Instance ORCL1 - Archival Error ORA-16014: log 1 sequence# 1289 not archived, no available destinations ORA-00312: online log 1 thread 1: '+DATA/orcl/onlinelog/group_1.269.1032629563' ORA-00312: online log 1 thread 1: '+FRA/orcl/onlinelog/group_1.262.1032629563' Mon Oct 12 09:35:00 2020 *********************************************************************** Fatal NI connect error 12170. VERSION INFORMATION: TNS for Linux: Version 11.2.0.4.0 - Production Oracle Bequeath NT Protocol Adapter for Linux: Version 11.2.0.4.0 - Production TCP/IP NT Protocol Adapter for Linux: Version 11.2.0.4.0 - Production Time: 12-OCT-2020 09:35:00 Tracing not turned on. Tns error struct: ns main err code: 12535 TNS-12535: TNS:operation timed out ns secondary err code: 12606 nt main err code: 0 nt secondary err code: 0 nt OS err code: 0 Client address: (ADDRESS=(PROTOCOL=tcp)(HOST=10.15.4.40)(PORT=2884)) WARNING: inbound connection timed out (ORA-3136) Mon Oct 12 09:38:24 2020 OS Audit file could not be created; failing after 6 retries Mon Oct 12 09:38:26 2020 Non critical error ORA-48180 caught while writing to trace file "/u01/app/oracle/diag/rdbms/orcl/ORCL1/trace/ORCL1_m002_27284.trc" Error message: Linux-x86_64 Error: 28: No space left on device Additional information: 1 Writing to the above trace file is disabled for now on... OS Audit file could not be created; failing after 6 retries Mon Oct 12 09:39:49 2020 ARCH: Archival stopped, error occurred. Will continue retrying ORACLE Instance ORCL1 - Archival Error ORA-16038: log 1 sequence# 1289 cannot be archived ORA-19504: failed to create file "" ORA-00312: online log 1 thread 1: '+DATA/orcl/onlinelog/group_1.269.1032629563' ORA-00312: online log 1 thread 1: '+FRA/orcl/onlinelog/group_1.262.1032629563' Mon Oct 12 09:39:49 2020 Non critical error ORA-48180 caught while writing to trace file "/u01/app/oracle/diag/rdbms/orcl/CAD1/trace/CAD1_m003_27424.trc" Error message: Linux-x86_64 Error: 28: No space left on device Additional information: 1 Writing to the above trace file is disabled for now on... Mon Oct 12 09:39:49 2020 ARCH: Archival stopped, error occurred. Will continue retrying ORACLE Instance ORCL1 - Archival Error ORA-16014: log 1 sequence# 1289 not archived, no available destinations ORA-00312: online log 1 thread 1: '+DATA/orcl/onlinelog/group_1.269.1032629563' ORA-00312: online log 1 thread 1: '+FRA/orcl/onlinelog/group_1.262.1032629563' Mon Oct 12 09:40:52 2020 OS Audit file could not be created; failing after 6 retries