今天上午有同事反映應用數據庫連接不上；於是排查數據庫：
[oracle@db ~]$ sqlplus / as sysdba SQL*Plus: Release 10.2.0.4.0 - Production on Thu Dec 12 08:54:39 2013 Copyright (c) 1982, 2007, Oracle. All Rights Reserved. ERROR: ORA-09925: Unable to create audit trail file Linux-x86_64 Error: 30: Read-only file system Additional information: 9925 ORA-09925: Unable to create audit trail file Linux-x86_64 Error: 30: Read-only file system Additional information: 9925 Enter user-name: ERROR: ORA-01017: invalid username/password; logon denied Enter user-name: ERROR: ORA-01017: invalid username/password; logon denied SP2-0157: unable to CONNECT to ORACLE after 3 attempts, exiting SQL*Plus
上網查了，說數據庫命令權限不對，解決辦法如下：
-rwx------ 1 oracle oinstall 3337 Jan 15 2013 onsctl -rwxr-xr-x 1 oracle oinstall 46 Nov 7 2000 oracg -rwsr-s--x 1 oracle oinstall 112468376 Jan 15 2013 oracle -rwxr-x--- 1 oracle oinstall 0 Mar 12 2008 oracleO -r-sr-s--- 1 root oinstall 14931 Mar 11 2008 oradism -rwxr-x--- 1 oracle oinstall 0 Mar 12 2008 oradismO
[oracle@db bin]$ chmod 6755 /u01/app/oracle/product/10.2.0/bin/oracle chmod: changing permissions of `/u01/app/oracle/product/10.2.0/bin/oracle': Read-only file system [oracle@db bin]$
改變權限，但提示read-only file system。
再排查系統：
[root@db ~]# mount /dev/mapper/LVMgroup-root on / type ext3 (rw) proc on /proc type proc (rw) sysfs on /sys type sysfs (rw) devpts on /dev/pts type devpts (rw,gid=5,mode=620) /dev/sda1 on /boot type ext3 (rw) tmpfs on /dev/shm type tmpfs (rw) none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw) mount: warning /etc/mtab is not writable (e.g. read-only filesystem). It's possible that information reported by mount(8) is not up to date. For actual information about system mount points check the /proc/mounts file. [root@db ~]# cat /proc/mounts rootfs / rootfs rw 0 0 /dev/root / ext3 ro,data=ordered 0 0 /dev /dev tmpfs rw 0 0 /proc /proc proc rw 0 0 /sys /sys sysfs rw 0 0 /proc/bus/usb /proc/bus/usb usbfs rw 0 0 devpts /dev/pts devpts rw 0 0 /dev/sda1 /boot ext3 rw,data=ordered 0 0 tmpfs /dev/shm tmpfs rw 0 0 tmpfs /dev/shm tmpfs rw 0 0 none /proc/sys/fs/binfmt_misc binfmt_misc rw 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw 0 0 /etc/auto.misc /misc autofs rw,fd=7,pgrp=3377,timeout=300,minproto=5,maxproto=5,indirect 0 0 -hosts /net autofs rw,fd=13,pgrp=3377,timeout=300,minproto=5,maxproto=5,indirect 0 0
root根目錄變成read-only，無法寫文件系統，Oracle因為無法寫audit文件也無法正常登陸 。
[root@db ~]# dmesg EXT3-fs error (device dm-0) in ext3_reserve_inode_write: Journal has aborted EXT3-fs error (device dm-0) in ext3_reserve_inode_write: Journal has aborted sd 0:0:0:0: timing out command, waited 1080s sd 0:0:0:0: SCSI error: return code = 0x06000008 end_request: I/O error, dev sda, sector 74923301 EXT3-fs error (device dm-0) in ext3_reserve_inode_write: Journal has aborted ************ *********** **********
通過dmesg發現EXT3 文件系統存在問題。
網上有資料說可以通過使用fsck命令修復文件系統，命令也執行不了（fsck可能會導致數據塊損壞，最好是不做fsck，重啟一下服務器）。
[root@db ~]# fsck fsck 1.39 (29-May-2006) e2fsck 1.39 (29-May-2006) /dev/LVMgroup/root: recovering journal fsck.ext3: Bad magic number in super-block while trying to re-open /dev/LVMgroup/root e2fsck: io manager magic bad!
[root@db ~]#
通過命令重啟服務器也不可行。
[root@db ~]# df -h -bash: df: command not found [root@db ~]# reboot -bash: reboot: command not found
到此，問題已經很明了，Linux系統出問題了，咨詢運維，服務器確實出問題。
幸好這是一個臨時環境，數據庫重新安裝即可。
如果是正式環境，麻煩大了。切記，數據庫要定期在其他系統上做備份！！