RAC數據庫的ORA-27123: Unable To Attach To Shared Memory Segment Linux-x86_64 Error: 22: Invalid argument

RAC數據庫的 ORA-27123: Unable To Attach To Shared Memory Segment Linux-x86_64 Error: 22: Invalid argument

RAC數據庫的場景

由於生產環境的某業務大量寫操作，導致出現了如下的錯誤:

Making DataFactory:com.raqsoft.report.dataset.SQLDataSetFactory failure (dataset named):ds1 Caused:ORA-01034: ORACLE not available ORA-27123: unable to attach to shared memory segment Linux-x86_64 Error: 22: Invalid argument Additional information: 25 Additional information: 1605653 Additional information: -1073741824

環境	版本信息
數據庫的版本	Release 11.2.0.4.0 Production
操作系統版本	2.6.32-696.el6.x86_64

查看環境的命令:

[root@newarpdb01 ~]# uname -r
2.6.32-696.el6.x86_64
[root@newarpdb01 ~]# grep memlock /etc/security/limits.conf
#        - memlock - max locked-in-memory address space (KB)
[root@newarpdb01 ~]# grep HugePsges /proc/meminfo
[root@newarpdb01 ~]# grep HugePages /proc/meminfo
AnonHugePages:  22540288 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
[root@newarpdb01 ~]# grep Hugepages /proc/meminfo
Hugepagesize:       2048 kB
[root@newarpdb01 ~]# sysctl -a |grep shm
kernel.shmmax = 536870912000
kernel.shmall = 131072000
kernel.shmmni = 4096
kernel.shm_rmid_forced = 0
vm.hugetlb_shm_group = 0
[root@newarpdb01 ~]# uname -r
2.6.32-696.el6.x86_64

基本定位在HugePages上面

系統進程是通過虛擬地址訪問內存，但是CPU必須把它轉換程物理內存地址才能真正訪問內存。為了提高這個轉換效率，CPU會緩存最近的虛擬內存地址和物理內存地址的映射關系，並保存在一個由CPU維護的映射表中。為了盡量提高內存的訪問速度，需要在映射表中保存盡量多的映射關系。

而在Linux中，內存都是以頁的形式划分的，默認情況下每頁是4K，這就意味着如果物理內存很大，則映射表的條目將會非常多，會影響CPU的檢索效率。因為內存大小是固定的，為了減少映射表的條目，可采取的辦法只有增加頁的尺寸。

page table是操作系統上的虛擬內存系統的數據結構模型，用於存儲虛擬地址與物理地址的對應關系。當我們訪問內存時，首先訪問page table，然后Linux在通過page table的mapping來訪問真實物理內存（ram+swap）

解決思路

如何計算nr_hugepages,可以使用如下的shell，hugepages_settings.sh

#!/bin/bash
#
# hugepages_settings.sh
#
# Linux bash script to compute values for the
# recommended HugePages/HugeTLB configuration
# on Oracle Linux
#
# Note: This script does calculation for all shared memory
# segments available when the script is run, no matter it
# is an Oracle RDBMS shared memory segment or not.
#
# This script is provided by Doc ID 401749.1 from My Oracle Support
# http://support.oracle.com

# Welcome text
echo "
This script is provided by Doc ID 401749.1 from My Oracle Support
(http://support.oracle.com) where it is intended to compute values for
the recommended HugePages/HugeTLB configuration for the current shared
memory segments on Oracle Linux. Before proceeding with the execution please note following:
 * For ASM instance, it needs to configure ASMM instead of AMM.
 * The 'pga_aggregate_target' is outside the SGA and
   you should accommodate this while calculating the overall size.
 * In case you changes the DB SGA size,
   as the new SGA will not fit in the previous HugePages configuration,
   it had better disable the whole HugePages,
   start the DB with new SGA size and run the script again.
And make sure that:
 * Oracle Database instance(s) are up and running
 * Oracle Database 11g Automatic Memory Management (AMM) is not setup
   (See Doc ID 749851.1)
 * The shared memory segments can be listed by command:
     # ipcs -m


Press Enter to proceed..."

read

# Check for the kernel version
KERN=`uname -r | awk -F. '{ printf("%d.%d\n",$1,$2); }'`

# Find out the HugePage size
HPG_SZ=`grep Hugepagesize /proc/meminfo | awk '{print $2}'`
if [ -z "$HPG_SZ" ];then
    echo "The hugepages may not be supported in the system where the script is being executed."
    exit 1
fi

# Initialize the counter
NUM_PG=0

# Cumulative number of pages required to handle the running shared memory segments
for SEG_BYTES in `ipcs -m | cut -c44-300 | awk '{print $1}' | grep "[0-9][0-9]*"`
do
    MIN_PG=`echo "$SEG_BYTES/($HPG_SZ*1024)" | bc -q`
    if [ $MIN_PG -gt 0 ]; then
        NUM_PG=`echo "$NUM_PG+$MIN_PG+1" | bc -q`
    fi
done

RES_BYTES=`echo "$NUM_PG * $HPG_SZ * 1024" | bc -q`

# An SGA less than 100MB does not make sense
# Bail out if that is the case
if [ $RES_BYTES -lt 100000000 ]; then
    echo "***********"
    echo "** ERROR **"
    echo "***********"
    echo "Sorry! There are not enough total of shared memory segments allocated for
HugePages configuration. HugePages can only be used for shared memory segments
that you can list by command:

    # ipcs -m

of a size that can match an Oracle Database SGA. Please make sure that:
 * Oracle Database instance is up and running
 * Oracle Database 11g Automatic Memory Management (AMM) is not configured"
    exit 1
fi

# Finish with results
case $KERN in
    '2.4') HUGETLB_POOL=`echo "$NUM_PG*$HPG_SZ/1024" | bc -q`;
           echo "Recommended setting: vm.hugetlb_pool = $HUGETLB_POOL" ;;
    '2.6') echo "Recommended setting: vm.nr_hugepages = $NUM_PG" ;;
    '3.8') echo "Recommended setting: vm.nr_hugepages = $NUM_PG" ;;
    '3.10') echo "Recommended setting: vm.nr_hugepages = $NUM_PG" ;;
    '4.1') echo "Recommended setting: vm.nr_hugepages = $NUM_PG" ;;
    '4.14') echo "Recommended setting: vm.nr_hugepages = $NUM_PG" ;;
    *) echo "Kernel version $KERN is not supported by this script (yet). Exiting." ;;
esac

# End

使用如下，可得:

[root@newarpdb01 u01]# ./hugepages_settings.sh 

This script is provided by Doc ID 401749.1 from My Oracle Support
(http://support.oracle.com) where it is intended to compute values for
the recommended HugePages/HugeTLB configuration for the current shared
memory segments on Oracle Linux. Before proceeding with the execution please note following:
 * For ASM instance, it needs to configure ASMM instead of AMM.
 * The 'pga_aggregate_target' is outside the SGA and
   you should accommodate this while calculating the overall size.
 * In case you changes the DB SGA size,
   as the new SGA will not fit in the previous HugePages configuration,
   it had better disable the whole HugePages,
   start the DB with new SGA size and run the script again.
And make sure that:
 * Oracle Database instance(s) are up and running
 * Oracle Database 11g Automatic Memory Management (AMM) is not setup
   (See Doc ID 749851.1)
 * The shared memory segments can be listed by command:
     # ipcs -m


Press Enter to proceed...

Recommended setting: vm.nr_hugepages = 117001

修改vm.nr_hugepages參數

# vi /etc/sysctl.conf
修改vm.nr_hugepages參數
vm.nr_hugepages = 117001
執行sysctl –p使配置生效：
# sysctl -p

備注: Hugepages是在分配后就會預留出來的，其大小一定要比服務器上所有實例的SGA總和要大

關於memlock, 修改內核參數memlock，單位是KB，如果內存是16G，memlock的大小要稍微小於物理內存。計划lock 12GB的內存大小。參數設置為大於SGA是沒有壞處的。

然后重啟各個實例的RAC的數據庫。

RAC數據庫無法重啟，出現ORA-27100: shared memory realm already exists Linux-x86_64 Error: 17: File exists Additional information: 2097152

查看shared memory

[root@newarpdb01 u01]# su - oracle
[oracle@newarpdb01 ~]$ $ORACLE_HOME/bin/sysresv
------ Shared Memory Segments --------
key        shmid      owner      perms      bytes      nattch     status      
0x00000000 1441809    grid       640        4096       0                       
0x00000000 1474578    grid       640        4096       0                       
0x6d9f45cc 1507347    grid       640        4096       0                       
0x00000000 1572884    oracle     640        1610612736 102                     
0x00000000 1605653    oracle     640        243739394048 101                     
0xb0b15d50 1638422    oracle     640        2097152    102   

定位了1605653。占用了
Linux: 
% ipcrm shm 1605653

然后 重啟某實例的RAC數據庫

srvctl stop instance  -d newarpdb -i newarpdb2
srvctl start instance  -d newarpdb -i newarpdb2

文獻參考：
Semaphores and Shared Memory - An Overview (Doc ID 153961.1)
Startup Error ORA-27123: Unable To Attach To Shared Memory Segment Linux-x86_64 Error: 22: Invalid argument (Doc ID 1607545.1)
Configuring HugePages for Oracle on Linux (x86-64)