oracle大頁無法使用問題

oracle 中的大頁問題
今天巡檢數據庫的時候，發現有一個備庫節點上的負載很大。這個備庫基本無人使用。僅做延時查詢使用，按理來說不應該有這么大負載，
環境說明  11.2.0.4 單節點dg庫。
問題 大頁不生效。且數據庫所在服務器負載極大
top - 09:33:00 up 326 days, 19 min,  3 users,  load average: 17.69, 11.65, 5.83   --負載說明
Tasks: 357 total,   1 running, 356 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.2%us,  0.3%sy,  0.0%ni, 68.4%id, 31.1%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  65970692k total, 64774768k used,  1195924k free,     6992k buffers
Swap: 33054712k total,  5563048k used, 27491664k free,   399336k cached
  179 root      20   0     0    0    0 D  0.3  0.0 517:34.66 kswapd0  
查詢當前大頁大小：
grep HugePages /proc/meminfo
AnonHugePages:         0 kB
HugePages_Total:    9960
HugePages_Free:     9960
HugePages_Rsvd:        0
HugePages_Surp:        0
數據庫中sga大小
SQL> show parameter sga
NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
lock_sga                             boolean     FALSE
pre_page_sga                         boolean     FALSE
sga_max_size                         big integer 31G
sga_target                           big integer 31G
看到這里問題有點明朗了，好像是數據庫沒有使用大頁導致的。滿心歡喜的去查詢了一下大頁修改的方法。
根據如下 shell腳本跑出了一個修改值。

#!/bin/bash  
#  
# hugepages_settings.sh  
#  
# Linux bash script to compute values for the  
# recommended HugePages/HugeTLB configuration  
#  
# Note: This script does calculation for all shared memory  
# segments available when the script is run, no matter it  
# is an Oracle RDBMS shared memory segment or not.  
#  
# This script is provided by Doc ID 401749.1 from My Oracle Support   
# http://support.oracle.com  
  
# Welcome text  
echo "  
This script is provided by Doc ID 401749.1 from My Oracle Support   
(http://support.oracle.com) where it is intended to compute values for   
the recommended HugePages/HugeTLB configuration for the current shared   
memory segments. Before proceeding with the execution please note following:  
 * For ASM instance, it needs to configure ASMM instead of AMM.  
 * The 'pga_aggregate_target' is outside the SGA and   
   you should accommodate this while calculating SGA size.  
 * In case you changes the DB SGA size,   
   as the new SGA will not fit in the previous HugePages configuration,   
   it had better disable the whole HugePages,   
   start the DB with new SGA size and run the script again.  
And make sure that:  
 * Oracle Database instance(s) are up and running  
 * Oracle Database 11g Automatic Memory Management (AMM) is not setup   
   (See Doc ID 749851.1)  
 * The shared memory segments can be listed by command:  
     # ipcs -m  
  
  
Press Enter to proceed..."  
  
read  
  
# Check for the kernel version  
KERN=`uname -r | awk -F. '{ printf("%d.%d\n",$1,$2); }'`  
  
# Find out the HugePage size  
HPG_SZ=`grep Hugepagesize /proc/meminfo | awk '{print $2}'`  
if [ -z "$HPG_SZ" ];then  
    echo "The hugepages may not be supported in the system where the script is being executed."  
    exit 1  
fi  
  
# Initialize the counter  
NUM_PG=0  
  
# Cumulative number of pages required to handle the running shared memory segments  
for SEG_BYTES in `ipcs -m | cut -c44-300 | awk '{print $1}' | grep "[0-9][0-9]*"`  
do  
    MIN_PG=`echo "$SEG_BYTES/($HPG_SZ*1024)" | bc -q`  
    if [ $MIN_PG -gt 0 ]; then  
        NUM_PG=`echo "$NUM_PG+$MIN_PG+1" | bc -q`  
    fi  
done  
  
RES_BYTES=`echo "$NUM_PG * $HPG_SZ * 1024" | bc -q`  
  
# An SGA less than 100MB does not make sense  
# Bail out if that is the case  
if [ $RES_BYTES -lt 100000000 ]; then  
    echo "***********"  
    echo "** ERROR **"  
    echo "***********"  
    echo "Sorry! There are not enough total of shared memory segments allocated for   
HugePages configuration. HugePages can only be used for shared memory segments   
that you can list by command:  
  
    # ipcs -m  
  
of a size that can match an Oracle Database SGA. Please make sure that:  
 * Oracle Database instance is up and running   
 * Oracle Database 11g Automatic Memory Management (AMM) is not configured"  
    exit 1  
fi  
  
# Finish with results  
case $KERN in    '2.2') echo "Kernel version $KERN is not supported. Exiting." ;;  
    '2.4') HUGETLB_POOL=`echo "$NUM_PG*$HPG_SZ/1024" | bc -q`;  
           echo "Recommended setting: vm.hugetlb_pool = $HUGETLB_POOL" ;;  
    '2.6') echo "Recommended setting: vm.nr_hugepages = $NUM_PG" ;;  
esac  
  
# End  

然后修改 /etc/sysctl.conf   
vm.nr_hugepages = 20000 
sysctl -p    --應用參數  到這里也沒有任何問題。
修改完成之后重啟數據庫  
recover managed standby database cancel;
shutdown immediate;
startup 
啟動之后繼續查詢大頁的參數 
[oracle@itunesdg4 ~]$ grep HugePages /proc/meminfo
AnonHugePages:         0 kB
HugePages_Total:    9960
HugePages_Free:     9960
HugePages_Rsvd:        0
HugePages_Surp:        0
發現居然沒有改變。。。
因為這個是一個測試庫，於是重啟了一下 再次查詢
[oracle@itunesdg4 ~]$  grep HugePages /proc/meminfo
AnonHugePages:    176128 kB
HugePages_Total:   20000
HugePages_Free:    20000
HugePages_Rsvd:        0
HugePages_Surp:        0
發現變HugePages_Total是變過來了。但是大頁依舊沒有使用上。
再次查詢數據庫相關的參數 
alter system reset memory_target scope=spfile;                              
alter system reset memory_max_target scope=spfile;
SQL> show parameter memory_target
NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
memory_target                        big integer 0
SQL> show parameter memory_max_target
NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
memory_max_target                    big integer 0
值也都是正常的，最后使出大招修改這個參數 
alter system set use_large_pages=only scope=spfile; 讓數據庫強制使用 大頁。重啟數據庫。
重啟時報錯了：
[oracle@itunesdg4 ~]$ sqlplus / as sysdba
SQL*Plus: Release 11.2.0.4.0 Production on Sat Feb 24 14:19:35 2018
Copyright (c) 1982, 2013, Oracle.  All rights reserved.
Connected to an idle instance.
SQL> startup
ORA-27137: unable to allocate large pages to create a shared memory segment
Linux-x86_64 Error: 1: Operation not permitted
Additional information: 201326592
Additional information: 1
上網查詢了一下解決方法。
When a process uses some memory, the CPU is marking the RAM as used by that process. For efficiency, the CPU allocate RAM by chunks of 4K bytes (it's the default value on many platforms). Those chunks are named pages. Those pages can be swapped to disk, etc.

Since the process address space are virtual, the CPU and the operating system have to remember which page belong to which process, and where it is stored. Obviously, the more pages you have, the more time it takes to find where the memory is mapped. When a process uses 1GB of memory, that's 262144 entries to look up (1GB / 4K). If one Page Table Entry consume 8bytes, that's 2MB (262144 * 8) to look-up.

Most current CPU architectures support bigger pages (so the CPU/OS have less entries to look-up), those are named Huge pages (on Linux), Super Pages (on BSD) or Large Pages (on Windows), but it all the same thing.

1: [root@gsp cdrom]# id oracle
2: uid=501(oracle) gid=501(oinstall) groups=501(oinstall),502(dba)
3: [root@gsp cdrom]# more /proc/sys/vm/hugetlb_shm_group
4: 0
5: [root@gsp cdrom]#  echo 502 >/proc/sys/vm/hugetlb_shm_group
按照網上給出的解決方法去設置了一下這個文件的值。
[oracle@itunesdg4 ~]$ id
uid=502(oracle) gid=501(oinstall) groups=501(oinstall),502(dba)
[oracle@itunesdg4 ~]$ echo 502 >/proc/sys/vm/hugetlb_shm_group
-bash: /proc/sys/vm/hugetlb_shm_group: Permission denied
[oracle@itunesdg4 ~]$ su -
Password: 
[root@itunesdg4 ~]# echo 502 >/proc/sys/vm/hugetlb_shm_group
[root@itunesdg4 ~]# exit
logout
[oracle@itunesdg4 ~]$ more /proc/sys/vm/hugetlb_shm_group
502
重啟數據庫 查詢大頁使用情況
[oracle@itunesdg4 ~]$  grep HugePages /proc/meminfo
AnonHugePages:    188416 kB
HugePages_Total:   20000
HugePages_Free:    17965
HugePages_Rsvd:    13838
HugePages_Surp:        0
發現能使用了。
那么究竟是 alter system set use_large_pages=only scope=spfile;  這個參數導致了大頁生效還是， hugetlb_shm_group 這個文件導致了大頁生效呢。做如下測試;
SQL> alter system set use_large_pages=true scope=spfile;
SQL> shutdown abort;               --我這里是測試庫 而且是dg庫所以直接abort 快速關閉。各位請勿模仿
[oracle@itunesdg4 ~]$  grep HugePages /proc/meminfo   --這里數據關閉所以釋放了大頁
AnonHugePages:    174080 kB
HugePages_Total:   20000
HugePages_Free:    20000
HugePages_Rsvd:        0
HugePages_Surp:        0
SQL> startup                       --重新啟動 
[oracle@itunesdg4 ~]$  grep HugePages /proc/meminfo    --大頁還是能正常使用 
AnonHugePages:    182272 kB
HugePages_Total:   20000
HugePages_Free:    18566
HugePages_Rsvd:    14439
HugePages_Surp:        0

第一個實驗做完了，發現並不是 use_large_pages這個參數導致的大頁能夠使用。那么嘗試修改 hugetlb_shm_group 這個文件 重啟數據庫 
[root@itunesdg4 ~]# echo 0 >/proc/sys/vm/hugetlb_shm_group    --修改文件的值 
[oracle@itunesdg4 ~]$ grep HugePages /proc/meminfo  --修改之后查詢正常使用大頁
AnonHugePages:    186368 kB
HugePages_Total:   20000
HugePages_Free:    17977
HugePages_Rsvd:    13850
HugePages_Surp:        0
SQL> shutdown abort;         --重啟 
SQL> startup
[oracle@itunesdg4 ~]$ grep HugePages /proc/meminfo   --這里並沒有使用大頁了。
AnonHugePages:    182272 kB
HugePages_Total:   20000
HugePages_Free:    20000
HugePages_Rsvd:        0
HugePages_Surp:        0
果然是因為這個文件的緣故，那么這個文件究竟是干什么用的呢 
網上查詢可知：
hugetlb_shm_group
hugetlb_shm_group contains group id that is allowed to create SysV shared memory segment using hugetlb page 
該文件表示允許使用hugetlb頁創建System VIPC共享內存段的系統組ID。
且這個文件在重啟之后會重置成0，所以需要在 /etc/sysctl.conf 中 添加一行 vm.hugetlb_shm_group = 502


最后總結一下大頁的配置步驟。
1.查詢是否開啟了大頁 
[oracle@itunesdg4 ~]$ cat /proc/meminfo |grep Huge    --如果total 為0則未開啟大頁。
AnonHugePages:    196608 kB
HugePages_Total:   20000
HugePages_Free:    17965
HugePages_Rsvd:    13838
HugePages_Surp:        0
2.如果未開啟，可以用上面的計算腳本算出一個系統給出的大頁值。
3.在/etc/sysctl.conf 中添加一行 vm.nr_hugepages= 計算出的值  
4.刷新sysctl.conf 這個文件 sysctl -p 並查詢 大頁的total是否為你設置的值，如果還是0多刷新幾次。
5.設置內存鎖：內存鎖的數量要大於所設置的大頁數量。可以設置為 -1 ，不限制。
vi /etc/security/limits.conf
oracle soft memlock -1
oracle hard memlock -1
6.重啟數據庫 查詢 cat /proc/meminfo |grep Huge 如果 free的值再減少則說明大頁正在被使用。如果沒使用，或許按照我的方法能給你們提供幫助。