磨礪技術珠磯，踐行數據之道，追求卓越價值
回到上一級頁面：PostgreSQL內部結構與源代碼研究索引頁 回到頂級頁面：PostgreSQL索引頁
[作者 高健@博客園 luckyjackgao@gmail.com]
首先學習網上例子，進行體驗性的試驗：
首先不限制內存使用來進行下載：
[root@cent6 Desktop]# free -m total used free shared buffers cached Mem: 2006 484 1522 0 29 175
-/+ buffers/cache: 279 1727 Swap: 4031 0 4031 [root@cent6 Desktop]#
然后，再下載約700M:
wget http://centos.arcticnetwork.ca/6.4/isos/x86_64/CentOS-6.4-x86_64-LiveCD.iso
然后看內存使用情況：
[root@cent6 Desktop]# free -m total used free shared buffers cached Mem: 2006 1224 782 0 33 878
-/+ buffers/cache: 312 1694 Swap: 4031 0 4031 [root@cent6 Desktop]#
確實是用掉了700多M內存。
然后，重新啟動，限制內存使用：
[root@cent6 Desktop]# service cgconfig status Stopped #mount -t cgroup -o memory memcg /cgroup # mkdir /cgroup/GroupA # echo 10M > /cgroup/GroupA/memory.limit_in_bytes # echo $$ > /cgroup/GroupA/tasks
然后，再看內存狀況：
[root@cent6 Desktop]# free -m total used free shared buffers cached Mem: 2006 481 1525 0 29 174
-/+ buffers/cache: 276 1729 Swap: 4031 0 4031 [root@cent6 Desktop]#
再下載約700M:
wget http://centos.arcticnetwork.ca/6.4/isos/x86_64/CentOS-6.4-x86_64-LiveCD.iso
再看內存使用前后對比:
[root@cent6 Desktop]# free -m total used free shared buffers cached Mem: 2006 512 1494 0 32 186
-/+ buffers/cache: 293 1713 Swap: 4031 0 4031 [root@cent6 Desktop]#
可以知道，大約的內存使用量為 1525-1494=31M。不過free命令觀察到的結果是有誤差的，程序執行時間長，free就是一個不斷累減的值，由於當前shell被限制使用內存最大10M，那么基數很小的情況下，時間越長，誤差越大。
下面，看看對PostgreSQL能否產生良好的限制：
再此之前，通過系統設定來看看對postgres用戶進行wget操作時的內存的控制：
[postgres@cent6 Desktop]$ cat /etc/cgconfig.conf #
# Copyright IBM Corporation. 2007
#
# Authors: Balbir Singh <balbir@linux.vnet.ibm.com>
# This program is free software; you can redistribute it and/or modify it # under the terms of version 2.1 of the GNU Lesser General Public License # as published by the Free Software Foundation.
#
# This program is distributed in the hope that it would be useful, but # WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
#
# See man cgconfig.conf for further details.
#
# By default, mount all controllers to /cgroup/<controller> mount { cpuset = /cgroup/cpuset; cpu = /cgroup/cpu; cpuacct = /cgroup/cpuacct; memory = /cgroup/memory; devices = /cgroup/devices; freezer = /cgroup/freezer; net_cls = /cgroup/net_cls; blkio = /cgroup/blkio; } group test1 { perm { task{ uid=postgres; gid=postgres; } admin{ uid=root; gid=root; } } memory { memory.limit_in_bytes=30M; } } [postgres@cent6 Desktop]$
還有一個文件，cgrules.conf，也很重要:
[postgres@cent6 Desktop]$ cat /etc/cgrules.conf # /etc/cgrules.conf #
#Each line describes a rule for a user in the forms: #
#<user> <controllers> <destination>
#<user>:<process name> <controllers> <destination>
#
#Where: # <user> can be: # - an user name # - a group name, with @group syntax # - the wildcard *, for any user or group.
# - The %, which is equivalent to "ditto". This is useful for
# multiline rules where different cgroups need to be specified # for various hierarchies for a single user.
#
# <process name> is optional and it can be: # - a process name # - a full command path of a process #
# <controller> can be: # - comma separated controller names (no spaces)
# - * (for all mounted controllers)
#
# <destination> can be: # - path with-in the controller hierarchy (ex. pgrp1/gid1/uid1)
#
# Note: # - It currently has rules based on uids, gids and process name.
#
# - Don't put overlapping rules. First rule which matches the criteria # will be executed.
#
# - Multiline rules can be specified for specifying different cgroups # for multiple hierarchies. In the example below, user "peter" has # specified 2 line rule. First line says put peter's task in test1/
# dir for "cpu" controller and second line says put peter's tasks in # test2/ dir for memory controller. Make a note of "%" sign in second line.
# This is an indication that it is continuation of previous rule.
#
#
#<user> <controllers> <destination>
#
#john cpu usergroup/faculty/john/
#john:cp cpu usergroup/faculty/john/cp #@student cpu,memory usergroup/student/
#peter cpu test1/
#% memory test2/
#@root * admingroup/
#* * default/
# End of file postgres memory test1/
# [postgres@cent6 Desktop]$
在root用戶，設置如下兩個服務隨系統啟動：
chkconfig cgconfig on
chkconfig cgred on
然后重新啟動系統后，用postgres用戶進行登錄，進行檢驗：
[postgres@cent6 Desktop]$ free -m total used free shared buffers cached Mem: 2006 381 1625 0 25 134 -/+ buffers/cache: 221 1785
[postgres@cent6 Desktop]$ wget http://centos.arcticnetwork.ca/6.4/isos/x86_64/CentOS-6.4-x86_64-LiveCD.iso
執行完畢后，看內存狀況，成功。
[postgres@cent6 Desktop]$ free -m total used free shared buffers cached Mem: 2006 393 1613 0 28 141 -/+ buffers/cache: 224 1782 Swap: 4031 67 3964 [postgres@cent6 Desktop]$
下面看對postgresql中執行sql 的限制如何：
步驟1: 對/etc/cgconfig.conf 文件和 /etc/cgrules.conf 文件的設置如前所述。
步驟2: 運行前查看內存狀況：
[postgres@cent6 Desktop]$ free -m
total used free shared buffers cached
Mem: 2006 384 1622 0 26 138
-/+ buffers/cache: 219 1787
Swap: 4031 87 3944
[postgres@cent6 Desktop]$
步驟3: 開始處理大量數據(約600MB)
postgres=# select count(*) from test01; count
-------
0 (1 row) postgres=# insert into test01 values(generate_series(1,614400),repeat( chr(int4(random()*26)+65),1024));
運行剛剛開始，就出現了如下的錯誤：
The connection to the server was lost. Attempting reset: Failed. !>
這和之前碰到的崩潰情形一致。
PostgreSQL的log本身是這樣的：
[postgres@cent6 pgsql]$ LOG: database system was shut down at 2013-09-09 16:20:29 CST LOG: database system is ready to accept connections LOG: autovacuum launcher started LOG: server process (PID 2697) was terminated by signal 9: Killed DETAIL: Failed process was running: insert into test01 values(generate_series(1,614400),repeat( chr(int4(random()*26)+65),1024)); LOG: terminating any other active server processes WARNING: terminating connection because of crash of another server process DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory. HINT: In a moment you should be able to reconnect to the database and repeat your command. FATAL: the database system is in recovery mode LOG: all server processes terminated; reinitializing LOG: database system was interrupted; last known up at 2013-09-09 17:35:42 CST LOG: database system was not properly shut down; automatic recovery in progress LOG: redo starts at 1/9E807C90 LOG: unexpected pageaddr 1/946BE000 in log file 1, segment 159, offset 7069696 LOG: redo done at 1/9F6BDB50 LOG: database system is ready to accept connections LOG: autovacuum launcher started
通過dmesg命令，可以看到，發生了Out of Memory錯誤，這次是 cgroup out of memory
[postgres@cent6 Desktop]$ dmesg | grep post [ 2673] 500 2673 64453 200 0 0 0 postgres [ 2675] 500 2675 64494 79 0 0 0 postgres [ 2676] 500 2676 64453 75 0 0 0 postgres [ 2677] 500 2677 64453 77 0 0 0 postgres [ 2678] 500 2678 64667 80 0 0 0 postgres [ 2679] 500 2679 28359 72 0 0 0 postgres [ 2697] 500 2697 64764 100 0 0 0 postgres [ 2673] 500 2673 64453 200 0 0 0 postgres [ 2675] 500 2675 64494 79 0 0 0 postgres [ 2676] 500 2676 64453 75 0 0 0 postgres [ 2677] 500 2677 64453 77 0 0 0 postgres [ 2678] 500 2678 64667 80 0 0 0 postgres [ 2679] 500 2679 28359 72 0 0 0 postgres [ 2697] 500 2697 64764 100 0 0 0 postgres [ 2673] 500 2673 64453 208 0 0 0 postgres [ 2675] 500 2675 64494 79 0 0 0 postgres [ 2676] 500 2676 64453 98 0 0 0 postgres [ 2677] 500 2677 64453 782 0 0 0 postgres [ 2678] 500 2678 64667 133 0 0 0 postgres [ 2679] 500 2679 28359 86 0 0 0 postgres [ 2697] 500 2697 73075 3036 0 0 0 postgres Memory cgroup out of memory: Kill process 2697 (postgres) score 1000 or sacrifice child Killed process 2697, UID 500, (postgres) total-vm:292300kB, anon-rss:8432kB, file-rss:3712kB [postgres@cent6 Desktop]$
我懷疑自己的內存開得過小了，影響到一些基本的運行。PostgreSQL本身也需要一些資源(shared_buffers、wal_buffers都需要用一些內存)
所以我調整了參數 memory.limit_in_bytes=300M ，再次運行：
前述的sql問處理1200MB數據，成功結束，內存沒有過多增長。
[作者 高健@博客園 luckyjackgao@gmail.com]
回到上一級頁面：PostgreSQL內部結構與源代碼研究索引頁 回到頂級頁面：PostgreSQL索引頁
磨礪技術珠磯，踐行數據之道，追求卓越價值