使用cgroups來控制磁盤IO帶寬

磨礪技術珠磯，踐行數據之道，追求卓越價值

回到上一級頁面：PostgreSQL內部結構與源代碼研究索引頁    回到頂級頁面：PostgreSQL索引頁

[作者 高健@博客園  [e-mail]]

 

可資參考的資料：

http://kaivanov.blogspot.com/2012/07/setting-up-linux-cgroups-control-groups.html

http://www.oracle.com/technetwork/articles/servers-storage-admin/resource-controllers-linux-1506602.html

首先用 hdparm測試磁盤讀取的最大帶寬:

hdparm --direct -t /dev/sda

測試結果為200MB/s以上。

然后給用戶postgres設置參數，保持磁盤讀寫I/O為10MB/秒以下（此用戶名下所有進程總和）。

# Configuration file generated by cgsnapshot mount { cpuset = /cgroup/cpuset; cpu = /cgroup/cpu; cpuacct = /cgroup/cpuacct; memory = /cgroup/memory; devices = /cgroup/devices; freezer = /cgroup/freezer; net_cls = /cgroup/net_cls; blkio = /cgroup/blkio; } group io-test { perm { task{ uid=postgres; gid=postgres; } admin{ uid=root; gid=root; } } blkio { blkio.throttle.write_iops_device=""; blkio.throttle.read_iops_device=""; blkio.throttle.write_bps_device="8:0 10485760"; blkio.throttle.read_bps_device="8:0 10485760"; blkio.reset_stats=""; blkio.weight="500"; blkio.weight_device=""; } } [postgres@cent6 Desktop]$ 

再看

[postgres@cent6 Desktop]$ cat /etc/cgrules.conf # /etc/cgrules.conf # #Each line describes a rule for a user in the forms: # #<user>            <controllers>        <destination> #<user>:<process name>    <controllers>        <destination> # #Where: # <user> can be: # - an user name # - a group name, with @group syntax # - the wildcard *, for any user or group. # - The %, which is equivalent to "ditto". This is useful for # multiline rules where different cgroups need to be specified # for various hierarchies for a single user. # # <process name> is optional and it can be: # - a process name # - a full command path of a process # # <controller> can be: # - comma separated controller names (no spaces) # - * (for all mounted controllers) # # <destination> can be: # - path with-in the controller hierarchy (ex. pgrp1/gid1/uid1) # # Note: # - It currently has rules based on uids, gids and process name. # # - Don't put overlapping rules. First rule which matches the criteria
# will be executed. # # - Multiline rules can be specified for specifying different cgroups # for multiple hierarchies. In the example below, user "peter" has # specified 2 line rule. First line says put peter's task in test1/
#   dir for "cpu" controller and second line says put peter's tasks in
#   test2/ dir for memory controller. Make a note of "%" sign in second line. # This is an indication that it is continuation of previous rule. # # #<user>      <controllers>      <destination> # #john cpu usergroup/faculty/john/ #john:cp       cpu        usergroup/faculty/john/cp #@student cpu,memory usergroup/student/ #peter cpu test1/ #%           memory        test2/ #@root *        admingroup/ #*        *        default/ postgres blkio io-test/ # End of file # # # [postgres@cent6 Desktop]$ 

先用dd命令實際測試看看：

[postgres@cent6 Desktop]$ dd if=/dev/zero of=testfile2 bs=4K count=1024 oflag=direct 1024+0 records in
1024+0 records out 4194304 bytes (4.2 MB) copied, 0.372953 s, 11.2 MB/s [postgres@cent6 Desktop]$ dd if=/dev/zero of=testfile2 bs=8K count=1024 oflag=direct 1024+0 records in
1024+0 records out 8388608 bytes (8.4 MB) copied, 0.733823 s, 11.4 MB/s [postgres@cent6 Desktop]$ dd if=/dev/zero of=testfile2 bs=8K count=1024 oflag=direct 1024+0 records in
1024+0 records out 8388608 bytes (8.4 MB) copied, 0.733256 s, 11.4 MB/s [postgres@cent6 Desktop]$ [postgres@cent6 Desktop]$ dd if=/dev/zero of=testfile2 bs=16K count=1024 oflag=direct 1024+0 records in
1024+0 records out [postgres@cent6 Desktop]$ 

當然，如果dd執行時，開數據塊太多，又是這種完全空的塊，誤差就會增大：

[postgres@cent6 Desktop]$ dd if=/dev/zero of=testfile1 bs=10240 count=300000
300000+0 records in
300000+0 records out 3072000000 bytes (3.1 GB) copied, 57.6779 s, 53.3 MB/s [postgres@cent6 Desktop]$

用實際的下載動作進行測試：

[postgres@cent6 Desktop]$ wget http://centos.arcticnetwork.ca/6.4/isos/x86_64/CentOS-6.4-x86_64-LiveCD.iso

再看看IO狀態：

[postgres@cent6 Desktop]$ iostat -x 20 Linux 2.6.32-279.el6.x86_64 (cent6.gao)     09/10/2013     _x86_64_    (1 CPU) avg-cpu:  %user   %nice %system %iowait  %steal   %idle 1.38    0.00    3.55   12.16    0.00   82.91 Device: rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util sda 7.50  1747.20    7.77   24.88   535.75 14224.09   452.17    12.54  384.25   4.41  14.40 dm-0              0.00     0.00   13.79 1771.99   524.65 14224.07     8.26  1975.37 1106.16   0.09  16.73 dm-1              0.00     0.00    0.38    0.00     3.03     0.00     8.00     0.00    3.68   2.34   0.09 avg-cpu:  %user   %nice %system %iowait  %steal   %idle 3.65    0.00    2.14    0.73    0.00   93.48 Device: rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util sda 0.00     1.82    1.09    1.56    49.64    25.44    28.24     0.03   11.78   3.57   0.95 dm-0              0.00     0.00    1.04    3.18    48.80    25.44    17.58     0.03    7.35   2.15   0.91 dm-1              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00 avg-cpu:  %user   %nice %system %iowait  %steal   %idle 1.65    0.00    1.65    0.98    0.00   95.72 Device: rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util sda 0.00    60.28    0.00    1.44     0.00   492.12   341.14     0.02   11.79   8.86   1.28 dm-0              0.00     0.00    0.00   61.51     0.00   492.12     8.00     0.59    9.54   0.21   1.28 dm-1              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00

除了一開始的峰值，util長期在2%-3%之間，基本達到效果。

 

[作者 高健@博客園  [e-mail]]

磨礪技術珠磯，踐行數據之道，追求卓越價值

回到上一級頁面：PostgreSQL內部結構與源代碼研究索引頁    回到頂級頁面：PostgreSQL索引頁