Linux IO/NFS tunning:
IO Test
=======
dd 測試讀性能的命令
# time dd if=/nfsfolder/testfile of=/dev/null bs=1k
dd 測試寫性能的命令
time -p dd if=/dev/zero of=test.file bs=1k count=100000
[1]Write
time -p dd if=/dev/zero of=test.file bs=1k count=100000
100000+0 records in
100000+0 records out
102400000 bytes (102 MB) copied, 25.2433 s, 4.1 MB/s
real 25.24
user 0.03
sys 1.49
time -p dd if=/dev/zero of=test.file bs=1024k count=100
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 0.477919 s, 219 MB/s
real 0.48
user 0.00
sys 0.11
[2] read file
time dd if=./test.file of=/dev/null bs=1k
100000+0 records in
100000+0 records out
102400000 bytes (102 MB) copied, 14.1445 s, 7.2 MB/s
real 0m14.150s
user 0m0.054s
sys 0m1.929s
time dd if=./test.file of=/dev/null bs=1024k
97+1 records in
97+1 records out
102400000 bytes (102 MB) copied, 0.0465829 s, 2.2 GB/s
real 0m0.052s
user 0m0.000s
sys 0m0.037s
time rm -rf test.file
real 0m0.093s
user 0m0.000s
sys 0m0.091s
time rm -rf test.file
real 0m0.101s
user 0m0.000s
sys 0m0.098s
Tools:
======
sar -b 1 10
-b Report I/O and transfer rate statistics. The following values are displayed:
tps
Total number of transfers per second that were issued to physical devices. A
transfer is an I/O request to a physical device. Multiple logical requests can
be combined into a single I/O request to the device. A transfer is of inde\u2010
terminate size.
rtps
Total number of read requests per second issued to physical devices.
wtps
Total number of write requests per second issued to physical devices.
bread/s
Total amount of data read from the devices in blocks per second. Blocks are
equivalent to sectors and therefore have a size of 512 bytes.
bwrtn/s
Total amount of data written to devices in blocks per second.
注：每1秒 顯示 1次 顯示 10次
sar -d -p 3 10 //3s 10 times.
Monitor NFS
1 second interval to monitor nfsstat
nfsstat -cn -Z1
Client nfs v3:
null getattr setattr lookup access readlink
0 0% 777 58% 0 0% 0 0% 111 8% 0 0%
read write create mkdir symlink mknod
0 0% 432 32% 0 0% 0 0% 0 0% 0 0%
remove rmdir rename link readdir readdirplus
0 0% 0 0% 0 0% 0 0% 0 0% 0 0%
fsstat fsinfo pathconf commit
0 0% 0 0% 0 0% 0 0%
nfsstat -rc
Client rpc stats:
calls retrans authrefrsh
66906770 5 66907434
If retrans value is quite high, meaning that retransmissions were often necessary since the last reboot. This is a clear indication that the number of available NFS kernel threads on the server is insufficient to handle the requests from this client. The default number of threads for rpc.nfsd to start is typically eight threads.
To tell rpc.nfsd to use more kernel threads, the number of threads must be passed as an argument to it. Typically, most distributions will have a file such as /etc/sysconfig/nfs to configure this;
NFS server side:
nfsstat --all
dstat用來監控cpu,disk,network等, 類似vmstat.如下,比較直觀.
[root@LMG3ADFBE2 ~] # dstat
----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system--
usr sys idl wai hiq siq| read writ| recv send| in out | int csw
1 1 98 0 0 0| 244B 66k| 0 0 | 0 0 |2139 2048
0 3 97 0 0 0| 0 0 | 195B 838B| 0 0 |2865 8355
0 1 99 0 0 0| 0 0 |1038B 628B| 0 0 | 723 537
0 2 98 0 0 0| 0 0 | 66B 358B| 0 0 |1884 1000
dstat常用的選項有：
-c 顯示cpu使用情況
-d 顯示磁盤使用情況
-g, 顯示頁面數據
-i 啟用中斷數據
-l 平均負載統計（1分鍾，5分鍾，15分鍾）
-m 顯示內存使用信息
-n 顯示網絡信息
-p enable process stats (runnable,uninterruptible, new)
-r enable I/O request stats(read, write requests)
-s 顯示交換分區使用信息
-t 啟用時間/日期輸出
-y 系統統計(中斷上下文切換)
--aio 顯示異步io統計數據
--ipc ipc相關的信息
--raw: raw socket
--tcp tcp socket
--udp udp socket
--socket raw, tcp, udp
--unix unix sock
--top-cpu 顯示最占用CPU的進程
--top-bio 顯示最占用block IO的進程
--top-mem: 顯示最耗費內存的進程
--top-io 最占用IO的進程
http://www.softpanorama.org/Net/Application_layer/NFS/nfs_performance_tuning.shtml
關於NFS ac/noac 和nocto
cto / nocto Selects whether to use close-to-open cache coherence semantics.
If neither option is specified (or if cto is specified), the
client uses close-to-open cache coherence semantics. If the nocto
option is specified, the client uses a non-standard heuristic to
determine when files on the server have changed.
Using the nocto option may improve performance for read-only
mounts, but should be used only if the data on the server changes
only occasionally. The DATA AND METADATA COHERENCE section dis-
cusses the behavior of this option in more detail.
設置這個參數如nocto后, 行為表現為.
Server A and Server B mount the same storage via NFS.
Server B nocto, ac, 當A寫入一行到一個文件， 在B上看不到A寫的內容。知道觸發ac的timeout， 即actimo
ac/noac
ac / noac Selects whether the client may cache file attributes. If neither
option is specified (or if ac is specified), the client caches
file attributes.
To improve performance, NFS clients cache file attributes. Every
few seconds, an NFS client checks the server\u2019s version of each
file\u2019s attributes for updates. Changes that occur on the server
in those small intervals remain undetected until the client checks
the server again. The noac option prevents clients from caching
file attributes so that applications can more quickly detect file
changes on the server.
In addition to preventing the client from caching file attributes,
the noac option forces application writes to become synchronous so
that local changes to a file become visible on the server immedi-
ately. That way, other clients can quickly detect recent writes
when they check the file\u2019s attributes.
Using the noac option provides greater cache coherence among NFS
clients accessing the same files, but it extracts a significant
performance penalty. As such, judicious use of file locking is
encouraged instead. The DATA AND METADATA COHERENCE section con-
tains a detailed discussion of these trade-offs.
如果ac enable， 則文件屬性及目錄屬性會在本地cache中（metadata cache）， 可以touch一個新文件來強制刷新這些cache。
如果server A with ac， server B with noac, 在A上向文件寫入一行， 則B上可以看到最新的內容。當然如果B上設置了nocto，則看不到。
- Shall noac option on server B trigger the sync operation on server A? Yes. When the client mounted with noac option reads a file from the server, It will always fetch the most recent copy from the NFS server. If the NFS server thinks that a lock has been acquired for the same file by another client (clients acquire locks before performing write), then the NFS server asks that client to forcefully write the data, then the updated copy is provided to the client mounted with noac option.