rebalacne場景：
向asm磁盤組中添加，刪除，resize等操作都將會引起asm實例rebalacne。
reblaacne過程:
第一階段planning:
計算出rebalance的計划，會根據磁盤大小個數，磁盤吞吐，au大小等計算出大致計划，該過程一般只需幾分鍾。
第二階段extent relocating：
真正進行重平衡的過程，將extent均勻的分配在各個磁盤中。該過程通常是整個rebalance最耗時的過程。
第三階段compacting:
asm11.1.0.7以上才開始支持，將磁盤上存的數據盡可能的移動到磁盤的外圈磁道上去（機械盤的外圈速度更快），花費時間較少。
rebalance實際大小以及所需時間
磁盤組的rebalance什么時候能完成？這可能是很多人最關心的問題。通常情況下可以通過gv$asm_operation.est_minutes來知道還剩下幾分鍾可以完成rebalance雖然這個值並不准確。
做個小實驗體驗下整個過程：
首先了解個公式
Formula: (TAU/N)*N-1
Where:
TAU= Total Allocation Unit to move.
N=number of disks
查看磁盤組多少個extent
13:59:11 SQL> Select count(pxn_kffxp) Extents, disk_kffxp disk_number, group_kffxp group_number from x$kffxp where group_kffxp=6 group by disk_kffxp, group_kffxp order by group_kffxp,disk_kffxp;
EXTENTS DISK_NUMBER GROUP_NUMBER
---------- ----------- ------------
194 0 6
Elapsed: 00:00:00.00
13:59:16 SQL> select (total_mb-free_mb) REQUIRED_MIRROR_FREE_MB,USABLE_FILE_MB, type,ALLOCATION_UNIT_SIZE/1024 from v$asm_diskgroup where name='TEST';
REQUIRED_MIRROR_FREE_MB USABLE_FILE_MB TYPE ALLOCATION_UNIT_SIZE/1024
----------------------- -------------- ------------ -------------------------
200 457144 EXTERN 1024
Elapsed: 00:00:00.03
200m使用大小跟查出來194個extent大小基本相同
23:27:36 SQL> show parameter asm ---------------------- 查看asm rebalance並行度
NAME TYPE VALUE
------------------------------------ ---------------------- ------------------------------
asm_power_limit integer 1
23:06:28 SQL> alter diskgroup archdg drop disk ARCHDG_0005;
Diskgroup altered.
14:01:09 SQL> select * from v$asm_operation; ---------------------可以看到實際需要rebalance的extent是129=194/3*2
GROUP_NUMBER OPERATION STATE POWER ACTUAL SOFAR EST_WORK EST_RATE EST_MINUTES ERROR_CODE
------------ ---------- -------- ---------- ---------- ---------- ---------- ---------- ----------- ----------------------------------------------------------------------------------------
6 REBAL RUN 1 1 2 129 5098 0
在看下 EST_MINUTES字段
23:06:45 SQL> select INST_ID, OPERATION, STATE, POWER, SOFAR, EST_WORK, EST_RATE, EST_MINUTES from GV$ASM_OPERATION where GROUP_NUMBER=1;
INST_ID OPERATION STATE POWER SOFAR EST_WORK EST_RATE EST_MINUTES
---------- ---------- -------- ---------- ---------- ---------- ---------- -----------
2 REBAL WAIT 1
1 REBAL RUN 1 6166 53989 10157 4
這里顯示4分鍾完成，可以看到后台日志真正完成時間從23:06-23:13是7分鍾，這里的數據較少所以結果還不是特別明顯。
SQL> alter diskgroup archdg drop disk ARCHDG_0005
NOTE: GroupBlock outside rolling migration privileged region
NOTE: requesting all-instance membership refresh for group=1
Wed Jul 04 23:06:42 2018
GMON updating for reconfiguration, group 1 at 61 for pid 35, osid 30893
NOTE: group ARCHDG: updated PST location: disk 0002 (PST copy 0)
NOTE: group ARCHDG: updated PST location: disk 0004 (PST copy 1)
NOTE: group 1 PST updated.
Wed Jul 04 23:06:42 2018
NOTE: membership refresh pending for group 1/0x8ab98994 (ARCHDG)
GMON querying group 1 at 62 for pid 18, osid 20082
SUCCESS: refreshed membership for 1/0x8ab98994 (ARCHDG)
SUCCESS: alter diskgroup archdg drop disk ARCHDG_0005
NOTE: Attempting voting file refresh on diskgroup ARCHDG
NOTE: Refresh completed on diskgroup ARCHDG. No voting file found.
NOTE: starting rebalance of group 1/0x8ab98994 (ARCHDG) at power 1
Starting background process ARB0
Wed Jul 04 23:06:48 2018
ARB0 started with pid=36, OS id=41870
NOTE: assigning ARB0 to group 1/0x8ab98994 (ARCHDG) with 1 parallel I/O
cellip.ora not found.
Wed Jul 04 23:11:33 2018
NOTE: GroupBlock outside rolling migration privileged region
NOTE: requesting all-instance membership refresh for group=1
Wed Jul 04 23:11:36 2018
GMON updating for reconfiguration, group 1 at 63 for pid 37, osid 52733
NOTE: group ARCHDG: updated PST location: disk 0002 (PST copy 0)
NOTE: group ARCHDG: updated PST location: disk 0004 (PST copy 1)
NOTE: group 1 PST updated.
SUCCESS: grp 1 disk ARCHDG_0005 emptied
NOTE: erasing header on grp 1 disk ARCHDG_0005
NOTE: process _x000_+asm1 (52733) initiating offline of disk 5.3916003717 (ARCHDG_0005) with mask 0x7e in group 1
NOTE: initiating PST update: grp = 1, dsk = 5/0xe9697985, mask = 0x6a, op = clear
GMON updating disk modes for group 1 at 64 for pid 37, osid 52733
NOTE: group ARCHDG: updated PST location: disk 0002 (PST copy 0)
NOTE: group ARCHDG: updated PST location: disk 0004 (PST copy 1)
NOTE: PST update grp = 1 completed successfully
NOTE: initiating PST update: grp = 1, dsk = 5/0xe9697985, mask = 0x7e, op = clear
GMON updating disk modes for group 1 at 65 for pid 37, osid 52733
NOTE: group ARCHDG: updated PST location: disk 0002 (PST copy 0)
NOTE: group ARCHDG: updated PST location: disk 0004 (PST copy 1)
NOTE: cache closing disk 5 of grp 1: ARCHDG_0005
NOTE: PST update grp = 1 completed successfully
GMON updating for reconfiguration, group 1 at 66 for pid 37, osid 52733
NOTE: cache closing disk 5 of grp 1: (not open) ARCHDG_0005
NOTE: group ARCHDG: updated PST location: disk 0002 (PST copy 0)
NOTE: group ARCHDG: updated PST location: disk 0004 (PST copy 1)
NOTE: group 1 PST updated.
Wed Jul 04 23:11:36 2018
NOTE: membership refresh pending for group 1/0x8ab98994 (ARCHDG)
GMON querying group 1 at 67 for pid 18, osid 20082
GMON querying group 1 at 68 for pid 18, osid 20082
NOTE: Disk ARCHDG_0005 in mode 0x0 marked for de-assignment
SUCCESS: refreshed membership for 1/0x8ab98994 (ARCHDG)
NOTE: Attempting voting file refresh on diskgroup ARCHDG
NOTE: Refresh completed on diskgroup ARCHDG. No voting file found.
Wed Jul 04 23:13:57 2018
NOTE: stopping process ARB0
SUCCESS: rebalance completed for group 1/0x8ab98994 (ARCHDG)
此時可以使用ls -lrth +ASM_arb0*找到最新的arb trace文件，該文件會帶有文件號類似+ASM1_arb0_70552.trc，其中可以看到arb0進程正在遷移數據
*** 2018-07-04 23:24:07.940
ARB0 relocating file +ARCHDG.2115.980618919 (120 entries)
*** 2018-07-04 23:24:09.072
ARB0 relocating file +ARCHDG.2115.980618919 (120 entries)
*** 2018-07-04 23:24:11.826
ARB0 relocating file +ARCHDG.2115.980618919 (120 entries)
*** 2018-07-04 23:24:14.527
ARB0 relocating file +ARCHDG.2115.980618919 (120 entries)
通過pstack也可以看到這個進程使用了
kfgbRebalExecute - kfdaExecute - kffRelocate函數
#pstack 70552
#0 0x00007fe8467bc644 in __io_getevents_0_4 () from /lib64/libaio.so.1
#1 0x0000000002baea09 in skgfrliopo ()
#2 0x0000000002bae801 in skgfospo ()
#3 0x00000000086ba843 in skgfrwat ()
#4 0x0000000008599e5d in ksfdwtio ()
#5 0x000000000220019b in ksfdwat_internal ()
#6 0x0000000003edce17 in kfk_reap_ufs_async_io ()
#7 0x0000000003edcd5f in kfk_reap_ios_from_subsys ()
#8 0x0000000000adefa4 in kfk_reap_ios ()
#9 0x0000000003edc382 in kfk_io1 ()
#10 0x0000000003edbf28 in kfkRequest ()
#11 0x0000000003ee272e in kfk_transitIO ()
#12 0x0000000003e35d5e in kffRelocateWait ()
#13 0x0000000003e5cbf6 in kffRelocate ()
#14 0x0000000003dd22df in kfdaExecute ()
#15 0x0000000003eb563f in kfgbRebalExecute ()
#16 0x0000000003ea2414 in kfgbDriver ()
#17 0x00000000021a86d7 in ksbabs ()
#18 0x0000000003eb964c in kfgbRun ()
#19 0x00000000021ad44b in ksbrdp ()
#20 0x00000000023d3eef in opirip ()
#21 0x000000000167e7b5 in opidrv ()
#22 0x0000000001c58477 in sou2o ()
#23 0x00000000008472c2 in opimai_real ()
#24 0x0000000001c5e795 in ssthrdmain ()
#25 0x00000000008471b9 in main ()
23:09:13 SQL> /
INST_ID OPERATION STATE POWER SOFAR EST_WORK EST_RATE EST_MINUTES
---------- ---------- -------- ---------- ---------- ---------- ---------- -----------
2 REBAL WAIT 1
1 REBAL RUN 1 53989 53989 0 0
當剩余0min中的時候其實是進入了第三階段compact
同樣可以使用pstack看到使用了kfdCompact 函數
#pstack 85306
#0 0x00007fef99cdb644 in __io_getevents_0_4 () from /lib64/libaio.so.1
#1 0x0000000002baea09 in skgfrliopo ()
#2 0x0000000002bae801 in skgfospo ()
#3 0x00000000086ba843 in skgfrwat ()
#4 0x0000000008599e5d in ksfdwtio ()
#5 0x000000000220019b in ksfdwat_internal ()
#6 0x0000000003edce17 in kfk_reap_ufs_async_io ()
#7 0x0000000003edcd5f in kfk_reap_ios_from_subsys ()
#8 0x0000000000adefa4 in kfk_reap_ios ()
#9 0x0000000003edc382 in kfk_io1 ()
#10 0x0000000003edbf28 in kfkRequest ()
#11 0x0000000003ee272e in kfk_transitIO ()
#12 0x0000000003e35d5e in kffRelocateWait ()
#13 0x0000000003e5cbf6 in kffRelocate ()
#14 0x0000000003dd22df in kfdaExecute ()
#15 0x0000000003da1281 in kfdCompact ()
#16 0x0000000003da221a in kfdExecute ()
#17 0x0000000003eb5342 in kfgbRebalExecute ()
#18 0x0000000003ea2414 in kfgbDriver ()
#19 0x00000000021a86d7 in ksbabs ()
#20 0x0000000003eb964c in kfgbRun ()
#21 0x00000000021ad44b in ksbrdp ()
#22 0x00000000023d3eef in opirip ()
#23 0x000000000167e7b5 in opidrv ()
#24 0x0000000001c58477 in sou2o ()
#25 0x00000000008472c2 in opimai_real ()
#26 0x0000000001c5e795 in ssthrdmain ()
#27 0x00000000008471b9 in main ()
那么如何提升rebalance速度
影響rebalance速度因素:：
- Asm_power_limit
- Diskgroup redundancy
- Disks throughput
- Disks RAID
- Disk provider
- AU size
- How many disks are being added or dropped
- Failure groups
這里列出幾點不一一細說了：
1、不要過度rebalance
當使用alter diskgroup rebalance的時候，將會使asm承受很高的IO負載，所以我們應該量避免過度使用rebalance，能用1次操作的不要使用兩次。
15:42:29 SQL> alter diskgroup testdd add disk '/dev/qdata/mpath-s03.3264.01.P0B00S05' rebalance power 20 wait;
Diskgroup altered.
Elapsed: 00:00:26.82
15:42:50 SQL> alter diskgroup testdd add disk '/dev/qdata/mpath-s02.3264.01.P0B00S05' rebalance power 20 wait;
Diskgroup altered.
Elapsed: 00:00:25.97
15:41:32 SQL> alter diskgroup testdd add disk '/dev/qdata/mpath-s03.3264.01.P0B00S05', '/dev/qdata/mpath-s02.3264.01.P0B00S05' rebalance power 20 wait;
Diskgroup altered.
Elapsed: 00:00:16.77
2、power參數越大，rebalance越快
15:48:00 SQL> alter diskgroup testdd add disk '/dev/qdata/mpath-s03.3264.01.P0B00S05', '/dev/qdata/mpath-s02.3264.01.P0B00S05' rebalance power 1 wait;
Diskgroup altered.
Elapsed: 00:00:20.11
3、跳過compact階段：
設置_DISABLE_REBALANCE_COMPACT=TRUE，ALTER DISKGROUP SET ATTRIBUTE "_rebalance_compact”="FALSE";在設置該參數前還是建議咨詢SR，或者去問asm support guy Bane Radulovic。
rebalance新特性：
asm fast reblacne:
關閉數據庫，將asm實例restrict模式重啟，此時數據庫無法連接，進行rebalance會快速一點，等待rebalacne完畢以后重啟asm實例，似乎沒什么用。
asm fast mirror resync:
當一個磁盤損壞修復后還原磁盤組冗余度時，有時候會很耗時間。這種場景下，oracle asm fast mirror resync能夠有效的減少重同步時間。必須將兼容性設置11.1.0以上才能生效。fast mirror resync在這樣的場景下，會跟蹤磁盤offline期間extents的變化，等待online以后再進行重平衡。
這個參數默認為3.6h，通過設置disk_repaire_time來設置。在drop之前要確定沒有offline disk
12c版本中，可以通過V$ASM_OPERATION.pass字段觀察到resync階段。
SQL> SELECT GROUP_NUMBER, PASS, STATE FROM V$ASM_OPERATION;
GROUP_NUMBER PASS STAT
------------ --------- ----
1 RESYNC RUN
1 REBALANCE WAIT
1 COMPACT WAIT
2、power參數越大，rebalance越快