http://www.chenshake.com/megaraid-manager-centos/
對於OpenStack來說，存儲基本都是Ceph，那么磁盤一般都會做成raid0或者使用直通模式。對於直通模式，SATA盤的性能，會相對差不少，通過Raid卡，由於有raid卡的緩存，raid0，磁盤的性能會好很多。那么通過Raid卡，設置Raid0，一個問題就是，換盤，機器必須重啟設置raid，能不能在運行的機器，把磁盤的raid 重建一下呢？
我在一台3年前的廣達的四字星服務器上做驗證。
我的raid卡是：LSI MegaRAID SAS PCI Express ROMB
Contents [hide]
確認Raid卡
# dmesg | grep -i raid [ 2.981279] megaraid_sas 0000:06:00.0: FW now in Ready state [ 2.981330] megaraid_sas 0000:06:00.0: irq 36 for MSI/MSI-X [ 2.981532] megaraid_sas 0000:06:00.0: irq 51 for MSI/MSI-X [ 2.981585] megaraid_sas 0000:06:00.0: firmware supports msix : (16) [ 2.981587] megaraid_sas 0000:06:00.0: current msix/online cpus : (16/24) [ 2.981589] megaraid_sas 0000:06:00.0: RDPQ mode : (disabled) [ 2.982991] megaraid_sas 0000:06:00.0: FW supports sync cache : No [ 3.003325] megaraid_sas 0000:06:00.0: Init cmd success [ 3.025320] megaraid_sas 0000:06:00.0: firmware type : Legacy(64 VD) firmware [ 3.025323] megaraid_sas 0000:06:00.0: controller type : MR(1024MB) [ 3.025324] megaraid_sas 0000:06:00.0: Online Controller Reset(OCR) : Enabled [ 3.025325] megaraid_sas 0000:06:00.0: Secure JBOD support : No [ 3.125498] scsi host0: Avago SAS based MegaRAID driver
我的理解市場上99%服務器，基本都會是LSI的raid卡，都可以通過mega raid的工具進行管理。
安裝megacli
https://www.broadcom.com/support/download-search
搜索框里搜索 megacli
一個zip文件，上傳到linux下，解壓，進入linux目錄下，就一個rpm包
rpm -ivh MegaCli-8.07.14-1.noarch.rpm
查看
現在就根據文檔，一步一步進行測試
查看硬盤信息
/opt/MegaRAID/MegaCli/MegaCli64 -PDList -aALL
信息量很多。
可以通過命令過濾一下
/opt/MegaRAID/MegaCli/./MegaCli64 -PDList -aALL \ | egrep 'Adapter|Enclosure Device ID|Coerced Size|Drive Temperature|PD Type|Slot|Inquiry'
這樣輸出有效信息
Adapter #0 Enclosure Device ID: 252 Slot Number: 0 PD Type: SATA Non Coerced Size: 446.630 GB [0x37d436b0 Sectors] Coerced Size: 446.625 GB [0x37d40000 Sectors] Inquiry Data: BTWL502105DX480QGN INTEL SSDSC2BB480G4 D2010370 Drive Temperature : N/A Enclosure Device ID: 252 Slot Number: 1 PD Type: SATA Non Coerced Size: 446.630 GB [0x37d436b0 Sectors] Coerced Size: 446.625 GB [0x37d40000 Sectors] Inquiry Data: BTWL502201GP480QGN INTEL SSDSC2BB480G4 D2010370 Drive Temperature : N/A Enclosure Device ID: 252 Slot Number: 2 PD Type: SATA Non Coerced Size: 931.012 GB [0x74606db0 Sectors] Coerced Size: 931.0 GB [0x74600000 Sectors] Inquiry Data: 9XG6PQX4ST91000640NS SN03 Drive Temperature :25C (77.00 F) Enclosure Device ID: 252 Slot Number: 3 PD Type: SATA Non Coerced Size: 931.012 GB [0x74606db0 Sectors] Coerced Size: 931.0 GB [0x74600000 Sectors] Inquiry Data: 9XG2XPCBST91000640NS SN03 Drive Temperature :24C (75.20 F) Enclosure Device ID: 252 Slot Number: 4 PD Type: SATA Non Coerced Size: 931.012 GB [0x74606db0 Sectors] Coerced Size: 931.0 GB [0x74600000 Sectors] Inquiry Data: 9XG6Q1F6ST91000640NS SN03 Drive Temperature :24C (75.20 F) Enclosure Device ID: 252 Slot Number: 5 PD Type: SATA Non Coerced Size: 931.012 GB [0x74606db0 Sectors] Coerced Size: 931.0 GB [0x74600000 Sectors] Inquiry Data: 9XG5ARPZST91000640NS SN03 Drive Temperature :25C (77.00 F)
我的理解
- Adapter0：第一塊卡的意思
- Enclosure Device ID：252 這個參數很重要，設置raid要用到
- slot neumber :0，硬盤插入的位置
- PD type：接口是SATA接口，不是SAS接口
- core size：容量
- inquiry data：硬盤產商，可以看出是intel出的ssd
- Drive temperature：磁盤溫度，SATA盤是有溫度，ssd沒有
查看單盤的詳細信息
/opt/MegaRAID/MegaCli/MegaCli64 -pdInfo -PhysDrv[252:3] -aALL
252，就是Enclosure Device ID
3：就是slot neumber
查看邏輯盤詳細信息
/opt/MegaRAID/MegaCli/MegaCli64 -LdPdInfo -aALL
查看raid卡全部信息
# /opt/MegaRAID/MegaCli/MegaCli64 -AdpAllInfo -aAll
基本上raid卡的功能，都是可以通過這個命令查看。
你所使用的raid卡型號是：LSI MegaRAID SAS PCI Express ROMB
另外默認 JBOD是關閉的，底下有介紹，可以通過命令打開這個功能。
Raid配置過程
點亮指定硬盤（定位）
/opt/MegaRAID/MegaCli/MegaCli64 -PdLocate -start -physdrv[252:5] -a0
一排綠色的硬盤燈，有一個變成紅色
start改成stop，就會恢復綠色。很實用。
拔出亮紅燈磁盤，再次插入
# /opt/MegaRAID/MegaCli/MegaCli64 -PDList -aALL -Nolog|grep '^Firm' Firmware state: Online, Spun Up Firmware state: Online, Spun Up Firmware state: Online, Spun Up Firmware state: Online, Spun Up Firmware state: Online, Spun Up Firmware state: Unconfigured(bad)
修改磁盤狀態
# /opt/MegaRAID/MegaCli/MegaCli64 -PDMakeGood -PhysDrv[252:5]-a0
Adapter: 0: EnclId-252 SlotId-5 state changed to Unconfigured-Good.
查看狀態
]# /opt/MegaRAID/MegaCli/MegaCli64 -PDList -aALL -Nolog|grep '^Firm' Firmware state: Online, Spun Up Firmware state: Online, Spun Up Firmware state: Online, Spun Up Firmware state: Online, Spun Up Firmware state: Online, Spun Up Firmware state: Unconfigured(good), Spun Up
Foreign
掃描 Foreign
/opt/MegaRAID/MegaCli/MegaCli64 -cfgforeign -scan -a0
There are 1 foreign configuration(s) on controller 0.
Exit Code: 0x00
發現有一個 Foreign
清除Foreign
/opt/MegaRAID/MegaCli/MegaCli64 -cfgforeign -clear –a0
Foreign configuration 0 is cleared on controller 0. Exit Code: 0x00
再次掃描
/opt/MegaRAID/MegaCli/MegaCli64 -cfgforeign -scan –a0
There is no foreign configuration on controller 0. Exit Code: 0x00
創建raid0
# /opt/MegaRAID/MegaCli/MegaCli64 -CfgLdAdd -r0[252:5] WT Direct -a0
Adapter 0: Created VD 5
Adapter 0: Configured the Adapter!!
Exit Code: 0x00
252，就是raid卡里的 Enclosure Device ID
5：Slot Number
a0：表示Adapter #0
JBOD模式
啟用raid卡的JBOD模式
# /opt/MegaRAID/MegaCli/MegaCli64 -AdpSetProp -EnableJBOD -1 -aALL
Adapter 0: Set JBOD to Enable success.
查看raid卡的功能的時候，就會發現
刪掉 raid0
# /opt/MegaRAID/MegaCli/MegaCli64 -CfgLdDel -L5 -a0
Adapter 0: Deleted Virtual Drive-5(target id-5)
磁盤狀態
# /opt/MegaRAID/MegaCli/MegaCli64 -PDList -aALL -Nolog|grep '^Firm' Firmware state: Online, Spun Up Firmware state: Online, Spun Up Firmware state: Online, Spun Up Firmware state: Online, Spun Up Firmware state: Online, Spun Up Firmware state: Unconfigured(good), Spun Up
JBOD模式
# /opt/MegaRAID/MegaCli/MegaCli64 -PDMakeJBOD -PhysDrv[252:5] -a0
Adapter: 0: EnclId-252 SlotId-5 state changed to JBOD.
再查看磁盤狀態
# /opt/MegaRAID/MegaCli/MegaCli64 -PDList -aALL -Nolog|grep '^Firm' Firmware state: Online, Spun Up Firmware state: Online, Spun Up Firmware state: Online, Spun Up Firmware state: Online, Spun Up Firmware state: Online, Spun Up Firmware state: JBOD
參考文章
http://blog.csdn.net/msdnchina/article/details/52475383
http://blog.sina.com.cn/s/blog_57c70e190101ebl9.html
http://www.tuicool.com/articles/nmU3i2f
腳本
/opt/MegaRAID/MegaCli/MegaCli64 -LdPdInfo -aAll -NoLog|grep -Ei '(^Virtual Disk|^RAID Level|^PD type|^Raw Size|^Enclosure|^Slot|error|firmware)' | awk '{if($0~/^Virtual/||$0~/^RAID/){printf("\033[35m%s\033[0m\n",$0)}else if($0 ~ /^Enclosure/){printf("\033[31m%s: %s\033[0m ",$1,$4)}else if($0 ~ /^Slot/){printf("\033[31m%s\033[0m\n",$0)}else if($0~/^Other/||$0~/Firmware/){printf("\033[33m%s\033[0m\n",$0)}else if($0~/^Raw/){printf("\033[33m%s%s\033[0m\n",$2,$3)}else{printf("\033[33m%s\033[0m ",$0)}}'
會顯示所有硬盤raid的情況
RAID Level : Primary-0, Secondary-0, RAID Level Qualifier-0 Enclosure: 252 Slot Number: 0 Enclosure: Media Error Count: 0 Other Error Count: 0 PD Type: SATA Size:447.130 Firmware state: Online, Spun Up Device Firmware Level: 0370 RAID Level : Primary-0, Secondary-0, RAID Level Qualifier-0 Enclosure: 252 Slot Number: 1 Enclosure: Media Error Count: 0 Other Error Count: 0 PD Type: SATA Size:447.130 Firmware state: Online, Spun Up Device Firmware Level: 0370 RAID Level : Primary-0, Secondary-0, RAID Level Qualifier-0 Enclosure: 252 Slot Number: 2 Enclosure: Media Error Count: 0 Other Error Count: 0 PD Type: SATA Size:931.512 Firmware state: Online, Spun Up Device Firmware Level: SN03 RAID Level : Primary-0, Secondary-0, RAID Level Qualifier-0 Enclosure: 252 Slot Number: 3 Enclosure: Media Error Count: 0 Other Error Count: 0 PD Type: SATA Size:931.512 Firmware state: Online, Spun Up Device Firmware Level: SN03 RAID Level : Primary-0, Secondary-0, RAID Level Qualifier-0 Enclosure: 252 Slot Number: 4 Enclosure: Media Error Count: 0 Other Error Count: 0 PD Type: SATA Size:931.512 Firmware state: Online, Spun Up Device Firmware Level: SN03