hadoop fsck詳解


我們知道fsck是用來檢測hdfs上文件、block信息的,但是fsck輸出的結果我們是否能看明白呢?
 
下面我們來看一個fsck輸出的結果
hadoop fsck /
########################## 情況一 ####################
Status: HEALTHY
Total size: 3107919020687 B
Total dirs: 142
Total files: 321
Total symlinks: 0
Total blocks (validated): 11738 (avg. block size 264774154 B)
Minimally replicated blocks: 11738 (100.0 %)
Over-replicated blocks: 3605 (30.712217 %)
Under-replicated blocks: 0 (0.0 %)
Mis-replicated blocks: 8011 (68.24842 %)
Default replication factor: 3
Average block replication: 3.3083148
Corrupt blocks: 0
Missing replicas: 0 (0.0 %)
Number of data-nodes: 11
Number of racks: 2
FSCK ended at Fri Nov 10 15:11:47 CST 2017 in 418 milliseconds
 
 
The filesystem under path '/' is HEALTHY
 
注:
這種情況是我們在原來的機架上擴增了一個不同機房的機架
由於hadoop對擴增機房的數據平衡策略是:
1)先拷貝一份數據到新增機架的機器上,然后再在原來機架上刪除一份數據
2)所以這里的 Over-replicated blocks會顯示,是3605 。代表的是集群新增了3605 副本數,超過了默認的副本數
 
########################## 情況二 ####################
.....................Status: HEALTHY
Total size: 3130802412834 B
Total dirs: 143
Total files: 321
Total symlinks: 0 (Files currently being written: 1)
Total blocks (validated): 11824 (avg. block size 264783695 B)
Minimally replicated blocks: 11824 (100.0 %)
Over-replicated blocks: 0 (0.0 %)
Under-replicated blocks: 755 (6.385318 %)
Mis-replicated blocks: 0 (0.0 %)
Default replication factor: 3
Average block replication: 2.937331
Corrupt blocks: 0
Missing replicas: 755 (2.1275997 %)
Number of data-nodes: 10
Number of racks: 1
FSCK ended at Mon Nov 13 16:59:13 CST 2017 in 69 milliseconds
 
注:
當新增機房的slave節點和之前的slave節點數據重新平衡后,我把新增機房的節點網絡中斷,
然后后就出現了 Under-replicated blocks,意思就是集群中有這么多副本數是小於集群指定的副本數。
 
 
 
########################## 情況三 ####################
Status: HEALTHY
Total size: 3130802412834 B
Total dirs: 143
Total files: 322
Total symlinks: 0
Total blocks (validated): 11824 (avg. block size 264783695 B)
Minimally replicated blocks: 11824 (100.0 %)
Over-replicated blocks: 0 (0.0 %)
Under-replicated blocks: 0 (0.0 %)
Mis-replicated blocks: 0 (0.0 %)
Default replication factor: 3
Average block replication: 3.001184
Corrupt blocks: 0
Missing replicas: 0 (0.0 %)
Number of data-nodes: 14
Number of racks: 2
FSCK ended at Mon Nov 13 11:00:37 CST 2017 in 642 milliseconds
 
注:
這里的是最終達到平衡后的檢測結果
 
參數解釋:
status:代表這次hdfs上block檢測的結果
Total size: 代表/目錄下文件總大小
Total dirs:代表檢測的目錄下總共有多少個目錄
Total files:代表檢測的目錄下總共有多少文件
Total symlinks:代表檢測的目錄下有多少個符號連接
Total blocks(validated):代表檢測的目錄下有多少個block塊是有效的
Minimally replicated blocks:代表拷貝的最小block塊數
Over-replicated blocks:指的是副本數大於指定副本數的block數量
Under-replicated blocks:指的是副本數小於指定副本數的block數量
Mis-replicated blocks:指丟失的block塊數量
Default replication factor: 3 指默認的副本數是3份( 自身一份,需要拷貝兩份
Missing replicas:丟失的副本數
Number of data-nodes:有多少個節點
Number of racks:有多少個機架


免責聲明!

本站轉載的文章為個人學習借鑒使用,本站對版權不負任何法律責任。如果侵犯了您的隱私權益,請聯系本站郵箱yoyou2525@163.com刪除。



 
粵ICP備18138465號   © 2018-2025 CODEPRJ.COM