1、hdfs web界面出現文件block丟失
2、block丟失的時候報的什么錯？
hadoop fs -get /user/bizlogcenter/warehouse/eunomia/160/warnning/2019-02-26/10.149.32.154.log
19/04/12 21:06:27 WARN hdfs.DFSClient: Found Checksum error for BP-1472871384-10.143.46.220-1508649897332:blk_1074936834_1198570811 from 10.143.57.89:1004 at 3123200
19/04/12 21:06:27 WARN hdfs.DFSClient: Found Checksum error for BP-1472871384-10.143.46.220-1508649897332:blk_1074936834_1198570811 from 10.143.59.46:1004 at 3123200
19/04/12 21:06:27 INFO hdfs.DFSClient: Could not obtain BP-1472871384-10.143.46.220-1508649897332:blk_1074936834_1198570811 from any node: java.io.IOException: No live nodes contain current block No live nodes contain current block Block locations: 10.143.57.89:1004 10.143.59.46:1004 Dead nodes: 10.143.57.89:1004 10.143.59.46:1004. Will get new block locations from namenode and retry...
19/04/12 21:06:27 WARN hdfs.DFSClient: DFS chooseDataNode: got # 1 IOException, will wait for 801.7790779945808 msec.
get: Checksum error: /user/bizlogcenter/warehouse/eunomia/160/warnning/2019-02-26/10.149.32.154.log at 3123200 exp: 584468584 got: -486844696
3、為什么會報block丟失呢？
參考官方jira：https://issues.apache.org/jira/browse/HDFS-6804
原因：
After checking the code of Datanode block transferring, I found some race condition during transferring the block to the other datanode. And the race condition causes the source datanode transfers the wrong checksum of the last chunk in replica.
Here is the root cause.
-
Datanode DN1 receives transfer block command from NameNode, say, the command needs DN1 to transfer block B1 to DataNode DN2.
-
DN1 creates a new DataTransfer thread, which is responsible for transferring B1 to DN2.
-
When DataTransfer thread is created, the replica of B1 is in Finalized state. Then, DataTransfer reads replica content and checksum directly from disk, sends them to DN2.
-
During DataTransfer is sending data to DN2. The block B1 is opened for appending. If the last data chunk of B1 is not full, the last checksum will be overwritten by the BlockReceiver thread.
-
In DataTransfer thread, it records the block length as the length before appending. Then, here comes the problem. When DataTransfer thread sends the last data chunk to ND2, it reads the checksum of the last chunk from the disk and sends the checksum too. But at this time, the last checksum is changed, because some more data is appended in the last data chunk.
-
When DN2 receives the last data chunk and checksum, it will throw the checksum mismatch exception.
解決辦法：
Some thoughts about how to fix this issue. In my mind, there are 2 ways to fix this.
-
Option1
When the block is opened for appending, check if there are some DataTransfer threads which are transferring block to other DNs. Stop these DataTransferring threads.
We can stop these threads because the generation timestamp of the block is increased because it is opened for appending. So, the DataTransfer threads are sending outdated blocks.
-
Option2
In DataTransfer thread, if the replica of the block is finalized, the DataTransfer thread can read the last data chunk checksum into the memory, record the replica length in memory too. Then, when sending the last data chunk, use the checksum in memory instead of reading it from the disk.
This is similar to what we deal with a RBW replica in DataTransfer.
For Option1, it is hard to stop the DataTransfer thread unless we add some code in DataNode to manage DataTransfer threads.
For Option2, we should lock FsDatasetImpl object in DataNode when reading the last chunk checksum from disk. Otherwise, the last block might be overwritten. But reading from the disk needs time, putting the expensive disk IO operations during locking FsDatasetImpl might cause some performance downgrade in DataNodes.
需要升級Hadoop版本：
4、有一個replica是正常的為什么namenode不能自動修復？
雖然有一個replica是正常的，但是當remote transfer異常之后，異常的datanode向namenode匯報這個錯誤，namenode面對這種報錯的處理方式是，認為所有副本都壞了，盡管source replica上的block是正常的
5、怎么恢復missing的文件呢？
其實文件missing是namenode自己對iNode進行標記的，但是還是有block塊是正常的，是因為append操作導致了block塊的checksum有問題，但是讀能正常讀取，所有我們可以get或者cat hdfs上的文件到本地，然后通過fsck -delete刪除文件，最后put文件來修復該missing文件
6、刪除missing file后，block會自動刪除？
DataNode上的missing block，之前的兩副本，其中一個是正常的，另一個是當時remote transfer異常的，所以刪除該文件之后，正常的block file能刪除，異常的不能刪除，需手動刪除
dataNode有個directoryScanner應該會自動處理，不過貌似啟動的間隔周期很長，6個小時