使用Maven將Hadoop2.2.0源碼編譯成Eclipse項目


編譯環境:

OS:RHEL 6.3 x64

Maven:3.2.1

Eclipse:Juno SR2 Linux x64

libprotoc:2.5.0

JDK:1.7.0_51 x64

步驟:

1. 下載Hadoop2.2.0源碼包  http://mirror.bit.edu.cn/apache/hadoop/common/hadoop-2.2.0/hadoop-2.2.0-src.tar.gz

 

2. 解壓縮到Eclipse的workspace中。這里最好解壓縮到workspace中,編譯完成后直接導入就好了,不用移動,移動的話容易造成依賴關系的確實,我比較懶,所以就放在workspace中編譯,省的還要build path。

 

3. 安裝Maven。Hadoop前期使用的是ant+ivy,后面改成了Maven,在源碼包的BUILD文件中有寫。下載Maven  http://apache.fayea.com/apache-mirror/maven/maven-3/3.2.1/binaries/apache-maven-3.2.1-bin.tar.gz,解壓縮到安裝目錄,在.bashrc中添加以下內容

export MAVEN_HOME=/root/Software/Maven321
export PATH=/root/Software/Maven321/bin:$PATH
在控制台中輸入mvn -version,打印以下信息則安裝成功
Apache Maven 3.2.1 (ea8b2b07643dbb1b84b6d16e1f08391b666bc1e9; 2014-02-15T01:37:52+08:00)
Maven home: /root/Software/maven321
Java version: 1.7.0_51, vendor: Oracle Corporation
Java home: /usr/java/jdk1.7.0_51/jre
Default locale: zh_CN, platform encoding: UTF-8
OS name: "linux", version: "2.6.32-431.5.1.el6.x86_64", arch: "amd64", family: "unix"

4. 安裝protobuf-2.5.0,首先安裝一下gcc

yum install gcc
yum install gcc-c++
然后下載protocbuf https://protobuf.googlecode.com/files/protobuf-2.5.0.tar.gz 下載完成后解壓到安裝目錄。進入安裝目錄執行如下命令進行安裝
./configure
 make
 make check
 make install
安裝完成后在控制台輸入protoc --version,有以下輸出則安裝成功。安裝protoc的原因是要用到它啦,並且官方文檔里貌似也沒提到,就是編譯的時候遇到錯誤了。並且這里是要編譯Eclipse項目,如果要編譯成可執行的Hadoop的話需要安裝更多的軟件進行支持,詳情可以見這里 http://my.oschina.net/cloudcoder/blog/192224
[root@dell ~]# protoc --version
libprotoc 2.5.0
 
        

5. 下面就可以開始編譯了,進入到Hadoop2.2.0的源碼包目錄里,ls查看一下,可以看到hadoop-maven-plugins文件夾,先進入到這個文件家,執行mvn install。過程有點長,如果顯示說有jar包下載不下來就多執行幾次,總有下載下來的一天。顯示BUILD SUCCESS后則返回到hadoop-2.2.0-src的根目錄下,執行mvn eclipse:eclipse –DskipTests,生成Eclipse項目。同樣,顯示BUILD SUCCESS后就是編譯成功了。

[root@dell hadoop-2.2.0-src]# ls
BUILDING.txt           hadoop-hdfs-project       hadoop-tools
dev-support            hadoop-mapreduce-project  hadoop-yarn-project
hadoop-assemblies      hadoop-maven-plugins      LICENSE.txt
hadoop-client          hadoop-minicluster        NOTICE.txt
hadoop-common-project  hadoop-project            pom.xml
hadoop-dist            hadoop-project-dist       README.txt
 
        

6. 將編譯好的項目導入到Eclipse中,依次執行[File] > [Import] > [Existing Projects into Workspace]即可。由於生成了很多個項目,所以導入后是這個樣子的。並且還會有一些錯誤,下面對如何修復錯誤寫一下。

image

Error#1. hadoop-streaming里面的build path有問題,顯示/root/workspace/hadoop-2.2.0-src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/conf(missing)

解決辦法,remove掉引用就好。

Error#2. hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientFailover.java中報sun.net.spi.nameservice.NameService錯誤,這是一個需要import的包,存在於openjdk中,在Oracle Jdk中沒找到,需要下載一個。NameService是一個接口,在網上找一個NameService放到該包中就好。 http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/7u40-b43/sun/net/spi/nameservice/NameService.java#NameService

Error#3. /hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineEditsViewer/XmlEditsVisitor.java里面顯示
import com.sun.org.apache.xml.internal.serialize.OutputFormat;
import com.sun.org.apache.xml.internal.serialize.XMLSerializer;
失敗,這是由於Eclipse的強檢查原則,打開Java -> Compiler -> Errors/Warnings and under "Deprecated and restricted API" change the setting of "Forbidden reference (access rules)" 將error級別調整到warning級別就好。

Error#4. /hadoop-common/src/test/java/org/apache/hadoop/io/serializer/avro/TestAvroSerialization.java顯示沒有AvroRecord類,在網上搜索到AvroRecord類放入到同級包中就行了。  http://grepcode.com/file/repo1.maven.org/maven2/org.apache.hadoop/hadoop-common/2.2.0/org/apache/hadoop/io/serializer/avro/AvroRecord.java#AvroRecord

Error#5. org.apache.hadoop.ipc.protobuf包是空的,需要在/hadoop-common/target/generated-sources/java中找到profobuf拷貝到/hadoop-common/src/test/java中就好了. 同時包里面還缺少了以下三個引用,在GrepCode上找一下,把hadoop-common2.2.0的相應文件下下來導入。

org.apache.hadoop.ipc.protobuf.TestProtos.EchoRequestProto;
org.apache.hadoop.ipc.protobuf.TestProtos.EchoResponseProto;
org.apache.hadoop.ipc.protobuf.TestRpcServiceProtos.TestProtobufRpcProto;

Error#6. /hadoop-auth/org/apache/hadoop/security/authentication/client/AuthenricatorTestCase.java中顯示server.start()和server.stop()錯誤,還沒找到原因所在,待檢查~~~


免責聲明!

本站轉載的文章為個人學習借鑒使用,本站對版權不負任何法律責任。如果侵犯了您的隱私權益,請聯系本站郵箱yoyou2525@163.com刪除。



 
粵ICP備18138465號   © 2018-2025 CODEPRJ.COM