2019-09-02 23:35:22
前言：首先，我想吐槽下自己，居然花費了4到5個夜晚和中午的時間來做這件事情，直到剛才才順利解決，我也挺佩服自己的！
我在這個過程中參考其他人的博客，非常感謝他們，寫博客是個很好的習慣，警戒自己也幫助他人！
正文：
1、你需要下載3種必備品：Hadoop、eclipse、Hadoop-eclipse-plus
2、Hadoop、eclipse都是直接從官網下載，如果說從官網下載慢的話，可以用迅雷下載，這樣快點。Hadoop-eclipse-plus需要從GitHub上搜索進行下載，最好下載Hadoop3.x-eclipse-plus
3.安裝好Hadoop、eclipse后，進入本片主要講解操作，編譯Hadoop-eclipse-plus-3.x.x。因為我都是下載最新的版本，所以Hadoop用的3.2.0版本，故在本文中我自己編譯的是Hadoop-eclipse-plus-3.2.0版本。
4.開始第一步解壓Hadoop3.x-eclipse-plus壓縮包到/home/hadoop/hadoop-3.2.0下，會形成eclipse-hadoop3x-master目錄，接下來我們所有的操作都是在這個目錄下進行的。
5.修改該目錄下/src/contrib/eclipse-plugin中的build.xml文件，代碼中出現add的地方就是我修改的地方
<!----------------------- part one---------------------------------------> <!-- Override classpath to include Eclipse SDK jars --> <path id="classpath"> <pathelement location="${build.classes}"/> <!--pathelement location="${hadoop.root}/build/classes"/--> <path refid="eclipse-sdk-jars"/> <path refid="hadoop-sdk-jars"/> <!-- new add start--> <fileset dir="${hadoop.root}"> <include name="**/*.jar" /> </fileset> <!-- new add end--> <!----------------------- part two---------------------------------------> <!--<target name="compile" depends="init, ivy-retrieve-common" unless="skip.contrib">--> <echo message="contrib: ${name}"/> <javac encoding="${build.encoding}" srcdir="${src.dir}" includes="**/*.java" destdir="${build.classes}" debug="${javac.debug}" deprecation="${javac.deprecation}" includeantruntime="on" <!-- new add --> > <classpath refid="classpath"/> </javac> </target> <!----------------------- part three---------------------------------------
注：這部分需要根據自己的需要添加，並且需要和已安裝的Hadoop里面的包版本對應一致，且要在libraries.properties中對的上> <copy file="${hadoop.home}/share/hadoop/common/lib/protobuf-java-${protobuf.version}.jar" todir="${build.dir}/lib" verbose="true"/> <copy file="${hadoop.home}/share/hadoop/common/lib/log4j-${log4j.version}.jar" todir="${build.dir}/lib" verbose="true"/> <copy file="${hadoop.home}/share/hadoop/common/lib/commons-cli-${commons-cli.version}.jar" todir="${build.dir}/lib" verbose="true"/> <copy file="${hadoop.home}/share/hadoop/common/lib/commons-configuration2-${commons-configuration2.version}.jar" todir="${build.dir}/lib" verbose="true"/> <copy file="${hadoop.home}/share/hadoop/common/lib/commons-lang3-${commons-lang3.version}.jar" todir="${build.dir}/lib" verbose="true"/> <copy file="${hadoop.home}/share/hadoop/common/lib/commons-collections-${commons-collections.version}.jar" todir="${build.dir}/lib" verbose="true"/> <copy file="${hadoop.home}/share/hadoop/common/lib/jackson-core-asl-${jackson.version}.jar" todir="${build.dir}/lib" verbose="true"/> <copy file="${hadoop.home}/share/hadoop/common/lib/jackson-mapper-asl-${jackson.version}.jar" todir="${build.dir}/lib" verbose="true"/> <copy file="${hadoop.home}/share/hadoop/common/lib/slf4j-log4j12-${slf4j-log4j12.version}.jar" todir="${build.dir}/lib" verbose="true"/> <copy file="${hadoop.home}/share/hadoop/common/lib/slf4j-api-${slf4j-api.version}.jar" todir="${build.dir}/lib" verbose="true"/> <copy file="${hadoop.home}/share/hadoop/common/lib/guava-${guava.version}.jar" todir="${build.dir}/lib" verbose="true"/> <copy file="${hadoop.home}/share/hadoop/common/lib/hadoop-auth-${hadoop.version}.jar" todir="${build.dir}/lib" verbose="true"/> <copy file="${hadoop.home}/share/hadoop/common/lib/netty-${netty.version}.jar" todir="${build.dir}/lib" verbose="true"/> <copy file="${hadoop.home}/share/hadoop/common/lib/javax.servlet-api-${servlet-api.version}.jar" todir="${build.dir}/lib" verbose="true"/> <copy file="${hadoop.home}/share/hadoop/common/lib/commons-io-${commons-io.version}.jar" todir="${build.dir}/lib" verbose="true"/> <copy file="${hadoop.home}/share/hadoop/common/lib/htrace-core4-${htrace.version}.jar" todir="${build.dir}/lib" verbose="true"/> <copy file="${hadoop.home}/share/hadoop/common/lib/woodstox-core-5.0.3.jar" todir="${build.dir}/lib" verbose="true"/> <copy file="${hadoop.home}/share/hadoop/common/lib/stax2-api-3.1.4.jar" todir="${build.dir}/lib" verbose="true"/> <jar jarfile="${build.dir}/hadoop-${name}-${hadoop.version}.jar" manifest="${root}/META-INF/MANIFEST.MF"> <manifest> <attribute name="Bundle-ClassPath" value="classes/, lib/hadoop-mapreduce-client-core-${hadoop.version}.jar, lib/hadoop-mapreduce-client-common-${hadoop.version}.jar, lib/hadoop-mapreduce-client-jobclient-${hadoop.version}.jar, lib/hadoop-auth-${hadoop.version}.jar, lib/hadoop-common-${hadoop.version}.jar, lib/hadoop-hdfs-${hadoop.version}.jar, lib/protobuf-java-${protobuf.version}.jar, lib/log4j-${log4j.version}.jar, lib/commons-cli-${commons-cli.version}.jar, lib/commons-configuration2-${commons-configuration2.version}.jar, lib/commons-httpclient-${commons-httpclient.version}.jar, lib/commons-lang3-${commons-lang3.version}.jar, lib/commons-collections-${commons-collections.version}.jar, lib/jackson-core-asl-${jackson.version}.jar, lib/jackson-mapper-asl-${jackson.version}.jar, lib/slf4j-log4j12-${slf4j-log4j12.version}.jar, lib/slf4j-api-${slf4j-api.version}.jar, lib/guava-${guava.version}.jar, lib/hadoop-auth-${hadoop.version}.jar, lib/netty-${netty.version}.jar, lib/javax.servlet-api-${servlet-api.version}.jar, lib/commons-io-${commons-io.version}.jar, lib/woodstox-core-5.0.3.jar, lib/stax2-api-3.1.4.jar"/>
這里需要注意幾個地方：
1.第三部分中加入的woodstox-core-5.0.3.jar和stax2-api-3.1.4.jar是為了解決“安裝完成后啟動Eclipse，點擊Add New Location沒有反應”的問題
2.第三部分加入的lib下的包必須和自己安裝Hadoop里面的版本一致，這里需要自己一個個去核對，具體在下一步操作中就會遇見，非常費時的一個操作
3.上面代碼中忘記加了，將depends="init, ivy-retrieve-common"去掉，我看別人是這樣解釋的“執行上面的命令的時候，會查找相應的jar包，若沒有的會一直停在那里，但實際上編譯Hadoop-eclipse-plugin並不需要common相關的包.”
<target name="compile" unless="skip.contrib"> <!--<target name="compile" depends="init, ivy-retrieve-common" unless="skip.contrib">-->
4.另外要注意build.xlm中涉及到Hadoop安裝目錄下/share/hadoop/common/lib/路徑時，要觀察與Hadoop的路徑是否一致，如果不是要修改成一樣的，不然無法正常引用jar包。
6、修改該目錄下/ivy中的libraries.properties文件，要求和Hadoop安裝目錄下share/hadoop中lib里面jar包的版本能對的上就行，遇見找不到的就不修改
以下是我修改后的代碼：
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#This properties file lists the versions of the various artifacts used by hadoop and components.
#It drives ivy and the generation of a maven POM
# This is the version of hadoop we are generating
hadoop.version=3.2.0
hadoop-gpl-compression.version=0.1.0
#These are the versions of our dependencies (in alphabetical order)
apacheant.version=1.7.0
ant-task.version=2.0.10
asm.version=5.0.4
aspectj.version=1.6.5
aspectj.version=1.6.11
checkstyle.version=4.2
commons-cli.version=1.2
commons-codec.version=1.11
commons-collections.version=3.2.2
commons-configuration2.version=2.1.1
commons-daemon.version=1.0.13
commons-httpclient.version=3.0.1
commons-lang3.version=3.7
commons-logging.version=1.1.13
commons-logging-api.version=1.1.13
commons-math.version=3.1.1
commons-el.version=1.0
commons-fileupload.version=1.2
commons-io.version=2.5
commons-net.version=3.6
core.version=3.1.1
coreplugin.version=1.3.2
hsqldb.version=2.3.4
ivy.version=2.1.0
jasper.version=5.5.12
jackson.version=1.9.13
#not able to figureout the version of jsp & jsp-api version to get it resolved throught ivy
# but still declared here as we are going to have a local copy from the lib folder
jsp.version=2.1
jsp-api.version=5.5.12
jsp-api-2.1.version=6.1.14
jsp-2.1.version=6.1.14
jets3t.version=0.6.1
jetty.version=6.1.26
jetty-util.version=6.1.26
jersey-core.version=1.19
jersey-json.version=1.19
jersey-server.version=1.19
junit.version=4.11
jdeb.version=0.8
jdiff.version=1.0.9
json.version=1.0
protobuf.version=2.5.0
kfs.version=0.1
netty.version=3.10.5.Final
log4j.version=1.2.17
lucene-core.version=2.3.1
htrace.version=4.1.0-incubating
mockito-all.version=1.8.5
jsch.version=0.1.45
oro.version=2.0.8
rats-lib.version=0.5.1
servlet.version=4.0.6
servlet-api.version=3.1.0
slf4j-api.version=1.7.25
slf4j-log4j12.version=1.7.25
guava.version=11.0.2
wagon-http.version=1.0-beta-2
xmlenc.version=0.52
xerces.version=1.4.4
7.執行編譯指令，具體可參考下載下來插件解壓后的eclipse-hadoop3x-master文件下的README.md參考說明書，代碼如下
[hadoop@hadoop01 ~]$ cd /home/hadoop/hadoop-3.2.0/eclipse-hadoop3x-master/src/contrib/eclipse-plugin
--要進入root權限進行編譯，不然會報錯
[hadoop@hadoop01 eclipse-plugin]$ su root
Password:
[root@hadoop01 eclipse-plugin]# ant jar -Dversion=3.2.0 -Dhadoop.version=3.2.0 -Declipse.home=/home/hadoop/eclipse -Dhadoop.home=/home/hadoop/hadoop-3.2.0
Buildfile: /home/hadoop/hadoop-3.2.0/eclipse-hadoop3x-master/src/contrib/eclipse-plugin/build.xml
compile:
[echo] contrib: eclipse-plugin
jar:
[jar] Building jar: /home/hadoop/hadoop-3.2.0/eclipse-hadoop3x-master/build/contrib/eclipse-plugin/hadoop-eclipse-plugin-3.2.0.jar
BUILD SUCCESSFUL
Total time: 5 seconds
注意幾點：一是先將README.md讀懂，知道如何去編譯Hadoop-eclipse-plus插件；而是需要使用root權限，不然會報錯，無法編譯成功，昨天其實我應該設置好所有的參數了，就是沒有使用root權限，所以沒有編譯成功。
吐槽下：其實自己編譯成功，結果eclipse的版本（我下載的是c++）下載錯誤，導致在eclipse中一直無法出現Hadoop Map/Reduce選項卡，要下載eclipse-jee-linux-gtk-x86_64.tar.gz版本。
8、將hadoop-eclipse-plugin-2.x.x.jar放到eclipse的plugins目錄下，啟動eclipse
9、打開window===>prefernces, 出現Hadoop Map/Reduce選項卡就說明編譯成功了
10、但是到上步就完事大吉了，你還需要設置Hadoop Map/Reduce的里面Hadoop的安裝地址，然后再進行后續操作才能確保編譯正確，不然要返回去修改參數，重新編譯。
【new error】好吧，發表這篇文章之后，這幾天我又陸續在整這個插件問題，因為我以為可以用了，實際也可以用了，但是在DFS location下的文件夾一直點不開，最開始報eclipse org/apache/htrace/core/Tracer$Builder這個錯誤，中間折騰了很久，不知道怎么在弄，最后報No FileSystem for scheme: hdfs這個錯誤，我一直以為是我編譯的插件有問題，其實不然。
然后在網上各種查找，終於在"https://blog.csdn.net/junior19/article/details/79594192"這篇文章中按照它的步驟找到了解決方法，反正現在是能看見文件夾了。哎。。。。。。。。。。。。
操作是在edit hadoop location 里面的advanced parameter的參數設置里面找到hadoop.tmp.dir將其內容填成與你之前配置hadoop文件的core-site.xml的要一致
其他出現的問題，請參考下面的地址，這都是我在編譯過程中參考的文章，再次感謝他們！
【文章1】https://www.cnblogs.com/ljy2013/articles/4417933.html
【文章2】https://www.cnblogs.com/zhangchao0515/p/7099002.html
【文章3】https://www.cnblogs.com/PurpleDream/p/4014751.html
【文章4】https://www.cnblogs.com/sissie-coding/p/9449941.html
附build.xml全代碼：
<?xml version="1.0" encoding="UTF-8" standalone="no"?> <!-- Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> <project default="jar" name="eclipse-plugin"> <import file="../build-contrib.xml"/> <path id="eclipse-sdk-jars"> <fileset dir="${eclipse.home}/plugins/"> <include name="org.eclipse.ui*.jar"/> <include name="org.eclipse.jdt*.jar"/> <include name="org.eclipse.core*.jar"/> <include name="org.eclipse.equinox*.jar"/> <include name="org.eclipse.debug*.jar"/> <include name="org.eclipse.osgi*.jar"/> <include name="org.eclipse.swt*.jar"/> <include name="org.eclipse.jface*.jar"/> <include name="org.eclipse.team.cvs.ssh2*.jar"/> <include name="com.jcraft.jsch*.jar"/> </fileset> </path> <path id="hadoop-sdk-jars"> <fileset dir="${hadoop.home}/share/hadoop/mapreduce"> <include name="hadoop*.jar"/> </fileset> <fileset dir="${hadoop.home}/share/hadoop/hdfs"> <include name="hadoop*.jar"/> </fileset> <fileset dir="${hadoop.home}/share/hadoop/common"> <include name="hadoop*.jar"/> </fileset> </path> <!-- Override classpath to include Eclipse SDK jars --> <path id="classpath"> <pathelement location="${build.classes}"/> <!--pathelement location="${hadoop.root}/build/classes"/--> <path refid="eclipse-sdk-jars"/> <path refid="hadoop-sdk-jars"/> <!-- new add --> <fileset dir="${hadoop.root}"> <include name="**/*.jar" /> </fileset> </path> <!-- Skip building if eclipse.home is unset. --> <target name="check-contrib" unless="eclipse.home"> <property name="skip.contrib" value="yes"/> <echo message="eclipse.home unset: skipping eclipse plugin"/> </target> <target name="compile" unless="skip.contrib"> <!--<target name="compile" depends="init, ivy-retrieve-common" unless="skip.contrib">--> <echo message="contrib: ${name}"/> <javac encoding="${build.encoding}" srcdir="${src.dir}" includes="**/*.java" destdir="${build.classes}" debug="${javac.debug}" deprecation="${javac.deprecation}" includeantruntime="on"> <classpath refid="classpath"/> </javac> </target> <!-- Override jar target to specify manifest --> <target name="jar" depends="compile" unless="skip.contrib"> <mkdir dir="${build.dir}/lib"/> <copy todir="${build.dir}/lib/" verbose="true"> <fileset dir="${hadoop.home}/share/hadoop/mapreduce"> <include name="hadoop*.jar"/> </fileset> </copy> <copy todir="${build.dir}/lib/" verbose="true"> <fileset dir="${hadoop.home}/share/hadoop/common"> <include name="hadoop*.jar"/> </fileset> </copy> <copy todir="${build.dir}/lib/" verbose="true"> <fileset dir="${hadoop.home}/share/hadoop/hdfs"> <include name="hadoop*.jar"/> </fileset> </copy> <copy todir="${build.dir}/lib/" verbose="true"> <fileset dir="${hadoop.home}/share/hadoop/yarn"> <include name="hadoop*.jar"/> </fileset> </copy> <copy todir="${build.dir}/classes" verbose="true"> <fileset dir="${root}/src/java"> <include name="*.xml"/> </fileset> </copy> <copy file="${hadoop.home}/share/hadoop/common/lib/protobuf-java-${protobuf.version}.jar" todir="${build.dir}/lib" verbose="true"/> <copy file="${hadoop.home}/share/hadoop/common/lib/log4j-${log4j.version}.jar" todir="${build.dir}/lib" verbose="true"/> <copy file="${hadoop.home}/share/hadoop/common/lib/commons-cli-${commons-cli.version}.jar" todir="${build.dir}/lib" verbose="true"/> <copy file="${hadoop.home}/share/hadoop/common/lib/commons-configuration2-${commons-configuration2.version}.jar" todir="${build.dir}/lib" verbose="true"/> <copy file="${hadoop.home}/share/hadoop/common/lib/commons-lang3-${commons-lang3.version}.jar" todir="${build.dir}/lib" verbose="true"/> <copy file="${hadoop.home}/share/hadoop/common/lib/commons-collections-${commons-collections.version}.jar" todir="${build.dir}/lib" verbose="true"/> <copy file="${hadoop.home}/share/hadoop/common/lib/jackson-core-asl-${jackson.version}.jar" todir="${build.dir}/lib" verbose="true"/> <copy file="${hadoop.home}/share/hadoop/common/lib/jackson-mapper-asl-${jackson.version}.jar" todir="${build.dir}/lib" verbose="true"/> <copy file="${hadoop.home}/share/hadoop/common/lib/slf4j-log4j12-${slf4j-log4j12.version}.jar" todir="${build.dir}/lib" verbose="true"/> <copy file="${hadoop.home}/share/hadoop/common/lib/slf4j-api-${slf4j-api.version}.jar" todir="${build.dir}/lib" verbose="true"/> <copy file="${hadoop.home}/share/hadoop/common/lib/guava-${guava.version}.jar" todir="${build.dir}/lib" verbose="true"/> <copy file="${hadoop.home}/share/hadoop/common/lib/hadoop-auth-${hadoop.version}.jar" todir="${build.dir}/lib" verbose="true"/> <copy file="${hadoop.home}/share/hadoop/common/lib/netty-${netty.version}.jar" todir="${build.dir}/lib" verbose="true"/> <copy file="${hadoop.home}/share/hadoop/common/lib/javax.servlet-api-${servlet-api.version}.jar" todir="${build.dir}/lib" verbose="true"/> <copy file="${hadoop.home}/share/hadoop/common/lib/commons-io-${commons-io.version}.jar" todir="${build.dir}/lib" verbose="true"/> <copy file="${hadoop.home}/share/hadoop/common/lib/htrace-core4-${htrace.version}.jar" todir="${build.dir}/lib" verbose="true"/> <copy file="${hadoop.home}/share/hadoop/common/lib/woodstox-core-5.0.3.jar" todir="${build.dir}/lib" verbose="true"/> <copy file="${hadoop.home}/share/hadoop/common/lib/stax2-api-3.1.4.jar" todir="${build.dir}/lib" verbose="true"/> <jar jarfile="${build.dir}/hadoop-${name}-${hadoop.version}.jar" manifest="${root}/META-INF/MANIFEST.MF"> <manifest> <attribute name="Bundle-ClassPath" value="classes/, lib/hadoop-mapreduce-client-core-${hadoop.version}.jar, lib/hadoop-mapreduce-client-common-${hadoop.version}.jar, lib/hadoop-mapreduce-client-jobclient-${hadoop.version}.jar, lib/hadoop-auth-${hadoop.version}.jar, lib/hadoop-common-${hadoop.version}.jar, lib/hadoop-hdfs-${hadoop.version}.jar, lib/protobuf-java-${protobuf.version}.jar, lib/log4j-${log4j.version}.jar, lib/commons-cli-${commons-cli.version}.jar, lib/commons-configuration2-${commons-configuration2.version}.jar, lib/commons-httpclient-${commons-httpclient.version}.jar, lib/commons-lang3-${commons-lang3.version}.jar, lib/commons-collections-${commons-collections.version}.jar, lib/jackson-core-asl-${jackson.version}.jar, lib/jackson-mapper-asl-${jackson.version}.jar, lib/slf4j-log4j12-${slf4j-log4j12.version}.jar, lib/slf4j-api-${slf4j-api.version}.jar, lib/guava-${guava.version}.jar, lib/hadoop-auth-${hadoop.version}.jar, lib/netty-${netty.version}.jar, lib/javax.servlet-api-${servlet-api.version}.jar, lib/commons-io-${commons-io.version}.jar, lib/woodstox-core-5.0.3.jar, lib/stax2-api-3.1.4.jar"/> </manifest> <fileset dir="${build.dir}" includes="classes/ lib/"/> <!--fileset dir="${build.dir}" includes="*.xml"/--> <fileset dir="${root}" includes="resources/ plugin.xml"/> </jar> </target> </project>
附README.md
hadoop3x-eclipse-plugin ======================= eclipse plugin for hadoop 3.x.x How to build ---------------------------------------- [hdpusr@demo hadoop2x-eclipse-plugin]$ cd src/contrib/eclipse-plugin # Assume hadoop installation directory is /usr/share/hadoop [hdpusr@apclt eclipse-plugin]$ ant jar -Dversion=3.1.0 -Dhadoop.version=3.1.0 -Declipse.home=/opt/eclipse -Dhadoop.home=/usr/share/hadoop final jar will be generated at directory ${hadoop2x-eclipse-plugin}/build/contrib/eclipse-plugin/hadoop-eclipse-plugin-2.4.1.jar release version included ------------------------------------- release/hadoop-eclipse-kepler-plugin-2.4.1.jar # not tested yet release/hadoop-eclipse-kepler-plugin-2.2.0.jar options required -------------------------------------- version: plugin version hadoop.version: hadoop version you want to compiled with eclipse.home: path of eclipse home hadoop.home: path of hadoop 3.x home How to debug -------------------------------------- start eclipse with debug parameter: /opt/eclipse/eclipse -clean -consolelog -debug Note: compile issues resolve: -------------------------------------- 1. For different hadoop, adjust ${hadoop2x-eclipse-plugin-master}/ivy/libraries.properties, to match hadoop dependency lib version. 2. modify ${hadoop3x-eclipse}/src/contrib/eclipse-plugin/build.xml, in the node: <attribute name="Bundle-ClassPath" .... to add the jar needed. 3. For eclipse, do **not** use eclipse-inst, install eclipse **FULL VERSION(SDK)** instead.