以hadoop 2.7.3為例, jdk1.7.0_80 , Eclipse Mars(4.5), Apache Maven 3.0.5 mvn時需要使用root權限,即縣切換到root用戶,用命令su root.
本節將介紹如何創建一個Hadoop源代碼工程以方便閱讀源代碼. Hadoop源代碼hadoop-2.7.3-src不能直接導入Eclipse,會報很多錯,因為該源代碼工程沒有包含它所需的所有JAR包等, 需要Maven編譯引入.
首先,介紹一下我自己成功的辦法. ( 后面會介紹書上的辦法,不過書上的辦法會出錯,大家僅做參考)
我們可以參考hadoop2.7.3源碼編譯eclipse項目
還可以看hadoop-2.7.3-src下的BUILDING.txt文件, 該文件教你如何構建Hadoop項目.如下所示:
Build instructions for Hadoop ---------------------------------------------------------------------------------- Requirements: * Unix System * JDK 1.7+ * Maven 3.0 or later * Findbugs 1.3.9 (if running findbugs) * ProtocolBuffer 2.5.0 * CMake 2.6 or newer (if compiling native code), must be 3.0 or newer on Mac * Zlib devel (if compiling native code) * openssl devel ( if compiling native hadoop-pipes and to get the best HDFS encryption performance ) * Linux FUSE (Filesystem in Userspace) version 2.6 or above ( if compiling fuse_dfs ) * Internet connection for first build (to fetch all Maven and Hadoop dependencies) ---------------------------------------------------------------------------------- Installing required packages for clean install of Ubuntu 14.04 LTS Desktop: * Oracle JDK 1.7 (preferred) $ sudo apt-get purge openjdk* $ sudo apt-get install software-properties-common $ sudo add-apt-repository ppa:webupd8team/java $ sudo apt-get update $ sudo apt-get install oracle-java7-installer * Maven $ sudo apt-get -y install maven * Native libraries $ sudo apt-get -y install build-essential autoconf automake libtool cmake zlib1g-dev pkg-config libssl-dev * ProtocolBuffer 2.5.0 (required) $ sudo apt-get -y install libprotobuf-dev protobuf-compiler Optional packages: * Snappy compression $ sudo apt-get install snappy libsnappy-dev * Bzip2 $ sudo apt-get install bzip2 libbz2-dev * Jansson (C Library for JSON) $ sudo apt-get install libjansson-dev * Linux FUSE $ sudo apt-get install fuse libfuse-dev ---------------------------------------------------------------------------------- Maven main modules: hadoop (Main Hadoop project) - hadoop-project (Parent POM for all Hadoop Maven modules. ) (All plugins & dependencies versions are defined here.) - hadoop-project-dist (Parent POM for modules that generate distributions.) - hadoop-annotations (Generates the Hadoop doclet used to generated the Javadocs) - hadoop-assemblies (Maven assemblies used by the different modules) - hadoop-common-project (Hadoop Common) - hadoop-hdfs-project (Hadoop HDFS) - hadoop-mapreduce-project (Hadoop MapReduce) - hadoop-tools (Hadoop tools like Streaming, Distcp, etc.) - hadoop-dist (Hadoop distribution assembler) ---------------------------------------------------------------------------------- Where to run Maven from? It can be run from any module. The only catch is that if not run from utrunk all modules that are not part of the build run must be installed in the local Maven cache or available in a Maven repository. ---------------------------------------------------------------------------------- Maven build goals: * Clean : mvn clean * Compile : mvn compile [-Pnative] * Run tests : mvn test [-Pnative] * Create JAR : mvn package * Run findbugs : mvn compile findbugs:findbugs * Run checkstyle : mvn compile checkstyle:checkstyle * Install JAR in M2 cache : mvn install * Deploy JAR to Maven repo : mvn deploy * Run clover : mvn test -Pclover [-DcloverLicenseLocation=${user.name}/.clover.license] * Run Rat : mvn apache-rat:check * Build javadocs : mvn javadoc:javadoc * Build distribution : mvn package [-Pdist][-Pdocs][-Psrc][-Pnative][-Dtar] * Change Hadoop version : mvn versions:set -DnewVersion=NEWVERSION Build options: * Use -Pnative to compile/bundle native code * Use -Pdocs to generate & bundle the documentation in the distribution (using -Pdist) * Use -Psrc to create a project source TAR.GZ * Use -Dtar to create a TAR with the distribution (using -Pdist) Snappy build options: Snappy is a compression library that can be utilized by the native code. It is currently an optional component, meaning that Hadoop can be built with or without this dependency. * Use -Drequire.snappy to fail the build if libsnappy.so is not found. If this option is not specified and the snappy library is missing, we silently build a version of libhadoop.so that cannot make use of snappy. This option is recommended if you plan on making use of snappy and want to get more repeatable builds. * Use -Dsnappy.prefix to specify a nonstandard location for the libsnappy header files and library files. You do not need this option if you have installed snappy using a package manager. * Use -Dsnappy.lib to specify a nonstandard location for the libsnappy library files. Similarly to snappy.prefix, you do not need this option if you have installed snappy using a package manager. * Use -Dbundle.snappy to copy the contents of the snappy.lib directory into the final tar file. This option requires that -Dsnappy.lib is also given, and it ignores the -Dsnappy.prefix option. OpenSSL build options: OpenSSL includes a crypto library that can be utilized by the native code. It is currently an optional component, meaning that Hadoop can be built with or without this dependency. * Use -Drequire.openssl to fail the build if libcrypto.so is not found. If this option is not specified and the openssl library is missing, we silently build a version of libhadoop.so that cannot make use of openssl. This option is recommended if you plan on making use of openssl and want to get more repeatable builds. * Use -Dopenssl.prefix to specify a nonstandard location for the libcrypto header files and library files. You do not need this option if you have installed openssl using a package manager. * Use -Dopenssl.lib to specify a nonstandard location for the libcrypto library files. Similarly to openssl.prefix, you do not need this option if you have installed openssl using a package manager. * Use -Dbundle.openssl to copy the contents of the openssl.lib directory into the final tar file. This option requires that -Dopenssl.lib is also given, and it ignores the -Dopenssl.prefix option. Tests options: * Use -DskipTests to skip tests when running the following Maven goals: 'package', 'install', 'deploy' or 'verify' * -Dtest=<TESTCLASSNAME>,<TESTCLASSNAME#METHODNAME>,.... * -Dtest.exclude=<TESTCLASSNAME> * -Dtest.exclude.pattern=**/<TESTCLASSNAME1>.java,**/<TESTCLASSNAME2>.java ---------------------------------------------------------------------------------- Building components separately If you are building a submodule directory, all the hadoop dependencies this submodule has will be resolved as all other 3rd party dependencies. This is, from the Maven cache or from a Maven repository (if not available in the cache or the SNAPSHOT 'timed out'). An alternative is to run 'mvn install -DskipTests' from Hadoop source top level once; and then work from the submodule. Keep in mind that SNAPSHOTs time out after a while, using the Maven '-nsu' will stop Maven from trying to update SNAPSHOTs from external repos. ---------------------------------------------------------------------------------- Protocol Buffer compiler The version of Protocol Buffer compiler, protoc, must match the version of the protobuf JAR. If you have multiple versions of protoc in your system, you can set in your build shell the HADOOP_PROTOC_PATH environment variable to point to the one you want to use for the Hadoop build. If you don't define this environment variable, protoc is looked up in the PATH. ---------------------------------------------------------------------------------- Importing projects to eclipse When you import the project to eclipse, install hadoop-maven-plugins at first. $ cd hadoop-maven-plugins $ mvn install Then, generate eclipse project files. $ mvn eclipse:eclipse -DskipTests At last, import to eclipse by specifying the root directory of the project via [File] > [Import] > [Existing Projects into Workspace]. ---------------------------------------------------------------------------------- Building distributions: Create binary distribution without native code and without documentation: $ mvn package -Pdist -DskipTests -Dtar Create binary distribution with native code and with documentation: $ mvn package -Pdist,native,docs -DskipTests -Dtar Create source distribution: $ mvn package -Psrc -DskipTests Create source and binary distributions with native code and documentation: $ mvn package -Pdist,native,docs,src -DskipTests -Dtar Create a local staging version of the website (in /tmp/hadoop-site) $ mvn clean site; mvn site:stage -DstagingDirectory=/tmp/hadoop-site ---------------------------------------------------------------------------------- Installing Hadoop Look for these HTML files after you build the document by the above commands. * Single Node Setup: hadoop-project-dist/hadoop-common/SingleCluster.html * Cluster Setup: hadoop-project-dist/hadoop-common/ClusterSetup.html ---------------------------------------------------------------------------------- Handling out of memory errors in builds ---------------------------------------------------------------------------------- If the build process fails with an out of memory error, you should be able to fix it by increasing the memory used by maven -which can be done via the environment variable MAVEN_OPTS. Here is an example setting to allocate between 256 and 512 MB of heap space to Maven export MAVEN_OPTS="-Xms256m -Xmx512m" ---------------------------------------------------------------------------------- Building on Windows ---------------------------------------------------------------------------------- Requirements: * Windows System * JDK 1.7+ * Maven 3.0 or later * Findbugs 1.3.9 (if running findbugs) * ProtocolBuffer 2.5.0 * CMake 2.6 or newer * Windows SDK 7.1 or Visual Studio 2010 Professional * Windows SDK 8.1 (if building CPU rate control for the container executor) * zlib headers (if building native code bindings for zlib) * Internet connection for first build (to fetch all Maven and Hadoop dependencies) * Unix command-line tools from GnuWin32: sh, mkdir, rm, cp, tar, gzip. These tools must be present on your PATH. Unix command-line tools are also included with the Windows Git package which can be downloaded from http://git-scm.com/download/win. If using Visual Studio, it must be Visual Studio 2010 Professional (not 2012). Do not use Visual Studio Express. It does not support compiling for 64-bit, which is problematic if running a 64-bit system. The Windows SDK 7.1 is free to download here: http://www.microsoft.com/en-us/download/details.aspx?id=8279 The Windows SDK 8.1 is available to download at: http://msdn.microsoft.com/en-us/windows/bg162891.aspx Cygwin is neither required nor supported. ---------------------------------------------------------------------------------- Building: Keep the source code tree in a short path to avoid running into problems related to Windows maximum path length limitation. (For example, C:\hdc). Run builds from a Windows SDK Command Prompt. (Start, All Programs, Microsoft Windows SDK v7.1, Windows SDK 7.1 Command Prompt.) JAVA_HOME must be set, and the path must not contain spaces. If the full path would contain spaces, then use the Windows short path instead. You must set the Platform environment variable to either x64 or Win32 depending on whether you're running a 64-bit or 32-bit system. Note that this is case-sensitive. It must be "Platform", not "PLATFORM" or "platform". Environment variables on Windows are usually case-insensitive, but Maven treats them as case-sensitive. Failure to set this environment variable correctly will cause msbuild to fail while building the native code in hadoop-common. set Platform=x64 (when building on a 64-bit system) set Platform=Win32 (when building on a 32-bit system) Several tests require that the user must have the Create Symbolic Links privilege. All Maven goals are the same as described above with the exception that native code is built by enabling the 'native-win' Maven profile. -Pnative-win is enabled by default when building on Windows since the native components are required (not optional) on Windows. If native code bindings for zlib are required, then the zlib headers must be deployed on the build machine. Set the ZLIB_HOME environment variable to the directory containing the headers. set ZLIB_HOME=C:\zlib-1.2.7 At runtime, zlib1.dll must be accessible on the PATH. Hadoop has been tested with zlib 1.2.7, built using Visual Studio 2010 out of contrib\vstudio\vc10 in the zlib 1.2.7 source tree. http://www.zlib.net/ ---------------------------------------------------------------------------------- Building distributions: * Build distribution with native code : mvn package [-Pdist][-Pdocs][-Psrc][-Dtar]
---------------------------------------------------------------------------------- Importing projects to eclipse # 在源代碼目錄下, 我的源代碼是在 /usr/local/hadoop-2.7.3-src $ su root # 這是我添加的, 用root權限 When you import the project to eclipse, install hadoop-maven-plugins at first. $ cd hadoop-maven-plugins $ mvn install Then, generate eclipse project files. $ cd .. # 這是我添加的 $ mvn eclipse:eclipse -DskipTests At last, import to eclipse by specifying the root directory of the project via [File] > [Import] > [Existing Projects into Workspace]. ----------------------------------------------------------------------------------
具體操作: ( mvn時需要使用root權限,即縣切換到root用戶,用命令su root )
第一步:
$ su root
不這樣做很容易因為權限之類的問題導致mvn失敗. 而且即使mvn 顯示成功,也可能是錯的,如我碰到過這種情況,即mvn時沒花費多長時間就顯示BUILD SUCCESS, 但是它並沒有下載東西, 導入Eclipse會出現各種錯誤. 正確的是第一次mvn是會下載很多東西,花很長時間.
第二步:
$ cd ${HADOOP_HOME}/hadoop-maven-plugins // 這一步的HADOOP_HOME表示hadoop-2.7.3-src所在的目錄, 比如我的是在 /home/hadoop/hadoop-2.7.3-src
$ mvn install // 這一步會下載很多東西( JAR包或 .pom 文件等 ) , 等一段時間
第三步:
$ cd ${HADOOP_HOME} $ mvn eclipse:eclipse -DskipTests // 這一步會下載很多東西( JAR包或 .pom 文件等 ) , 等一段時間
成功的話, 會顯示 BUILD SUCCESS .
到這一步,准備工作做好了,接下來將源代碼該導入Eclipse, 方法是: "File" --> "Import" --> "Existing Projects into Workspace" . 如下圖所示:
最后項目名稱是hadoop-main, 結果如下圖所示:
我看書上介紹,總體上說,目前存在兩種Hadoop源代碼閱讀環境搭建方法,分別是構建Maven工程和構建Java工程.兩種方法各有利弊:前者可通過網絡自動下載依賴的第三方庫,但源代碼會被分散到多個工程中進而帶來閱讀上的不便; 后者可將所有源代碼組織在一個工程中,但需要自己添加依賴的第三方庫,大家可根據自己的喜好選擇一種方法.本節將依次介紹這兩種方法.(書上介紹的辦法不太好用,會出現錯誤
1. 構建Maven工程
Maven 是根據pom.xml進行編譯的,只要文件中包含pom.xml, 就可編譯成單獨的項目.
通過Maven工程搭建Hadoop源代碼閱讀環境的步驟如下:
步驟1 解壓縮Hadoop源代碼.
將下載到的Hadoop源代碼壓縮包解壓到工作目錄下,比如 hadoop-2.7.3-src.tar.gz
步驟2 導入Maven工程.
在Eclipse中, 依次選擇"File" --> "Import" --> "Maven" --> "Existing Maven Project" , 在彈出的對話框中的 "Root Directory" 后面, 選擇Java源代碼所在的目錄.
單擊"Next" 按鈕, 在彈出的對話框中選擇"Resolve All Later", 並單擊"Finish" 按鈕完成Maven 項目導入. 之后, Eclipse 會自動通過網絡從Maven庫中下載依賴的第三方庫(JAR包等). 注意, 你所使用的電腦必須能夠聯網.
將Hadoop 2.7.3源代碼導入Maven項目后,會生成50個左右的工程,這些都是通過Maven構建出來的,每個工程是一個代碼模塊,且彼此相對獨立,可以單獨編譯. 你可以在某個工程下的"src/main/java" 目錄下查看相應的源代碼.
(我導入之后,還有很多錯誤,原因未知)
2. 構建Java工程
通過Java工程搭建Hadoop源代碼閱讀環境的步驟如下:
步驟1 解壓縮Hadoop源代碼
同"構建Maven工程" 中的步驟1類似.
步驟2 新建Java工程
打開Eclipse, 進入Eclipse可視化界面后,如圖1所示, 依次單擊"File" --> "New" --> "Java Project", 並在彈出的對話框中去掉"Use default location" 前的勾號, 然后選擇Hadoop安裝目錄的位置,默認情況下,工程名稱與Hadoop安裝目錄名稱相同,用戶可自行修改. 單擊"完成"按鈕, Hadoop 源代碼工程創建完畢.
圖1 新建Hadoop工程
回到Eclipse主界面后,打開新建的Hadoop工程,可看到整個工程按圖2所示組織代碼: 按目錄組織源代碼,且每個目錄下以JAR包為單位顯示各個Java文件.
圖2 Hadoop工程展示(部分)源代碼方式
除了使用源代碼壓縮包導入Eclipse工程的方法外,讀者可嘗試直接從Hadoop SVN上導入Hadoop源代碼.
需要注意的是,通過以上方法導入Hadoop 2.7.3 源代碼后,很多類或着方法找不到對應的JAR包, 為了解決這個問題,你需要將第三方JAR包導入到工程中,如圖3所示,方法如下:解壓存放JAR包的壓縮包(即hadoop-2.7.3.tar.gz), 然后右擊Project名稱,在彈出的快捷菜單中選擇"Properties" 命令, 將會彈出一個界面,然后在該界面依次選擇 "Java Build Path" --> "Libraries" --> "Add External JARs..." , 將解壓目錄中的share/hadoop 目錄下各個子目錄的中lib文件夾下的JAR包導入工程.
圖3 導入依賴的第三方JAR包 (我們以project2017為例,類似)
前面提到CDH版本將源代碼和JAR包放在了一起,因此,如果使用CDH版本,則直接按照上述方法將源碼導入Eclipse工程即可.
細心的讀者在閱讀源代碼的過程中仍然會發現部分類或者函數無法找到,這是因為Hadoop 2.7.3 使用了Protocol Buffers定義了RPC協議, 而這些Protocol Buffers文件在Maven編譯源代碼時才會生成對應的Java類,因此若其他類在源代碼中引用這些類則暫時無法找到,解決辦法是先編譯Hadoop 2.7.3 源代碼再導入到Eclipse工程,具體方法如下.
首先,使用以下命令安裝Eclipse插件hadoop-maven-plugins:
$ cd ${HADOOP_HOME}/hadoop-maven-plugins
$ mvn install
然后生成Eclipse工程文件:
$ cd ${HADOOP_HOME}
$ mvn eclipse:eclipse -DskipTests
最后在Eclipse中按照以下流程導入源代碼: "File" --> "Import" --> "Existing Projects into Workspace" .