虛擬機環境下用docker在centos環境中搭建Ambari HDP大數據平台

利用虛擬機安裝docker，在docker環境中搭建HDP大數據平台

前言 : 在一台虛擬機上搭建3個docker容器 作為安裝HDP的環境，虛擬機環境centos7.6 docker環境為centos7.6

因為想要對ambari進行二次開發，所以本次選擇手動編譯ambari。

https://hub.docker.com/_/centos 在此網站查詢想要的版本號

一、在虛擬機上准備工作

1.1安裝docker

yum -y install docker

 

1.2下載鏡像

docker search centos      　　　　   //搜索鏡像
docker pull centos:7.6.1810        //從官網拉取鏡像
docker images            　　　　　　//查看本地鏡像
docker ps -a              　　　　  //查看容器

 

1.3啟動容器

因為是在1台虛擬機搭建3個docker的緣故，所以這次選擇的docker 網卡類型為bridge模式 網段為 172.17.0.0/16 網關為172.17.0.1

三台容器的主機名和ip分別為

server     172.17.0.2 agent1 172.17.0.3 agent2 172.17.0.4

※注意點：bridge模式容器ip無法自己指定

開始啟動3台容器，如下圖

 

 

 

 

 

docker run -it -d --name server --network bridge --hostname server docker.io/centos:7.6.1810 /bin/bash

1.4查看容器並進入

 

 

 

 

 

docker ps 

 

進入容器

 

 

 

docker exec -it server /bin/bash

 

二、搭建編譯環境

2.1獲取Ambari2.7.5源碼

下載地址：http://apache.communilink.net/ambari/ambari-2.7.5/apache-ambari-2.7.5-src.tar.gz

wget http://apache.communilink.net/ambari/ambari-2.7.5/apache-ambari-2.7.5-src.tar.gz tar zxvf apache-ambari-2.7.5-src.tar.gz

2.2搭建編譯環境

 

安裝包已放到百度雲 maven jdk nodejs 鏈接：https://pan.baidu.com/s/1SVf7n5d2M_byquSpwN5yXg 提取碼：i4e6 

2.2.1安裝網絡工具

yum install net-tools.x86_64

 2.2.2 安裝jdk

rpm -ivh jdk-8u74-linux-x64.rpm

2.2.3 安裝maven，nodejs

tar -zxvf apache-maven-3.3.9-bin.tar.gz
tar -zxvf node-v4.5.0-linux-x64.tar.gz
yum install epel-release

2.2.3.1 升級nodejs

npm install n -g n stable

 

2.2.4 配置maven，nodejs的環境變量

vim /etc/profile

 

 

 

 

export MAVEN_HOME=/opt/apache-maven-3.3.9
export PATH=$MAVEN_HOME/bin:$PATH

export NODE_HOME=/opt/node-v4.5.0-linux-x64

export PATH=$PATH:$NODE_HOME/bin

export NODE_PATH=$NODE_HOME/lib/node_modules

 

source /etc/profile

2.2.5 python 2.6或2.7（系統自帶）

 

2.2.6安裝rpmbuild

yum install rpm-build

 

2.2.7安裝g++

yum install gcc-c++

 

2.2.8安裝python-devel

yum install python-devel

 

2.2.9安裝bower、gulp

npm install -g bower npm install -g gulp

 

2.2.10安裝git

yum install git

 

2.2.11安裝yarn

yum install yarn

 

三、提前下載並設置

 下載源碼,設置版本信息

wget https://www-eu.apache.org/dist/ambari/ambari-2.7.5/apache-ambari-2.7.5-src.tar.gz tar -xfvz apache-ambari-2.7.5-src.tar.gz cd apache-ambari-2.7.5-src mvn versions:set -DnewVersion=2.7.5.0.0 pushd ambari-metrics mvn versions:set -DnewVersion=2.7.5.0.0 popd

 

 

有些包比較大，或者編譯時下載時間較長，可以提前下載到本地目錄，再修改pom.xml文件指定到本地目錄（pom文件位置:/opt/apache-ambari-2.7.5-src/ambari-metrics/pom.xml）

 

 

 

 

3.1 hbase下載

mkdir -p /tmp/hbase wget -O /tmp/hbase/hbase-2.0.2.3.1.4.0-315.tar.gz https://s3.amazonaws.com/dev.hortonworks.com/HDP/centos7/3.x/BUILDS/3.1.4.0-315/tars/hbase/hbase-2.0.2.3.1.4.0-315-bin.tar.gz

 

3.2 grafana 包下載

mkdir -p /tmp/grafana wget -O /tmp/grafana/grafana-6.4.2.linux-amd64.tar.gz  https://dl.grafana.com/oss/release/grafana-6.4.2.linux-amd64.tar.gz

 

3.3 hadoop包下載

mkdir -p /tmp/hadoop

wget -O /tmp/hadoop/hadoop-3.1.1.3.1.4.0-315.tar.gz https://s3.amazonaws.com/dev.hortonworks.com/HDP/centos7/3.x/BUILDS/3.1.4.0-315/tars/hadoop/hadoop-3.1.1.3.1.4.0-315.tar.gz

 

3.4phoenix包下載

mkdir -p /tmp/phoenix

wget -O /tmp/hadoop/phoenix-5.0.0.3.1.4.0-315.tar.gz https://s3.amazonaws.com/dev.hortonworks.com/HDP/centos7/3.x/BUILDS/3.1.4.0-315/tars/phoenix/phoenix-5.0.0.3.1.4.0-315.tar.gz

 

3.5 開始編譯 

mvn -B clean install rpm:rpm -DnewVersion=2.7.5.0.0 -DbuildNumber=5895e4ed6b30a2da8a90fee2403b6cab91d19972 -DskipTests -Dpython.ver="python >= 2.6" -Drat.skip=true -Preplaceurl

 

最后

 

 

編譯好后的rpm包在

cd ambari-server/target/rpm/ambari-server/RPMS/x86_64/ ambari-server-2.7.4.0-0.x86_64.rpm

 

 

編譯遇到的問題

問題1：

 [INFO] ------------------------------------------------------------------------
 [ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.2.1:exec (Bower install) on project ambari-admin: Command execution failed. Process exited with an error: 1 (Exit value: 1) -> [Help 1] org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.2.1:exec (Bower install) on project ambari-admin: Command execution failed. at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:212) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80) at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51) at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128) at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307) at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193) at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106) at org.apache.maven.cli.MavenCli.execute(MavenCli.java:863) at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288) at org.apache.maven.cli.MavenCli.main(MavenCli.java:199) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289) at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229) at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415) at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356) Caused by: org.apache.maven.plugin.MojoExecutionException: Command execution failed. at org.codehaus.mojo.exec.ExecMojo.execute(ExecMojo.java:362) at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:207) ... 20 more Caused by: org.apache.commons.exec.ExecuteException: Process exited with an error: 1 (Exit value: 1) at org.apache.commons.exec.DefaultExecutor.executeInternal(DefaultExecutor.java:377) at org.apache.commons.exec.DefaultExecutor.execute(DefaultExecutor.java:160) at org.codehaus.mojo.exec.ExecMojo.executeCommandLine(ExecMojo.java:610) at org.codehaus.mojo.exec.ExecMojo.execute(ExecMojo.java:352) ... 22 more [ERROR] [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] [http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException] [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn <goals> -rf :ambari-admin

解決方法：修改 /XXXX/ambari-admin/src/main/resources/ui/admin-web/package.json 的bower 版本將1.3.8 改為 1.8.8 問題解決

/ambari-admin/src/main/resources/ui/admin-web/package.json @@ -3,7 +3,7 @@
   "version": "0.0.0", "dependencies": {}, "devDependencies": { -    "bower": "1.3.8", +    "bower": "1.8.8", "gulp": "^3.8.8", "gulp-add-src": "^0.2.0", "gulp-autoprefixer": "0.0.7",

 

問題2：Too many files with unapproved license

解決辦法：

由於許可license沒有造成的，需要再mvn命令中增加如下內容：

-Drat.skip=true

 

問題3：Unexpected character 0x0 in identifier
編譯ambari-utility，報錯信息：

 

說明：

出錯的Java文件編碼和CheckStyle設置的編碼不同。CheckStyle里設置的編碼是UTF-8

解決辦法：

將“._xxx.java”文件直接刪除即可。

（注：其實有很多類似這樣的文件報錯，將類似這樣的文件刪除掉即可）

 

問題4： Download HBase超時

 

解決辦法：

將./ambari-metrics/pom.xml的

<hbase.tar>http://dev.hortonworks.com.s3.amazonaws.com/HDP/centos7/3.x/BUILDS/3.0.0.0-1634/tars/hbase/hbase-2.0.0.3.0.0.0-1634-bin.tar.gz</hbase.tar>

替換為本地路徑

<hbase.tar>file:///tmp/hbase/hbase-2.0.0.3.0.0.0-1634-bin.tar.gz</hbase.tar>

 

問題5：Unable to build the RPM

Unable to build the RPM: Error while executing process. Cannot run program “rpmbuild”

解決辦法：

rpm -qa | grep rpm-build #檢測是否已安裝rpmbuild
yum install rpm-build #如果沒安裝則手動安裝

 

問題6. buildNumber : unbound variable

vim /usr/sbin/ambari-server 將${buildNumber}這行換成 HASH="${VERSION}"

 

該文件源碼位置位於：./ambari-server/sbin/ambari-server，修改后，在進行編譯，可以一勞永逸。

 

問題7：

[INFO] Ambari Admin View .................................. FAILURE

[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.6.0:exec (Bower install) on project ambari-admin: Command execution failed. Process exited with an error: 1 (Exit value: 1) -> [Help 1]

org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.6.0:exec (Bower install) on project ambari-admin: Command execution failed.

單獨編譯報錯：

ECMDERR Failed to execute "git ls-remote --tags --heads https://github.com/twbs/bootstrap.git", exit code of #128

Additional error details:
fatal: unable to access 'https://github.com/twbs/bootstrap.git/': Peer reports incompatible or unsupported protocol version.

解決：個人將ambari-admin（包括子目錄ambari-admin/src/main/resources/ui/admin-web）里面生成的node、node_modules、package-lock.json刪除，保持干凈的源碼環境，再單獨編譯ambari-admin

#cd ambari-admin/src/main/resources/ui/admin-web目錄下，編輯 .bowerrc ，修改后的內容如下：

{ "directory": "app/bower_components", "allow_root": true }

 

在上述目錄下，安裝npm依賴包，全局安裝gulp、bower

npm install -g bower

npm install -g gulp

 

安裝bower的依賴包（個人在ambari-admin/src/main/resources/ui/admin-web/下執行）

bower install

 

(上述錯誤原因在於bower install命令失敗，個人將之前成功編譯的bower_components目錄（ambari-admin/src/main/resources/ui/admin-web/app目錄下）拷貝到ambari-release-2.6.1/ambari-admin/src/main/resources/ui/admin-web/app目錄下，再次運行成功)

安裝 gulp-webserver

npm install gulp-webserver --save-dev

 

運行命令mvn install -Drat.skip=true -Preplaceurl -X重新單獨編譯ambari-admin模塊(此命令在ambari-admin目錄下運行)