hadoop生態圈框架工具甚多，自己整合起來很是繁瑣，特別是oozie以及hue結合來實現hive、mr、spark等定時依賴任務更是一步一個坑，為了減少踩坑，只好從apache hadoop，轉向CDH。
兩個各有利弊，CDH貴在整合方便。本次教程大部分來自http://www.cnblogs.com/fujiangong/p/5620050.html，稍作修改，記錄踩下的坑。
JDK環境
JDK版本：jdk-8u121-linux-x64.rpm
CM包
CM版本：5.10.0 cloudera-manager-centos7-cm5.10.0_x86_64.tar.gz
下載地址：http://archive.cloudera.com/cm5/cm/5/cloudera-manager-centos7-cm5.10.0_x86_64.tar.gz
CDH包
CDH版本：5.10.0
CDH-5.10.0-1.cdh5.10.0.p0.41-el7.parcel CDH-5.10.0-1.cdh5.10.0.p0.41-el7.parcel.sha1 manifest.json
下載地址
http://archive.cloudera.com/cdh5/parcels/5.10.0/CDH-5.10.0-1.cdh5.10.0.p0.41-el7.parcel
http://archive.cloudera.com/cdh5/parcels/5.10.0/CDH-5.10.0-1.cdh5.10.0.p0.41-el7.parcel.sha1
http://archive.cloudera.com/cdh5/parcels/5.10.0/manifest.json
java-mysql連接jar包
下載地址：http://central.maven.org/maven2/mysql/mysql-connector-java/5.1.22/mysql-connector-java-5.1.22.jar
備注：mysql的jar包不要使用6.x版本，會有兼容問題，這里使用的是5.1.22
數據庫
類型：MariaDB
節點
10.0.0.1 slave1
10.0.0.2 slave2
10.0.0.3 slave3
10.0.0.4 slave4
10.0.0.5 master
10.0.0.6 standby
二、底層環境准備
注：准備安裝前需要把相關的包文件下載下來，並上傳到服務器上
1、安裝JDK
所有節點
1）安裝系統原有java
yum -y
install
java
|
2）卸載OpenJDK
cdh指定使用oracle jdk，所以卸載掉
openjdk-headless
openjdk
保留其它工具包
3）安裝下載的java
rpm -ivh jdk-8u121-linux-x64.rpm
|
2、更改主機名和hosts
所有節點
注意每個主機名對應的IP，按實際情況更改。
/etc/hostname
/etc/hosts
3、關閉防火牆
systemctl stop firewalld
systemctl disable firewalld
sed
-i
"s/SELINUX=enforcing/SELINUX=disabled/"
/etc/selinux/config
reboot
|
4、安裝NTP服務
所有節點
yum -y
install
ntp
|
配置略過
5、SSH無密碼登陸
配置Cloudera Management Service所在節點無密碼登陸到其他節點
本次在節點standby節點
ssh
-keygen -t rsa
Generating public
/private
rsa key pair.
Enter
file
in
which
to save the key (
/root/
.
ssh
/id_rsa
): 回車
Enter passphrase (empty
for
no passphrase): 回車
Enter same passphrase again: 回車
Your identification has been saved
in
/root/
.
ssh
/id_rsa
.
Your public key has been saved
in
/root/
.
ssh
/id_rsa
.pub.
The key fingerprint is:
bc
:14:e2:59:42:1f:21:80:eb:49:d8:3b:7c:8d:b7:f9 root@bigdata05.tfpay.com
The key's randomart image is:
+--[ RSA 2048]----+
| ...o o. |
| . . o . |
| o . o + |
|. + . * . |
| + o oo S |
| * o o. . |
| o . o. |
| o |
| .E |
+-----------------+
ssh
-copy-
id
master
ssh
-copy-
id
slave1
ssh
-copy-
id
slave2
ssh
-copy-
id
slave3
ssh
-copy-
id
slave4
|
6、安裝數據庫
service節點
yum -y
install
mariadb-servervi
/etc/my
.cnf
[mysqld]
datadir=
/var/lib/mysql
socket=
/var/lib/mysql/mysql
.sock
symbolic-links=0
character-
set
-server=utf8 \\添加的行
[mysqld_safe]
log-error=
/var/log/mariadb/mariadb
.log
pid-
file
=
/var/run/mariadb/mariadb
.pidsystemctl start mariadb
systemctl
enable
mariadb
|
初始化數據庫
mysql_secure_installation
/usr/bin/mysql_secure_installation
: line 379: find_mysql_client:
command
not found
NOTE: RUNNING ALL PARTS OF THIS SCRIPT IS RECOMMENDED FOR ALL MariaDB
SERVERS IN PRODUCTION USE! PLEASE READ EACH STEP CAREFULLY!
In order to log into MariaDB to secure it, we'll need the current
password
for
the root user. If you've just installed MariaDB, and
you haven't
set
the root password yet, the password will be blank,
so you should just press enter here.
Enter current password
for
root (enter
for
none):
OK, successfully used password, moving on...
Setting the root password ensures that nobody can log into the MariaDB
root user without the proper authorisation.
Set root password? [Y
/n
] y \\設置數據庫root密碼
New password: \\設置密碼
Re-enter new password: \\再次輸入密碼
Password updated successfully!
Reloading privilege tables..
... Success!
By default, a MariaDB installation has an anonymous user, allowing anyone
to log into MariaDB without having to have a user account created
for
them. This is intended only
for
testing, and to
make
the installation
go a bit smoother. You should remove them before moving into a
production environment.
Remove anonymous
users
? [Y
/n
] y \\移除匿名用戶
... Success!
Normally, root should only be allowed to connect from
'localhost'
. This
ensures that someone cannot guess at the root password from the network.
Disallow root login remotely? [Y
/n
] y \\禁止root用戶遠程登錄
... Success!
By default, MariaDB comes with a database named
'test'
that anyone can
access. This is also intended only
for
testing, and should be removed
before moving into a production environment.
Remove
test
database and access to it? [Y
/n
] y \\刪除測試數據庫
- Dropping
test
database...
... Success!
- Removing privileges on
test
database...
... Success!
Reloading the privilege tables will ensure that all changes made so far
will take effect immediately.
Reload privilege tables now? [Y
/n
]
//
... Success!
Cleaning up...
All
done
! If you've completed all of the above steps, your MariaDB
installation should now be secure.
Thanks
for
using MariaDB!
|
7、其他安裝與配置
所有節點
# yum -y install psmisc MySQL-python at bc bind-libs bind-utils cups-client cups-libs cyrus-sasl-gssapi cyrus-sasl-plain ed fuse fuse-libs httpd httpd-tools keyutils-libs-devel krb5-devel libcom_err-devel libselinux-devel libsepol-devel libverto-devel mailcap noarch mailx mod_ssl openssl-devel pcre-devel postgresql-libs python-psycopg2 redhat-lsb-core redhat-lsb-submod-security x86_64 spax time zlib-devel
# chmod +x /etc/rc.d/rc.local
# echo "echo 0 > /proc/sys/vm/swappiness" >>/etc/rc.d/rc.local
# echo "echo never > /sys/kernel/mm/transparent_hugepage/defrag" >>/etc/rc.d/rc.local
# echo 0 > /proc/sys/vm/swappiness
# echo never > /sys/kernel/mm/transparent_hugepage/defrag
# yum -y install rpcbind
# systemctl start rpcbind
# echo "systemctl start rpcbind" >> /etc/rc.d/rc.local
|
三、開始安裝CDH
1、傳寶、解包
把下載好的一下相應包文件上傳到service節點內
mysql-java連接包在所有節點上都要上傳
tar
zxvf cloudera-manager-centos7-cm5.10.0_x86_64.
tar
.gz -C
/opt/
|
2、創建數據庫
# mysql -uroot -p
Enter password: \\輸入數據庫密碼
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection
id
is 10
Server version: 5.5.47-MariaDB MariaDB Server
Copyright (c) 2000, 2015, Oracle, MariaDB Corporation Ab and others.
Type
'help;'
or
'\h'
for
help. Type
'\c'
to
clear
the current input statement.
MariaDB [(none)]> create database hive DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
Query OK, 1 row affected (0.00 sec)
MariaDB [(none)]> create database amon DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
Query OK, 1 row affected (0.00 sec)
MariaDB [(none)]> create database hue DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
Query OK, 1 row affected (0.00 sec)
MariaDB [(none)]> create database monitor DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
Query OK, 1 row affected (0.00 sec)
MariaDB [(none)]> create database oozie DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
Query OK, 1 row affected (0.00 sec)
MariaDB [(none)]> grant all on *.* to root@
"%"
Identified by
"1234.com"
;
Query OK, 0 rows affected (0.00 sec)
MariaDB [(none)]>
exit
# /opt/cm-5.10.0/share/cmf/schema/scm_prepare_database.sh mysql cm -hlocalhost -uroot -p1234.com --scm-host localhost scm scm scm
JAVA_HOME=
/usr/java/jdk1
.8.0_121
Verifying that we can write to
/opt/cm-5
.10.0
/etc/cloudera-scm-server
Creating SCM configuration
file
in
/opt/cm-5
.10.0
/etc/cloudera-scm-server
Executing:
/usr/java/jdk1
.8.0_121
/bin/java
-
cp
/usr/share/java/mysql-connector-java
.jar:
/usr/share/java/oracle-connector-java
.jar:
/opt/cm-5
.10.0
/share/cmf/schema/
..
/lib/
* com.cloudera.enterprise.dbutil.DbCommandExecutor
/opt/cm-5
.10.0
/etc/cloudera-scm-server/db
.properties com.cloudera.cmf.db.
2016-06-17 13:32:20,562 [main] INFO com.cloudera.enterprise.dbutil.DbCommandExecutor - Successfully connected to database.
All
done
, your SCM database is configured correctly!
|
3、創建用戶
所有節點
useradd
--system --home=
/opt/cm-5
.10.0
/run/cloudera-scm-server
--shell=
/bin/false
--comment
"Cloudera SCM User"
cloudera-scm
|
4、制作本地源
service節點執行
cp
CDH-5.10.0-1.cdh5.10.0.p0.41-el7.parcel CDH-5.10.0-1.cdh5.10.0.p0.41-el7.parcel.sha1 manifest.json
/opt/cloudera/parcel-repo/
mv
CDH-5.10.0-1.cdh5.10.0.p0.41-el7.parcel.sha1 CDH-5.10.0-1.cdh5.10.0.p0.41-el7.parcel.sha
|
5、拷貝jar包
所有節點
cp
mysql-connector-java-5.1.22.jar
/usr/share/java/mysql-connector-java
.jar
|
service節點
cp
mysql-connector-java-5.1.22.jar
/opt/cm-5
.10.0
/share/cmf/lib/
|
6、更改配置
sed
-i
"s/server_host=localhost/server_host=10.0.0.6/"
/opt/cm-5
.10.0
/etc/cloudera-scm-agent/config
.ini
mkdir
/opt/cm-5
.10.0
/run/cloudera-scm-agent
|
7、啟動服務
service節點
/opt/cm-5
.10.0
/etc/init
.d
/cloudera-scm-server
start
scp
-r
/opt/cm-5
.10.0 master:
/opt
scp
-r
/opt/cm-5
.10.0 slave1:
/opt
………………
|
所有節點
/opt/cm-5
.10.0
/etc/init
.d
/cloudera-scm-agent
start
|
8、瀏覽頁面開始安裝
1）登陸
打開瀏覽器，訪問service節點的7180端口
輸入用戶名和密碼admin、admin
安裝的時候沒有截圖，圖就用網上的教程里的了，CDH5都差不多，小版本改動不大
2）接受協議
3）選擇版本
本文檔選擇Cloudera Express免費開源版本
4）指定主機
5）選擇CDH版本
注：選擇在制作本地源時選擇的版本，如果選擇別的版本的話會去官網下載那樣安裝速度會很慢
上圖里我們選5.10.0
6）檢查主機正確性
確保全部驗證通過
7）選擇安裝的服務
本文選擇的是安裝所有服務
8）角色分配
這里應該根據自己的實際需求和服務器的硬件參數自己選擇
9）數據庫設置
10）群集設置
可以根據自己的實際情況更改設置，本文采用默認配置。這些配置可以在集群安裝完成以后后續更改
11）開始安裝
無錯誤信息全部通過
12）安裝完成
第一次安裝完成后，會出現一些配置的警告信息。這些可以根據提示信息更改。
四、問題總結
1、由於我們的服務器是在阿里雲上，無法通過內網IP訪問HUE，所有，需要設置HUE可通過外網訪問：
勾選之后重啟hue，但是過會兒就會提示運行狀態不良，這里是CDH有個問題，講HUE綁定0.0.0.0之后，agent檢測hue時，檢測IP地址也會使用0.0.0.0，而不是內網IP，導致通訊是由於安全原因出現問題，所以需要添加配置：
2、開啟hue的notebook選項卡
默認不開啟notebook，需要添加配置：
