文檔編寫目的
CDP DC7.0.3是Cloudera與Hortonworks合並后，第一個融合CDH和HDP所有組件的on-premise版本，CDP Data Center主要由Cloudera Runtime構成，Cloudera Runtime由超過35個開源項目組成，當然CDP Data Center還包括其它功能如管理功能Cloudera Manager，Key Management，專業支持等，如下圖所示：
Cloudera Runtime的主要組件版本如下圖所示：
|
Component |
Apache Version |
|---|---|
|
Atlas |
2.0.0 |
|
Avro |
1.8 |
|
DAS |
1.4 |
|
Hadoop |
3.1 |
|
HBase |
2.2 |
|
Hive |
3.1 |
|
Hive Warehouse Connector |
1.0 (Technical Preview) |
|
Hue |
4.3 |
|
Impala |
3.3 |
|
Kafka |
2.3 |
|
Kudu |
1.11 |
|
Oozie |
5.1 |
|
ORC |
1.5 |
|
Ozone |
0.4.0 (Technical Preview) |
|
Parquet |
1.1 |
|
Phoenix |
5 |
|
Ranger |
2 |
|
Solr |
7.4 |
|
Spark |
2.4 |
|
Sqoop |
1.4 |
|
Tez |
0.9 |
|
Zookeeper |
3.5.5 |
本文檔Fayson主要描述如何在Redhat7.6安裝CDP DC7.0.3。CDP DC7與CDH5/6的安裝步驟一致，主要包括以下四部分：
1.安全前置准備，包括安裝操作系統、關閉防火牆、同步服務器時鍾等；
2.外部數據庫如MySQL安裝
3.安裝Cloudera Manager；
4.安裝CDH集群；
- 測試環境
1.CM和CDH版本為7.0.3
2.Redhat7.6
3.OpenJDK1.8.0_232
4.MariaDB-5.5.64
5.root用戶安裝
前置准備
2.1 hostname及hosts配置
集群中各個節點之間能互相通信使用靜態IP地址。IP地址和主機名通過/etc/hosts配置，主機名通過/etc/hostname進行配置。
以cm節點（172.31.2.24）為例：
1.hostname配置，建議你將hostname配置為有意義的字母或單詞組合，比如cdp01，請不要使用localhost作為主機名，也不要使用下划線或者其它特殊字符作為主機名。/etc/hostname文件示例如下：
[root@ip-172-31-2-24 ~]# cat /etc/hostname ip-172-31-2-24.ap-southeast-1.compute.internal
2.hosts配置需要將集群的所有主機都加入進來，包括客戶端節點，管理節點和工作節點。/etc/hosts文件示例如下:
[root@ip-172-31-2-24 ~]# cat /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 172.31.2.24 ip-172-31-2-24.ap-southeast-1.compute.internal 172.31.15.60 ip-172-31-15-60.ap-southeast-1.compute.internal 172.31.8.194 ip-172-31-8-194.ap-southeast-1.compute.internal 172.31.13.81 ip-172-31-13-81.ap-southeast-1.compute.internal
以上兩步操作，在集群中其它節點做相同配置。確認CM的hosts文件已經同步到集群所有節點：
2.2 禁用SELinux
在所有節點執行setenforce 0 命令，此處使用批處理shell執行：
[root@ip-172-31-2-24 shell]# sh ssh_do_all.sh node.list "setenforce 0"
集群所有節點修改/etc/selinux/config文件如下：
# This file controls the state of SELinux on the system. # SELINUX= can take one of these three values: # enforcing - SELinux security policy is enforced. # permissive - SELinux prints warnings instead of enforcing. # disabled - No SELinux policy is loaded. SELINUX=disabled # SELINUXTYPE= can take one of three two values: # targeted - Targeted processes are protected, # minimum - Modification of targeted policy. Only selected processes are protected. # mls - Multi Level Security protection. SELINUXTYPE=targeted
2.3 關閉防火牆
集群所有節點執行 systemctl stop命令，此處通過shell批量執行命令如下：
[root@ip-172-31-2-24 shell]# sh ssh_do_all.sh node.list "systemctl stop firewalld" [root@ip-172-31-2-24 shell]# sh ssh_do_all.sh node.list "systemctl disable firewalld" [root@ip-172-31-2-24 shell]# sh ssh_do_all.sh node.list "systemctl status firewalld"
2.4 集群時鍾同步
在Redhat7.x的操作系統上，已經默認的安裝了chrony，我們這里先卸載chrony，然后安裝ntp。使用ntp來配置各台機器的時鍾同步，將cm（172.31.2.24）服務作為本地ntp服務器，其它3台服務器與其保持同步。
1.所有機器卸載chrony
[root@ip-172-31-2-24 shell]# sh ssh_do_all.sh node.list "yum -y remove chrony"
2.所有機器安裝ntp
[root@ip-172-31-2-24 shell]# sh ssh_do_all.sh node.list "yum -y install ntp"
3.cm機器配置時鍾與自己同步
[root@ ip-172-31-2-24 shell]# vim /etc/ntp.conf #server 0.rhel.pool.ntp.org iburst #server 1.rhel.pool.ntp.org iburst #server 2.rhel.pool.ntp.org iburst #server 3.rhel.pool.ntp.org iburst server 127.127.1.0 # local clock fudge 127.127.1.0 stratum 10
4.集群其它節點，配置找cm機器去同步
[root@ip-172-31-11-232 ~]# vim /etc/ntp.conf #server 0.rhel.pool.ntp.org iburst #server 1.rhel.pool.ntp.org iburst #server 2.rhel.pool.ntp.org iburst #server 3.rhel.pool.ntp.org iburst server 172.31.2.24
5.重啟所有機器的ntp服務
[root@ ip-172-31-2-24 shell]# sh ssh_do_all.sh node.list "systemctl restart ntpd" [root@ ip-172-31-2-24 shell]# sh ssh_do_all.sh node.list "systemctl enable ntpd" [root@ ip-172-31-2-24 shell]# sh ssh_do_all.sh node.list "systemctl status ntpd"
6.驗證始終同步，在所有節點執行ntpq -p命令，如下使用腳本批量執行
[root@ip-172-31-2-24 shell]# sh ssh_do_all.sh node.list "ntpq -p"
左邊出現*號表示同步成功。
2.5 設置swap
1.在所有機器執行以下命令以臨時設置swap為1，並即時生效
sh ssh_do_all.sh node.list "sysctl -a | grep vm.swappiness" sh ssh_do_all.sh node.list "echo 1 > /proc/sys/vm/swappiness" sh ssh_do_all.sh node.list "sysctl -a | grep vm.swappiness"
確保最后swap打印應該都為1
2.為所有機器永久設置swap為1，修改/etc/sysctl.conf中vm.swappiness為1，沒有則新增。
[root@ip-172-31-2-24 ~]# vim /etc/sysctl.conf vm.swappiness = 1
將/etc/sysctl.conf文件同步到集群所有機器
2.6 設置透明大頁面
1.所有節點執行以下命令關閉透明大頁面，並即時生效
sh ssh_do_all.sh node.list "echo never > /sys/kernel/mm/transparent_hugepage/defrag" sh ssh_do_all.sh node.list "echo never > /sys/kernel/mm/transparent_hugepage/enabled" sh ssh_do_all.sh node.list "cat /sys/kernel/mm/transparent_hugepage/enabled" sh ssh_do_all.sh node.list "cat /sys/kernel/mm/transparent_hugepage/defrag"
2.修改所有節點的/etc/rc.d/rc.local文件的權限以實現開機執行
[root@ip-172-31-2-24 shell]# sh ssh_do_all.sh node.list "chmod +x /etc/rc.d/rc.local" [root@ip-172-31-2-24 shell]# sh ssh_do_all.sh node.list "ls -l /etc/rc.d/rc.local"
3.在所有節點的/etc/rc.d/rc.local文件中新增如下內容，以實現開機自動關閉透明大頁面。
if test -f /sys/kernel/mm/transparent_hugepage/enabled; then echo never > /sys/kernel/mm/transparent_hugepage/enabled fi if test -f /sys/kernel/mm/transparent_hugepage/defrag; then echo never > /sys/kernel/mm/transparent_hugepage/defrag fi
將該文件同步到所有節點
2.7 配置操作系統repo
Fayson用的是AWS的環境，這步是可以省略的，放在這里供物理機部署的兄弟們參考。
1.掛載操作系統iso文件
[root@ip-172-31-2-24 ~]$ sudo mkdir /media/DVD1 [root@ip-172-31-2-24 ~]$ sudo mount -o loop CentOS-7-x86_64-DVD-1611.iso /media/DVD1/
2.配置操作系統repo
[root@ip-172-31-2-24 ~]$ sudo vim /etc/yum.repos.d/local_os.repo [local_iso] name=CentOS-$releasever - Media baseurl=file:///media/DVD1 gpgcheck=0 enabled=1 [root@ip-172-31-2-24 ~]$ sudo yum repolist
2.8 安裝httpd服務
1.安裝httpd服務
[root@ip-172-31-2-24 shell]# yum -y install httpd
2.啟動httpd服務
[root@ip-172-31-2-24 shell]# systemctl start httpd
3.安裝完httpd后，重新制作操作系統repo，換成http的方式方便其它服務器也可以訪問
[root@ip-172-31-2-24 ~]$ mkdir /var/www/html/iso [root@ip-172-31-2-24 ~]$ scp -r /media/DVD1/* /var/www/html/iso/ [root@ip-172-31-2-24 ~]$ vim /etc/yum.repos.d/os.repo [osrepo] name=os_repo baseurl=http:// 172.31.2.24/iso/ enabled=true gpgcheck=false [root@ip-172-31-2-24 ~]$ yum repolist
4.修改/etc/httpd/conf/httpd.conf配置文件，在<IfModule mime_module>中修改以下內容
AddType application/x-gzip .gz .tgz .parcel
5.保存httpd.conf的修改，並重啟httpd服務
[root@ip-172-31-2-24 java]# systemctl restart httpd
2.9 安裝MariaDB
1.安裝MariaDB
[root@ip-172-31-2-24 ~]# yum -y install mariadb [root@ip-172-31-2-24 ~]# yum -y install mariadb-server
2.啟動並配置MariaDB
[root@ip-172-31-2-24 ~]# systemctl start mariadb [root@ip-172-31-2-24 ~]# systemctl enable mariadb Created symlink from /etc/systemd/system/multi-user.target.wants/mariadb.service to /usr/lib/systemd/system/mariadb.service. [root@ip-172-31-2-24 ~]# /usr/bin/mysql_secure_installation NOTE: RUNNING ALL PARTS OF THIS SCRIPT IS RECOMMENDED FOR ALL MariaDB SERVERS IN PRODUCTION USE! PLEASE READ EACH STEP CAREFULLY! In order to log into MariaDB to secure it, we'll need the current password for the root user. If you've just installed MariaDB, and you haven't set the root password yet, the password will be blank, so you should just press enter here. Enter current password for root (enter for none): OK, successfully used password, moving on... Setting the root password ensures that nobody can log into the MariaDB root user without the proper authorisation. Set root password? [Y/n] Y New password: Re-enter new password: Password updated successfully! Reloading privilege tables.. ... Success! By default, a MariaDB installation has an anonymous user, allowing anyone to log into MariaDB without having to have a user account created for them. This is intended only for testing, and to make the installation go a bit smoother. You should remove them before moving into a production environment. Remove anonymous users? [Y/n] Y ... Success! Normally, root should only be allowed to connect from 'localhost'. This ensures that someone cannot guess at the root password from the network. Disallow root login remotely? [Y/n] n ... skipping. By default, MariaDB comes with a database named 'test' that anyone can access. This is also intended only for testing, and should be removed before moving into a production environment. Remove test database and access to it? [Y/n] Y - Dropping test database... ... Success! - Removing privileges on test database... ... Success! Reloading the privilege tables will ensure that all changes made so far will take effect immediately. Reload privilege tables now? [Y/n] Y ... Success! Cleaning up... All done! If you've completed all of the above steps, your MariaDB installation should now be secure. Thanks for using MariaDB!
3.建立CM，Hive等需要的表
[root@ip-172-31-2-24 shell]# mysql -u root -p Enter password: Welcome to the MariaDB monitor. Commands end with ; or \g. Your MariaDB connection id is 9 Server version: 5.5.64-MariaDB MariaDB Server Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. MariaDB [(none)]> create database metastore default character set utf8; Query OK, 1 row affected (0.00 sec) MariaDB [(none)]> CREATE USER 'hive'@'%' IDENTIFIED BY 'password'; Query OK, 0 rows affected (0.00 sec) MariaDB [(none)]> GRANT ALL PRIVILEGES ON metastore. * TO 'hive'@'%'; Query OK, 0 rows affected (0.01 sec) MariaDB [(none)]> FLUSH PRIVILEGES; Query OK, 0 rows affected (0.00 sec) MariaDB [(none)]> create database cm default character set utf8; Query OK, 1 row affected (0.00 sec) MariaDB [(none)]> CREATE USER 'cm'@'%' IDENTIFIED BY 'password'; Query OK, 0 rows affected (0.00 sec) MariaDB [(none)]> GRANT ALL PRIVILEGES ON cm. * TO 'cm'@'%'; Query OK, 0 rows affected (0.00 sec) MariaDB [(none)]> FLUSH PRIVILEGES; Query OK, 0 rows affected (0.00 sec) MariaDB [(none)]> create database rm default character set utf8; Query OK, 1 row affected (0.00 sec) MariaDB [(none)]> CREATE USER 'rm'@'%' IDENTIFIED BY 'password'; Query OK, 0 rows affected (0.00 sec) MariaDB [(none)]> GRANT ALL PRIVILEGES ON rm. * TO 'rm'@'%'; Query OK, 0 rows affected (0.00 sec) MariaDB [(none)]> FLUSH PRIVILEGES; Query OK, 0 rows affected (0.00 sec) MariaDB [(none)]> create database hue default character set utf8; Query OK, 1 row affected (0.00 sec) MariaDB [(none)]> CREATE USER 'hue'@'%' IDENTIFIED BY 'password'; Query OK, 0 rows affected (0.00 sec) MariaDB [(none)]> GRANT ALL PRIVILEGES ON hue. * TO 'hue'@'%'