HDP集群搭建全過程

基於虛擬機搭建hdp集群,以下是我搭建集群的虛擬機環境說明

一.環境預配置

參照官網配置步驟:

1. 配置hosts與主機名

vim /etc/hosts
# 添加以下內容(三台)
192.168.127.121 hadoop121
192.168.127.122 hadoop122
192.168.127.123 hadoop123

設置主機名
hostnamectl set-hostname hadoop121

2. 配置ntp

yum install ntp -y

#hadoop121
vim /etc/ntp.conf
server 127.127.1.0
fudge 127.127.1.0 stratum 10

#hadoop122,hadoop123
echo "server hadoop121 iburst" > /etc/ntp.conf

啟動ntp服務
systemctl start ntpd.service
設置開機自啟
systemctl enable ntpd.service

3. 配置免密

ssh-keygen
四下回車

ssh-copy-id hadoop121
輸入密碼

4. 關閉防火牆和selinux

#關閉防火牆
systemctl stop firewalld

# 並關閉防火牆開機自啟
systemctl disable firewalld

關閉selinux
vim /etc/selinux/config

SELINUX=disabled

手動關閉
setenforce 0
查看狀態為Permissive
getenforce 0

5. 安裝數據庫

[root@hadoop121 software]# rpm -qa | grep -I mariadb
mariadb-libs-5.5.60-1.el7_5.x86_64
[root@hadoop121 software]# rpm -e mariadb-libs-5.5.60-1.el7_5.x86_64
error: Failed dependencies:
	libmysqlclient.so.18()(64bit) is needed by (installed) postfix-2:2.10.1-7.el7.x86_64
	libmysqlclient.so.18(libmysqlclient_18)(64bit) is needed by (installed) postfix-2:2.10.1-7.el7.x86_64
[root@hadoop121 software]# rpm -e mariadb-libs-5.5.60-1.el7_5.x86_64 --nodeps
[root@hadoop121 software]# rpm -qa | grep -I mariadb
[root@hadoop121 software]# rpm -qa | grep -I mysql
[root@hadoop121 software]# rpm -ivh mysql-community-*.rpm --nodeps
warning: mysql-community-client-5.7.29-1.el7.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 5072e1f5: NOKEY
Preparing...                          ################################# [100%]
Updating / installing...
   1:mysql-community-common-5.7.29-1.e################################# [ 20%]
   2:mysql-community-libs-5.7.29-1.el7################################# [ 40%]
   3:mysql-community-client-5.7.29-1.e################################# [ 60%]
   4:mysql-community-server-5.7.29-1.e################################# [ 80%]
   5:mysql-community-libs-compat-5.7.2################################# [100%]

啟動mysql
查看初始密碼
[root@hadoop121 software]# cat /var/log/mysqld.log | grep password
2022-04-13T07:04:24.093457Z 1 [Note] A temporary password is generated for root@localhost: *yyPriqOh73*


mysql -uroot -p*yyPriqOh73*

修改密碼安全等級及root密碼,設置遠程連接
mysql> set global validate_password_policy=LOW;
Query OK, 0 rows affected (0.00 sec)

mysql> set global validate_password_length=4;
Query OK, 0 rows affected (0.00 sec)

mysql> ALTER USER 'root'@'localhost' IDENTIFIED BY 'root';
Query OK, 0 rows affected (0.07 sec)


mysql> update user set host='%' where user='root';
Query OK, 1 row affected (0.00 sec)
Rows matched: 1  Changed: 1  Warnings: 0

mysql> flush privileges;
Query OK, 0 rows affected (0.00 sec)

使用datagrip工具連接測試

6.其他配置

6.1 透明大頁面
6.2 文件句柄和進程數限制
6.3 安裝jdk,scala,python3,maven

二.安裝ambari

#提前准備好下面四個安裝包:
-rw-r--r--. 1 root root 2030700094 Apr 13 13:54 ambari-2.7.4.0-centos7.tar.gz
-rw-r--r--. 1 root root 9506255805 Apr 13 13:57 HDP-3.1.4.0-centos7-rpm.tar.gz
-rw-r--r--. 1 root root     162038 Apr 13 13:54 HDP-GPL-3.1.4.0-centos7-gpl.tar.gz
-rw-r--r--. 1 root root   90606616 Apr 13 13:54 HDP-UTILS-1.1.0.22-centos7.tar.gz

下載鏈接:
訪問Cloudera的官網，進入到如下網址界面：https://docs.cloudera.com/HDPDocuments/Ambari-2.7.5.0/bk_ambari-installation/content/ambari_repositories.html
請下載以下的資源包：
Ambari相關安裝包：http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.7.4.0/ambari-2.7.4.0-centos7.tar.gz
HDP安裝：http://public-repo-1.hortonworks.com/HDP/centos7/3.x/updates/3.1.4.0/HDP-3.1.4.0-centos7-rpm.tar.gz
HDP-UTIL安裝包：http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.22/repos/centos7/HDP-UTILS-1.1.0.22-centos7.tar.gz
HDP-GPL文件：http://public-repo-1.hortonworks.com/HDP-GPL/centos7/3.x/updates/3.1.4.0/HDP-GPL-3.1.4.0-centos7-gpl.tar.gz

1. 配置 Ambari本地YUM源

安裝httpd服務

mkdir -p /var/www/html/hdp314

將安裝包解壓到創建的目錄:
tar -zxvf /software/ambari-2.7.4.0-centos7.tar.gz -C /var/www/html/hdp314/
tar -zxvf /software/HDP-3.1.4.0-centos7-rpm.tar.gz -C /var/www/html/hdp314/
tar -zxvf /software/HDP-GPL-3.1.4.0-centos7-gpl.tar.gz -C /var/www/html/hdp314/
tar -zxvf /software/HDP-UTILS-1.1.0.22-centos7.tar.gz -C /var/www/html/hdp314/

vim ambari.repo
[CentOS7-HDP]
name=CentOS7-HDP
baseurl=http://hadoop121/hdp314/HDP/centos7/3.1.4.0-315
gpgcheck=0
enabled=1
[CentOS7-HDP-GPL]
name=CentOS7-HDP-GPL
baseurl=http://hadoop121/hdp314/HDP-GPL/centos7/3.1.4.0-315
gpgcheck=0
enabled=1
[CentOS7-HDP-UTILS]
name=CentOS7-HDP-UTILS
baseurl=http://hadoop121/hdp314/HDP-UTILS/centos7/1.1.0.22
gpgcheck=0
enabled=1
[CentOS7-ambari]
name=CentOS7-ambari
baseurl=http://hadoop121/hdp314/ambari/centos7/2.7.4.0-118
gpgcheck=0
enabled=1


yum clean all
yum makecache
yum repolist

2. 安裝ambari服務

yum -y install ambari-server

3. 初始化ambari數據庫

mysql> create database ambari DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
Query OK, 1 row affected (0.03 sec)

mysql> use ambari;
Database changed
mysql> CREATE USER 'ambari'@'%' IDENTIFIED BY 'ambari';
Query OK, 0 rows affected (0.02 sec)

mysql> GRANT ALL ON ambari.* TO 'ambari'@'%';
Query OK, 0 rows affected (0.01 sec)

mysql> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.00 sec)

切換ambari用戶登錄mysql進行初始化:

mysql -uambari -pambari
source /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql;

3. 配置ambari

執行配置命令,並按照提示填寫相應信息:

[root@hadoop121 yum.repos.d]# ambari-server setup
Using python  /usr/bin/python
Setup ambari-server
Checking SELinux...
SELinux status is 'enabled'
SELinux mode is 'permissive'
WARNING: SELinux is set to 'permissive' mode and temporarily disabled.
OK to continue [y/n] (y)? n
[root@hadoop121 yum.repos.d]# cat /etc/selinux/config 

# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#     enforcing - SELinux security policy is enforced.
#     permissive - SELinux prints warnings instead of enforcing.
#     disabled - No SELinux policy is loaded.
SELINUX=disabled
# SELINUXTYPE= can take one of three values:
#     targeted - Targeted processes are protected,
#     minimum - Modification of targeted policy. Only selected processes are protected. 
#     mls - Multi Level Security protection.
SELINUXTYPE=targeted 


[root@hadoop121 yum.repos.d]# ambari-server setup
Using python  /usr/bin/python
Setup ambari-server
Checking SELinux...
SELinux status is 'enabled'
SELinux mode is 'permissive'
WARNING: SELinux is set to 'permissive' mode and temporarily disabled.
OK to continue [y/n] (y)? y
Customize user account for ambari-server daemon [y/n] (n)? n
Adjusting ambari-server permissions and ownership...
Checking firewall status...
Checking JDK...
[1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8
[2] Custom JDK
==============================================================================
Enter choice (1): 2
WARNING: JDK must be installed on all hosts and JAVA_HOME must be valid on all hosts.
WARNING: JCE Policy files are required for configuring Kerberos security. If you plan to use Kerberos,please make sure JCE Unlimited Strength Jurisdiction Policy Files are valid on all hosts.
Path to JAVA_HOME: /usr/java/jdk1.8.0_141
Validating JDK on Ambari Server...done.
Check JDK version for Ambari Server...
JDK version found: 8
Minimum JDK version is 8 for Ambari. Skipping to setup different JDK for Ambari Server.
Checking GPL software agreement...
GPL License for LZO: https://www.gnu.org/licenses/old-licenses/gpl-2.0.en.html
Enable Ambari Server to download and install GPL Licensed LZO packages [y/n] (n)? y
Completing setup...
Configuring database...
Enter advanced database configuration [y/n] (n)? y
Configuring database...
==============================================================================
Choose one of the following options:
[1] - PostgreSQL (Embedded)
[2] - Oracle
[3] - MySQL / MariaDB
[4] - PostgreSQL
[5] - Microsoft SQL Server (Tech Preview)
[6] - SQL Anywhere
[7] - BDB
==============================================================================
Enter choice (1): 3
Hostname (localhost): hadoop121
Port (3306): 
Database name (ambari): 
Username (ambari): 
Enter Database Password (bigdata): 
Re-enter password: 
Configuring ambari database...
Should ambari use existing default jdbc /usr/share/java/mysql-connector-java.jar [y/n] (y)? y
Configuring remote database connection properties...
WARNING: Before starting Ambari Server, you must run the following DDL directly from the database shell to create the schema: /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql
Proceed with configuring remote database connection properties [y/n] (y)? y
Extracting system views...
ambari-admin-2.7.4.0.118.jar
....
Ambari repo file doesn't contain latest json url, skipping repoinfos modification
Adjusting ambari-server permissions and ownership...
Ambari Server 'setup' completed successfully.

4. 啟動ambari服務,登錄web界面

[root@hadoop121 yum.repos.d]# ambari-server start
Using python  /usr/bin/python
Starting ambari-server
Ambari Server running with administrator privileges.
Organizing resource files at /var/lib/ambari-server/resources...
Ambari database consistency check started...
Server PID at: /var/run/ambari-server/ambari-server.pid
Server out at: /var/log/ambari-server/ambari-server.out
Server log at: /var/log/ambari-server/ambari-server.log
Waiting for server start..................................................................................
Server started listening on 8080

DB configs consistency check: no errors and warnings were found.
Ambari Server 'start' completed successfully.

初始賬號為admin/admin

三.安裝集群

定義集群名:

選擇本地庫和版本:

我的是centos7,自行選擇對應系統類型進行填寫:

填寫主機列表並自動注冊:

顯示注冊成功:

主機檢測顯示均通過:

選擇需要安裝的組件進行安裝即可,這里已hdfs為例:

分配角色:

安裝步驟跟cdh差不多,根據需要修改配置信息即可:

安裝完成,后面再補充各個組件安裝步驟已經各組件集成方式.
第一次安裝沒遇到任何坑,一路綠燈安裝完成,運氣還是不錯