大數據平台搭建（Ambari +HDP）

安裝包

• ambari-2.7.3.0-centos7.tar.gz
• HDP-3.1.0.0-centos7-rpm.tar.gz
• HDP-UTILS-1.1.0.22-centos7.tar.gz
• HDP-GPL-3.1.0.0-centos7-gpl.tar.gz

Ambari

• Ambari 跟 Hadoop 等開源軟件一樣，也是 Apache Software Foundation 中的一個項目，並且是頂級項目。就 Ambari 的作用來說，就是創建、管理、監視 Hadoop 的集群，但是這里的 Hadoop 指的是 Hadoop 整個生態圈（例如 Hive，Hbase，Sqoop，Zookeeper 等）， 而並不僅是特指 Hadoop。用一句話來說，Ambari 就是為了讓 Hadoop 以及相關的大數據軟件更容易使用的一個管理工具。

• Ambari 自身也是一個分布式架構的軟件，主要由兩部分組成：Ambari Server 和 Ambari Agent。簡單來說，用戶通過 Ambari Server 通知 Ambari Agent 安裝對應的軟件；Agent 會定時地發送各個機器每個軟件模塊的狀態給 Ambari Server，最終這些狀態信息會呈現在 Ambari 的 GUI，方便用戶了解到集群的各種狀態，並進行相應的維護。

HDP

• HDP 是 hortonworks 的軟件棧，里面包含了 Hadoop 生態系統的所有軟件項目，比如 HBase、Zookeeper、Hive、Pig 等

HDP-UTILS

• HDP-UTILS 是工具類庫

准備工作

集群節點規划

主機名	IP	功能	內存	磁盤
yum.hdp	192.168.229.30	Ambari/HDP packages	1G	50G
node-01.hdp	192.168.229.31	Ambari Server/Agent	3G	50G
node-02.hdp	192.168.229.32	Ambari Agent	2G	50G
node-03.hdp	192.168.229.33	Ambari Agent	2G	50G

克隆 4 台虛擬機

修改主機名

# vi /etc/hostname

修改 IP 地址

# vi /etc/sysconfig/network-scripts/ifcfg-ens33
# service network restart
# ping www.baidu.com

hosts 文件添加主機名和IP地址映射

• windows 系統

hosts 文件路徑：C:\Windows\System32\drivers\etc

192.168.229.30 yum.hdp
192.168.229.31 node-01.hdp
192.168.229.32 node-02.hdp
192.168.229.33 ndoe-03.hpd

• centos 系統

[root@node-01 ~]# vi /etc/hosts
192.168.229.30 yum.hdp
192.168.229.31 node-01.hdp
192.168.229.32 node-02.hdp
192.168.229.33 node-03.hdp

配置免密

[root@yum ~]# ssh-keygen -t rsa
[root@yum ~]# ssh-copy-id yum.hdp
[root@yum ~]# ssh-copy-id node-01.hdp
[root@yum ~]# ssh-copy-id node-02.hdp
[root@yum ~]# ssh-copy-id node-03.hdp

[root@node-01 ~]# ssh-keygen -t rsa
[root@node-01 ~]# ssh-copy-id node-01.hdp
[root@node-01 ~]# ssh-copy-id node-02.hdp
[root@node-01 ~]# ssh-copy-id node-03.hdp

[root@node-02 ~]# ssh-keygen -t rsa
[root@node-02 ~]# ssh-copy-id node-02.hdp
[root@node-02 ~]# ssh-copy-id node-03.hdp

安裝 mysql

在 node-01.hdp 主機節點上安裝 mysql，安裝教程

在 yum.hdp 上安裝 Ambari

安裝 yum 相關工具

[root@yum ~]# yum install yum-utils -y
[root@yum ~]# yum repolist
[root@yum ~]# yum install createrepo -y

上傳安裝包到 yum.hdp

創建 hdp 目錄，將安裝包上傳到該目錄下

[root@yum ~]# mkdir hdp

安裝 Apache httpd

httpd 為 Apache HTTP 服務器程序，直接執行程序可啟動服務器的服務

[root@yum ~]# yum install httpd -y

• 安裝完成后，會生成 /var/www/html 目錄（相當於 Tomcat 的 webapps 目錄），進入到 /var/www/html 目錄下，創建 ambari 和 hdp 目錄，用來存放安裝文件

  [root@yum ~]# cd /var/www/html/
  [root@yum html]# mkdir ambari hdp hdp-utils hdp-gpl
  
  [root@yum ~]# cd /root/hdp/
  [root@yum hdp]# tar -zxvf ambari-2.7.3.0-centos7.tar.gz -C /var/www/html/ambari/
  [root@yum hdp]# tar -zxvf HDP-UTILS-1.1.0.22-centos7.tar.gz -C /var/www/html/hdp-utils
  [root@yum hdp]# tar -zxvf HDP-3.1.0.0-centos7-rpm.tar.gz -C /var/www/html/hdp/
  [root@yum hdp]# tar -zxvf HDP-GPL-3.1.0.0-centos7-gpl.tar.gz -C /var/www/html/hdp-gpl/
  

• 啟動 httpd 服務

  # 啟動 httpd 服務
  [root@yum ~]# systemctl start httpd
   Active: active (running) since 三 2021-05-26 15:24:34 CST; 16s ago #表示服務正在運行
       Docs: man:httpd(8)
             man:apachectl(8)
   Main PID: 1247 (httpd)
  # 查看 httpd 服務狀態
  [root@yum ~]# systemctl status httpd
  # 設置 httpd 服務開機啟動
  [root@yum ~]# systemctl enable httpd
  

• 默認端口80，瀏覽器輸入：http://192.168.229.30

配置本地 Repo 數據源

• 創建 ambari.repo 配置文件

  [root@yum yum]# cd /etc/yum.repos.d
  [root@yum yum.repos.d]# vim ambari.repo
  
  #VERSION_NUMBER=2.7.3.0-139
  [ambari-2.7.3.0]
  #json.url = http://public-repo-1.hortonworks.com/HDP/hdp_urlinfo.json
  name=ambari Version - ambari-2.7.3.0
  #baseurl=http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.7.3.0
  baseurl=http://yum.hdp/ambari/ambari/centos7/2.7.3.0-139/
  gpgcheck=1
  #gpgkey=http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.7.3.0/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
  gpgkey=http://yum.hdp/ambari/ambari/centos7/2.7.3.0-139/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
  enabled=1
  priority=1
  

• 創建 hdp.repo 配置文件

  [root@yum yum]# cd /etc/yum.repos.d
  [root@yum yum.repos.d]# vim hdp.repo
  
  #VERSION_NUMBER=3.1.0.0-78
  [HDP-3.1.0.0]
  name=HDP Version - HDP-3.1.0.0
  #baseurl=http://public-repo-1.hortonworks.com/HDP/centos7/3.x/updates/3.1.0.0
  baseurl=http://yum.hdp/hdp/HDP/centos7
  gpgcheck=1
  #gpgkey=http://public-repo-1.hortonworks.com/HDP/centos7/3.x/updates/3.1.0.0/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
  gpgkey=http://yum.hdp/hdp/HDP/centos7/3.1.0.0-78/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
  enabled=1
  priority=1
  
  [HDP-UTILS-1.1.0.22]
  name=HDP-UTILS Version - HDP-UTILS-1.1.0.22
  #baseurl=http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.22/repos/centos7
  baseurl=http://yum.hdp/hdp-utils/HDP-UTILS/centos7
  gpgcheck=1
  #gpgkey=http://public-repo-1.hortonworks.com/HDP/centos7/3.x/updates/3.1.0.0/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
  gpgkey=http://yum.hdp/hdp-utils/HDP-UTILS/centos7/1.1.0.22/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
  enabled=1
  
  [HDP-GPL-3.1.0.0]
  name=HDP-GPL Version - HDP-GPL-3.1.0.0
  baseurl=http://yum.hdp/hdp-gpl/HDP-GPL/centos7
  gpgcheck=1
  gpgkey=http://yum.hdp/hdp-gpl/HDP-GPL/centos7/3.1.0.0-78/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
  enabled=1
  priority=1
  
  

• 分發 ambari.repo 和 hdp.repo 配置文件

  [root@yum yum]# cd /etc/yum.repos.d
  [root@yum yum]# scp ambari.repo node-01.hdp:$PWD
  [root@yum yum]# scp ambari.repo node-02.hdp:$PWD
  [root@yum yum]# scp ambari.repo node-03.hdp:$PWD
  [root@yum yum]# scp hdp.repo node-01.hdp:$PWD
  [root@yum yum]# scp hdp.repo node-02.hdp:$PWD
  [root@yum yum]# scp hdp.repo node-03.hdp:$PWD
  

• 生成本地源

  使用 createrepo 命令，創建 yum 本地源（軟件倉庫），即為存放本地特定位置的眾多rpm包建立索引，描述各包所需依賴信息，並形成元數據

  [root@yum yum]# createrepo /var/www/html/hdp/HDP/centos7/
  Spawning worker 0 with 101 pkgs
  Spawning worker 1 with 100 pkgs
  Workers Finished
  Saving Primary metadata
  Saving file lists metadata
  Saving other metadata
  Generating sqlite DBs
  Sqlite DBs complete
  
  [root@yum yum]# createrepo /var/www/html/hdp-utils/HDP-UTILS/centos7/
  Spawning worker 0 with 8 pkgs
  Spawning worker 1 with 8 pkgs
  Workers Finished
  Saving Primary metadata
  Saving file lists metadata
  Saving other metadata
  Generating sqlite DBs
  Sqlite DBs complete
  
  [root@yum yum.repos.d]# createrepo /var/www/html/hdp-gpl/HDP-GPL/centos7/
  Spawning worker 0 with 2 pkgs
  Spawning worker 1 with 2 pkgs
  Workers Finished
  Saving Primary metadata
  Saving file lists metadata
  Saving other metadata
  Generating sqlite DBs
  Sqlite DBs complete
  

在 node-01.hdp 節點上 mysql 上創建相應的用戶和數據庫

• 創建 ambari 數據庫及數據庫的用戶名和密碼

  [root@node-01 mysql]# mysql -uroot -p
  Enter password: #輸入數據庫密碼
  
  mysql> create database ambari character set utf8;
  Query OK, 1 row affected (0.00 sec)
  
  mysql> CREATE USER 'ambari'@'%'IDENTIFIED BY 'Ambari123';
  Query OK, 0 rows affected (0.00 sec)
  
  mysql>  GRANT ALL PRIVILEGES ON ambari.* TO 'ambari'@'%';
  Query OK, 0 rows affected (0.00 sec)
  
  mysql> FLUSH PRIVILEGES;
  Query OK, 0 rows affected (0.00 sec)
  

• 創建hive數據庫及hive庫的用戶名和密碼

  [root@node-01 mysql]# mysql -uroot -p
  Enter password: #輸入數據庫密碼
  
  mysql> create database hive character set utf8;
  Query OK, 1 row affected (0.00 sec)
  
  mysql> CREATE USER 'hive'@'%'IDENTIFIED BY 'Hive123';
  Query OK, 0 rows affected (0.00 sec)
  
  mysql> GRANT ALL PRIVILEGES ON hive.* TO 'hive'@'%';
  Query OK, 0 rows affected (0.00 sec)
  
  mysql> FLUSH PRIVILEGES;
  Query OK, 0 rows affected (0.00 sec)
  
  

• 創建oozie數據庫及oozie庫的用戶名和密碼

  [root@node-01 mysql]# mysql -uroot -p
  Enter password: #輸入數據庫密碼
  
  mysql> create database oozie character set utf8;
  Query OK, 1 row affected (0.00 sec)
  
  mysql> CREATE USER 'oozie'@'%'IDENTIFIED BY 'Oozie123';
  Query OK, 0 rows affected (0.00 sec)
  
  mysql> GRANT ALL PRIVILEGES ON oozie.* TO 'oozie'@'%';
  Query OK, 0 rows affected (0.00 sec)
  
  mysql> FLUSH PRIVILEGES;
  Query OK, 0 rows affected (0.00 sec)
  

• 創建ranger數據庫及ranger庫的用戶名和密碼（不用創建）

  [root@node-01 mysql]# mysql -uroot -p
  Enter password: #輸入數據庫密碼
  
  mysql> create database ranger character set utf8;
  Query OK, 1 row affected (0.00 sec)
  
  mysql> CREATE USER 'rangeradmin'@'%'IDENTIFIED BY 'Ranger123';
  Query OK, 0 rows affected (0.00 sec)
  
  mysql> GRANT ALL PRIVILEGES ON ranger.* TO 'rangeradmin'@'%';
  Query OK, 0 rows affected (0.00 sec)
  
  mysql> FLUSH PRIVILEGES;
  Query OK, 0 rows affected (0.00 sec)
  

• 查看所有創建的數據庫

  mysql> show databases;
  +--------------------+
  | Database           |
  +--------------------+
  | information_schema |
  | ambari             |
  | hive               |
  | mysql              |
  | oozie              |
  | performance_schema |
  | ranger             |
  | sys                |
  +--------------------+
  8 rows in set (0.00 sec)
  

• 下載 mysql-connection-java

  [root@node-01 mysql]# yum install mysql-connector-java
  
  #查看是否下載 msyql 驅動 jar包
  [root@node-01 mysql]# ls /usr/share/java | grep mysql
  mysql-connector-java.jar
  

在 node-01.hdp 節點上安裝 Ambari-Server

[root@node-01 ~]# yum install ambari-server
[root@node-01 mysql]# ambari-server setup
Using python  /usr/bin/python
Setup ambari-server
Checking SELinux...
SELinux status is 'enabled'
SELinux mode is 'enforcing'
Temporarily disabling SELinux
WARNING: SELinux is set to 'permissive' mode and temporarily disabled.
OK to continue [y/n] (y)? y
Customize user account for ambari-server daemon [y/n] (n)? y
Enter user account for ambari-server daemon (root):root
Adjusting ambari-server permissions and ownership...
Checking firewall status...
Checking JDK...
Do you want to change Oracle JDK [y/n] (n)? y
[1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8
[2] Custom JDK
==============================================================================
Enter choice (1): 2
WARNING: JDK must be installed on all hosts and JAVA_HOME must be valid on all hosts.
WARNING: JCE Policy files are required for configuring Kerberos security. If you plan to use Kerberos,please make sure JCE Unlimited Strength Jurisdiction Policy Files are valid on all hosts.
Path to JAVA_HOME: /root/apps/jdk1.8.0_141
Validating JDK on Ambari Server...done.
Check JDK version for Ambari Server...
JDK version found: 8
Minimum JDK version is 8 for Ambari. Skipping to setup different JDK for Ambari Server.
Checking GPL software agreement...
Completing setup...
Configuring database...
Enter advanced database configuration [y/n] (n)? y
Configuring database...
==============================================================================
Choose one of the following options:
[1] - PostgreSQL (Embedded)
[2] - Oracle
[3] - MySQL / MariaDB
[4] - PostgreSQL
[5] - Microsoft SQL Server (Tech Preview)
[6] - SQL Anywhere
[7] - BDB
==============================================================================
Enter choice (1): 3
Hostname (localhost): node-01.hdp
Port (3306): 3306
Database name (ambari): ambari
Username (root): ambari
Enter Database Password (root):
Re-enter password:
Configuring ambari database...
Should ambari use existing default jdbc /usr/share/java/mysql-connector-java.jar [y/n] (y)? y
Configuring remote database connection properties...
WARNING: Before starting Ambari Server, you must run the following DDL directly from the database shell to create the schema: /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql
Proceed with configuring remote database connection properties [y/n] (y)? y
Extracting system views...
ambari-admin-2.7.3.0.139.jar
....
Ambari repo file contains latest json url http://public-repo-1.hortonworks.com/HDP/hdp_urlinfo.json, updating stacks repoinfos with it...
Adjusting ambari-server permissions and ownership...
Ambari Server 'setup' completed successfully.

• 使用 ambari 登錄數據庫

[root@node-01 mysql]# mysql -uambari -p
Enter password: #密碼為Ambari123
mysql> use ambari;
Database changed
#執行 sql 腳本創建ambari數據庫表
mysql> source /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql;
#查看數據庫表是否創建成功
mysql> show tables;

啟動Ambari-Server 服務

[root@node-01 mysql]# ambari-server start
Using python  /usr/bin/python
Starting ambari-server
Ambari Server running with administrator privileges.
Organizing resource files at /var/lib/ambari-server/resources...
Ambari database consistency check started...
Server PID at: /var/run/ambari-server/ambari-server.pid
Server out at: /var/log/ambari-server/ambari-server.out
Server log at: /var/log/ambari-server/ambari-server.log
Waiting for server start........................................
Server started listening on 8080

DB configs consistency check: no errors and warnings were found.
Ambari Server 'start' completed successfully.

停止 Ambari-Server 服務： # ambari-server stop

安裝Ambari-Agent

[root@node-01 ~]# yum -y install ambari-agent
[root@node-02 ~]# yum -y install ambari-agent
[root@node-03 ~]# yum -y install ambari-agent

訪問Ambari web頁面

默認端口8080，Username：admin；Password：admin；http://192.168.229.31:8080

開始安裝集群

• 啟用安裝向導創建集群

• 配置集群名稱

• 選擇版本

YUM 本地源如下：

HDP-3.1：http://yum.hdp/hdp/HDP/centos7

HDP-3.1-GPL:http://yum.hdp/hdp-gpl/HDP-GPL/centos7

HDP-UTILS-1.1.0.22:http://yum.hdp/hdp-utils/HDP-UTILS/centos7

• 配置集群節點、密鑰

• 主機確認

• 選擇大數據組件

• 節點分配

• 分配從屬與客戶端

• 定制服務

#重新執行命令
[root@node-01 ~]# ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/share/java/mysql-connector-java.jar
Using python  /usr/bin/python
Setup ambari-server
Copying /usr/share/java/mysql-connector-java.jar to /var/lib/ambari-server/resources/mysql-connector-java.jar
If you are updating existing jdbc driver jar for mysql with mysql-connector-java.jar. Please remove the old driver jar, from all hosts. Restarting services that need the driver, will automatically copy the new jar to the hosts.
JDBC driver was successfully initialized.
Ambari Server 'setup' completed successfully.

• 集群整體部署

• 開始安裝（漫長的等待...）

• 安裝完畢

  不用擔心警告，后期可以調整，搭建完成，可以在展示頁面進行查看集群狀態

• 查看監控界面

• 查看服務狀態