Ambari簡介
Ambari概述
Apache Ambari是一種基於Web的工具，支持Apache Hadoop集群的創建、管理和監控。Ambari已支持大多數Hadoop組件，包括HDFS、MapReduce、Hive、Pig、 Hbase、Zookeeper、Sqoop和Hcatalog等；除此之外，Ambari還支持Spark、Storm等計算框架及資源調度平台YARN。
Apache Ambari 從集群節點和服務收集大量信息，並把它們表現為容易使用的，集中化的接口：Ambari Web.
Ambari Web顯示諸如服務特定的摘要、圖表以及警報信息。可通過Ambari Web對Hadoop集群進行創建、管理、監視、添加主機、更新服務配置等；也可以利用Ambari Web執行集群管理任務，例如啟用 Kerberos 安全以及執行Stack升級。任何用戶都可以查看Ambari Web特性。擁有administrator-level 角色的用戶可以訪問比 operator-level 或 view-only 的用戶能訪問的更多選項。例如，Ambari administrator 可以管理集群安全，一個 operator 用戶可以監控集群，而 view-only 用戶只能訪問系統管理員授予他的必要的權限。
Ambari體系結構
Ambari 自身也是一個分布式架構的軟件，主要由兩部分組成：Ambari Server 和 Ambari Agent。簡單來說，用戶通過Ambari Server通知 Ambari Agent 安裝對應的軟件；Agent 會定時地發送各個機器每個軟件模塊的狀態給 Ambari Server，最終這些狀態信息會呈現在 Ambari 的 GUI，方便用戶了解到集群的各種狀態，並進行相應的維護。
Ambari Server 從整個集群上收集信息。每個主機上都有 Ambari Agent, Ambari Server 通過 Ambari Agent 控制每部主機。
Ambari安裝
http://doc.ailinux.net/docs/data/data-1ardi0l817b5q
安裝准備階段：
本文介紹ambari安裝及常用大數據組件安裝配置等HDP版本兼容性
CentOS 7 Python 2.7.x JDK1.8+ Mariadb 5.5 Ambari2.6.1.5 HDP 2.6.4查看一下地址獲取軟件版本兼容性支持
https://supportmatrix.hortonworks.com/
由於嚴格依賴版本，所以要先確定安裝的版本。
系統初始化
更新yum源
yum install -y wget wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo yum clean all yum makecache修改主機名
sudo hostnamectl set-hostname test-bg-xx sudo hostname test-bg-xx修改時區：
cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime同步集群時間
yum install –y ntp
ntpdate -u cn.pool.ntp.org
或：
ntpdate time1.aliyun.com
### 硬件時間和系統時間同步: clock -w 配置自動同步:/etc/crontab寫入 */1 * * * * root ntpdate -u cn.pool.ntp.org >> /dev/null 2>&1關閉防火牆和selinux
#關閉防火牆 systemctl stop firewalld systemctl disable firewalld #關閉selinux 修改配置 sed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/sysconfig/selinux 臨時關閉 setenforce 0配置免密鑰登錄
為了讓Ambari服務器在所有集群主機上自動安裝Ambari代理，您必須在Ambari服務器主機和集群中的所有其他主機之間設置無密碼SSH連接。Ambari服務器主機使用SSH公鑰認證來遠程訪問和安裝Ambari代理
ssh-keygen -t rsa
ssh-copy-id -i ~/.ssh/id_rsa.pub root@node1 ssh-copy-id -i ~/.ssh/id_rsa.pub root@node2 ssh-copy-id -i ~/.ssh/id_rsa.pub root@node3 注意：您可以選擇在每個集群主機上手動安裝Ambari代理。在這種情況下，您不需要生成和分發SSH密鑰 可以使用非root SSH帳戶，如果該帳戶可以執行sudo而不輸入密碼 # #如果ssh端口不是默認，可以在.ssh下創建config文件,修改文件權限 配置如下: touch config chmod 600 config [jt_ops@test-bg-m01 .ssh]$ cat config Host test-bg-m01 HostName test-bg-m01 User jt_ops Port xx Host test-bg-w01 HostName test-bg-w01 User jt_ops Port xx 也可以修改/etc/sshd/ssh_config配置文件中的端口系統初始化參數
cat >> /etc/sysctl.conf << EOF fs.file-max=1000000 net.ipv4.tcp_max_tw_buckets = 6000 net.ipv4.tcp_sack = 1 net.ipv4.tcp_window_scaling = 1 net.ipv4.tcp_rmem = 4096 87380 4194304 net.ipv4.tcp_wmem = 4096 16384 4194304 net.ipv4.tcp_max_syn_backlog = 16384 net.core.netdev_max_backlog = 32768 net.core.somaxconn = 32768 net.core.wmem_default = 8388608 net.core.rmem_default = 8388608 net.core.rmem_max = 16777216 net.core.wmem_max = 16777216 net.ipv4.tcp_timestamps = 1 net.ipv4.tcp_fin_timeout = 20 net.ipv4.tcp_synack_retries = 2 net.ipv4.tcp_syn_retries = 2 net.ipv4.tcp_syncookies = 1 #net.ipv4.tcp_tw_len = 1 net.ipv4.tcp_tw_reuse = 1 net.ipv4.tcp_mem = 94500000 915000000 927000000 net.ipv4.tcp_max_orphans = 3276800 net.ipv4.ip_local_port_range = 1024 65000 net.nf_conntrack_max = 6553500 net.netfilter.nf_conntrack_max = 6553500 net.netfilter.nf_conntrack_tcp_timeout_close_wait = 60 net.netfilter.nf_conntrack_tcp_timeout_fin_wait = 120 net.netfilter.nf_conntrack_tcp_timeout_time_wait = 120 net.netfilter.nf_conntrack_tcp_timeout_established = 3600 net.ipv6.conf.all.disable_ipv6 = 1 net.ipv6.conf.default.disable_ipv6 = 1 kernel.numa_balancing = 0 kernel.shmmax = 68719476736 kernel.printk = 5 kernel.sysrq = 1 vm.overcommit_memory = 0 vm.swappiness = 0 EOF文件描述符和用戶最大進程數
### 調整文件描述符 cat >> /etc/security/limits.conf <<EOF * soft nproc 65535 * hard nproc 65535 * soft nofile 65535 * hard nofile 65535 EOF ##用戶進程限制 cat >> /etc/security/limits.d/20-nproc.conf <<EOF * soft nproc 8192 root soft nproc unlimited EOF安裝
安裝有兩種方法：
在線安裝和搭建本地yum源倉庫安裝在線安裝
下載yum源配置文件repo
#如果做了免秘鑰登陸，則只需要在主節點下載yum源 sudo wget -nv http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.6.1.5/ambari.repo -O /etc/yum.repos.d/ambari.repo sudo wget –nv http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.6.4.0/hdp.repo -O /etc/yum.repos.d/hdp.repo sudo wget -nv http://public-repo-1.hortonworks.com/HDP-GPL/centos7/2.x/updates/2.6.4.0/hdp.gpl.repo -O /etc/yum.repos.d/hdp.gpl.repo離線安裝
制作本地源
制作本地鏡像源安裝HDP，可以在主節點或者單獨的一台服務器（建議）
選擇一台可以訪問internet的服務器配置本地源
apache httpd
提供離線倉庫的下載服務
#安裝apache yum install httpd –y 下載軟件源 在httpd網站根目錄,默認是即/var/www/html/，創建目錄ambari， 並且將下載的壓縮包解壓到/var/www/html/ambari目錄 cd /var/www/html/ mkdir ambari cd /var/www/html/ambari/ http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.6.1.5/ambari-2.6.1.5-centos7.tar.gz http://public-repo-1.hortonworks.com/HDP/centos6/2.x/updates/2.6.4.0/HDP-2.6.4.0-centos6-rpm.tar.gz http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.22/repos/centos6/HDP-UTILS-1.1.0.22-centos6.tar.gz http://public-repo-1.hortonworks.com/HDP-GPL/centos6/2.x/updates/2.6.4.0/HDP-GPL-2.6.4.0-centos6-rpm.tar.gz 解壓下載的tar包 [root@dev-bg-m01 html]# cd /var/www/html/ [root@dev-bg-m01 html]# pwd /var/www/html [root@dev-bg-m01 html]# [root@dev-bg-m01 html]# tree -L 2 ambari/ ambari/ ├── ambari │ └── centos7 ├── HDP │ └── centos7 ├── HDP-GPL │ └── centos7 └── HDP-UTILS-1.1.0.22 ├── hdp-utils.repo ├── openblas ├── repodata ├── RPM-GPG-KEY └── snappy 11 directories, 1 file [root@dev-bg-m01 html]#編輯ambari yum源文件
[root@dev-bg-m01 yum.repos.d]# cat ambari.repo #VERSION_NUMBER=2.6.1.5-3 [ambari-2.6.1.5] name=ambari Version - ambari-2.6.1.5 baseurl=http://192.168.103.3/ambari/ambari/centos7/2.6.1.5-3/ gpgcheck=1 gpgkey=http://192.168.103.3/ambari/ambari/centos7/2.6.1.5-3/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins enabled=1 priority=1 [root@dev-bg-m01 yum.repos.d]# [root@dev-bg-m01 yum.repos.d]# cat hdp.repo #VERSION_NUMBER=2.6.4.0-91 [HDP-2.6.4.0] name=HDP Version - HDP-2.6.4.0 baseurl=http://192.168.103.3/ambari/HDP/centos7/2.6.4.0-91/ gpgcheck=1 gpgkey=http://192.168.103.3/ambari/HDP/centos7/2.6.4.0-91/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins enabled=1 priority=1 [HDP-UTILS-1.1.0.22] name=HDP-UTILS Version - HDP-UTILS-1.1.0.22 baseurl=http://192.168.103.3/ambari/HDP-UTILS-1.1.0.22/ gpgcheck=1 gpgkey=http://192.168.103.3/ambari/HDP-UTILS-1.1.0.22/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins enabled=1 priority=1 [root@dev-bg-m01 yum.repos.d]# [root@dev-bg-m01 yum.repos.d]# cat hdp.gpl.repo [HDP-2.6-GPL-repo-1] name=HDP-2.6-GPL-repo-1 baseurl=http://192.168.103.3/ambari/HDP-GPL/centos7/2.6.4.0-91/ gpgkey=http://192.168.103.3/ambari/HDP-GPL/centos7/2.6.4.0-91/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins path=/ enabled=1 gpgcheck=0 [root@dev-bg-m01 yum.repos.d]# 更新yum 元數據 yum clean all yum makecache yum repolist安裝Ambari-server
在主節點安裝ambari-server
sudo yum install -y ambari-server配置Ambari-server
配置mysql連接器
由於ambari要訪問數據庫，所以要添加lib包，請自行去mysql官網下載
sudo mkdir -p /usr/share/java
sudo cp /opt/mysql-connector-java-5.1.47/mysql-connector-java-5.1.47-bin.jar /usr/share/java/ 在ambari-server配置文件中添加mysql lib包路徑地址 執行一下命令配置jdbc環境 ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/share/java/mysql-connector-java-5.1.47-bin.jar vim /etc/ambari-server/conf/ambari.properties ###在jdbc區下面找個位置添加如下如下語句 server.jdbc.driver.path=/usr/share/java/mysql-connector-java-5.1.47-bin.jar 檢查環境並配置 執行以下命令 sudo ambari-server setup 按提示操作 會檢查selinux，是否自定義用戶，檢查防火牆，設置JDK路徑，數據庫配置等，按提示操作即可 具體操作如下： [root@dev-bg-m01 opt]# vim /etc/ambari-server/conf/ambari.properties [root@dev-bg-m01 opt]# ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/share/java/mysql-connector-java-5.1.47-bin.jar Using python /usr/bin/python Setup ambari-server Copying /usr/share/java/mysql-connector-java-5.1.47-bin.jar to /var/lib/ambari-server/resources If you are updating existing jdbc driver jar for mysql with mysql-connector-java-5.1.47-bin.jar. Please remove the old driver jar, from all hosts. Restarting services that need the driver, will automatically copy the new jar to the hosts. JDBC driver was successfully initialized. Ambari Server 'setup' completed successfully. [root@dev-bg-m01 opt]# vim /etc/ambari-server/conf/ambari.properties [root@dev-bg-m01 opt]# ambari-server setup Using python /usr/bin/python Setup ambari-server Checking SELinux... SELinux status is 'disabled' Customize user account for ambari-server daemon [y/n] (n)? y Enter user account for ambari-server daemon (root):ambari Adjusting ambari-server permissions and ownership... Checking firewall status... Checking JDK... Do you want to change Oracle JDK [y/n] (n)? y [1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8 [2] Oracle JDK 1.7 + Java Cryptography Extension (JCE) Policy Files 7 [3] Custom JDK ============================================================================== Enter choice (1): 3 WARNING: JDK must be installed on all hosts and JAVA_HOME must be valid on all hosts. WARNING: JCE Policy files are required for configuring Kerberos security. If you plan to use Kerberos,please make sure JCE Unlimited Strength Jurisdiction Policy Files are valid on all hosts. Path to JAVA_HOME: /opt/jdk1.8.0_161 Validating JDK on Ambari Server...done. Checking GPL software agreement... Completing setup... Configuring database... Enter advanced database configuration [y/n] (n)? y Configuring database... ============================================================================== Choose one of the following options: [1] - PostgreSQL (Embedded) [2] - Oracle [3] - MySQL / MariaDB [4] - PostgreSQL [5] - Microsoft SQL Server (Tech Preview) [6] - SQL Anywhere [7] - BDB ========================================== Enter choice (1