一、環境准備
- centos7 安裝和系統環境處理
1）運行初始化腳本
!/bin/bash
-------------------------------------
系統環境初始化腳本
本腳本僅支持centos， centos7.7驗證成功
使用root用戶運行
------------------------------------
1 依賴安裝
yum clean all
yum makecache
yum update -y
yum install -y openssh-server vim gcc gcc-c++ glibc-headers bzip2-devel lzo-devel curl wget openssh-clients zlib-devel autoconf automake cmake libtool openssl-devel fuse-devel snappy-devel telnet unzip zip net-tools.x86_64 firewalld systemd svn ncurses-devel bzip2 psmisc libxslt-devel libxml2-devel libxslt.so.1 cyrus-sasl-plain yum groupinstall chinese-support mod_ssl ntp
2 關閉防火牆
echo "close firewall ----------------------------------------------------------"
firewall-cmd --state
systemctl status firewalld.service
systemctl stop firewalld.service
systemctl disable firewalld.service
systemctl status firewalld.service
3 關閉SELINUX
echo "close SELINUX ----------------------------------------------------------"
setenforce 0
sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' '/etc/selinux/config'
4 修改UMASK
echo "set umask 0022 ----------------------------------------------------------"
umask 0022
echo umask 0022 >> /etc/profile
5 關閉swappiness
swap_dev=swapon -s | awk '{if (NR>1){print $1}}'
echo "swap device is ${swap_dev} ----------------------------------------------------------"
echo "close swap device ${swap_dev} ----------------------------------------------------------"
swapoff ${swap_dev}
if [ "$?" != 0 ] ; then
echo "close swap device ${swap_dev} failed ----------------------------------------------------------"
else
echo "close swap device ${swap_dev} success ----------------------------------------------------------"
fi
6 資源限制
echo "set ulimit 102400 ----------------------------------------------------------"
ulimit -Hn 102400
ulimit -Sn 102400
echo "* - nproc 102400" >> /etc/security/limits.conf
echo "* - nofile 102400" >> /etc/security/limits.conf
7 時鍾同步
echo "start ntp ----------------------------------------------------------"
systemctl start ntpd
systemctl enable ntpd
systemctl status ntpd
8 禁止huge_page
echo "close huge_page ----------------------------------------------------------"
echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled
echo 'echo never > /sys/kernel/mm/transparent_hugepage/defrag' >> /etc/rc.local
echo 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' >> /etc/rc.local
9 生成公鑰私鑰
echo "ssh-keygen ----------------------------------------------------------"
mkdir ~/.ssh
ssh-keygen -t rsa -f ~/.ssh/id_rsa -P ''
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys
2）如果時區不合適，需要修改時區
[root@centos ~]# timedatectl
3）hostname 修改
[root@centos ~]# hostnamectl set-hostname hdp-master01
4）添加hosts
vim /etc/hosts
5）免密登陸
ambari-server root用戶ssh免密登陸所有其他節點
[root@centos ~]# ssh-copy-id root@node03
2. java8
在 /etc/profile.d/目錄下，增加java.sh
!/bin/bash
java.sh
JAVA_HOME=/usr/local/services/jdk1.8.0_191
JRE_HOME=/usr/local/services/jdk1.8.0_191/jre
PATH=$PATH:$JAVA_HOME/bin:$JRE_HOME/bin
CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$JRE_HOME/lib
export JAVA_HOME JRE_HOME PATH CLASSPATH
3. python2.7.5
yum -y install python-pip
pip install --upgrade pip
4. mysql
Node03:3306
root/12345678
yum install mysql-connector-java*
5. 目錄規划
mkdir /hadoop
將所有磁盤掛載或者鏈接到此目錄下
/hadoop/disk01
/hadoop/disk02
二、建設本地yum repo
有網絡可以直接通過網絡，無網絡需要先拷貝所有需要下載的文件
依賴安裝
yum install yum-utils createrepo httpd
創建目錄
mkdir -p /var/www/html
mkdir -p /var/www/html/ambari
mkdir -p /var/www/html/hdp
下載repo url文件
cd /etc/yum.repos.d/
wget http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.7.4.0/ambari.repo
wget http://public-repo-1.hortonworks.com/HDP/centos7/3.x/updates/3.1.4.0/hdp.repo
查看repo
yum repolist
下載repo文件到本地
cd /var/www/html/ambari
reposync -r ambari-2.7.4.0
cd /var/www/html/hdp
reposync -r HDP-3.1.4.0
reposync -r HDP-UTILS-1.1.0.22
新建本地repo
createrepo /var/www/html/ambari/ambari-2.7.4.0
createrepo /var/www/html/hdp/HDP-3.1.4.0
createrepo /var/www/html/hdp/HDP-UTILS-1.1.0.22
啟動http服務
systemctl start httpd.service
systemctl enable httpd.service
修改repo web地址
[root@node03 ~]#vim /etc/yum.repos.d/ambari.repo
[ambari-2.7.4.0]
baseurl=http://node03/ambari/ambari-2.7.4.0
...
[root@node03 ~]#vim /etc/yum.repos.d/hdp.repo
[HDP-3.1.4.0]
name=HDP Version - HDP-3.1.4.0
baseurl=http://node03/hdp/HDP-3.1.4.0
...
[HDP-UTILS-1.1.0.22]
name=HDP-UTILS Version - HDP-UTILS-1.1.0.22
baseurl=http://node03/hdp/HDP-UTILS-1.1.0.22
...
/etc/yum.repos.d/ambari.repo和hdp.repo需要拷貝到ambari-server將要部署的服務器
三、部署ambari-server
1.准備數據庫
CREATE DATABASE ambari default character set utf8mb4 collate utf8mb4_unicode_ci;
GRANT ALL PRIVILEGES ON . TO 'root'@'%' IDENTIFIED BY '12345678' WITH GRANT OPTION;
1.安裝ambari-server
yum install ambari-server
2.設置
[root@node01 /etc/yum.repos.d]#ambari-server setup
Customize user account for ambari-server daemon [y/n] (n)? y
Enter user account for ambari-server daemon (root):root
Checking JDK...
[1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8
[2] Custom JDK
Enter choice (1): 2
Path to JAVA_HOME: /usr/local/services/jdk1.8.0_191
GPL License for LZO: https://www.gnu.org/licenses/old-licenses/gpl-2.0.en.html
Enable Ambari Server to download and install GPL Licensed LZO packages [y/n] (n)? y
Enter advanced database configuration [y/n] (n)? y
Configuring database...
Choose one of the following options:
[3] - MySQL / MariaDB
Enter choice (1): 3
Hostname (localhost): node03
Port (3306): 3306
Database name (ambari): ambari
Username (ambari): root
Enter Database Password (bigdata):
Re-enter password: 12345678
Should ambari use existing default jdbc /usr/share/java/mysql-connector-java.jar [y/n] (y)? y
Configuring remote database connection properties...
WARNING: Before starting Ambari Server, you must run the following DDL directly from the database shell to create the schema: /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql
Proceed with configuring remote database connection properties [y/n] (y)? y
3.建表
use ambari
source /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql
4.啟動
端口 8080
ambari-server start
ambari-server status
ambari-server stop
ambari-server restart
四、添加組件
注冊 admin/admin
創建集群
選擇版本設置倉庫
添加節點
上傳ambari-server所在服務器的私鑰id_rsa
配置用戶和ssh端口
使用的host不是域名，下一步
注冊所有節點
添加組件
選擇組件實際部署節點
設置grafana 帳號密碼以及Activity 密碼
設置文件目錄（需要提前規划）
設置各個組件的啟動用戶
HDFS服務設置
部署
五、測試
測試hdfs 上傳文件
hadoop fs mkdir /tmp/test/in
hadoop fs -put text.txt /tmp/test/in
測試yarn 運行任務
hadoop jar /usr/hdp/3.1.4.0-315/hadoop-mapreduce/hadoop-mapreduce-examples.jar wordcount /tmp/test/in /tmp/test/out