1.操作系統 CentoOS Server with GUI(有GUI,有瀏覽器*ambari基於瀏覽器*推薦latest stable version), 安裝操作系統前先將bios設置Default一下,然后將legacy改成utfi;
2.分區 默認 + /hadoop
3.網絡設置:內網IP,外網IP,自動網絡連接
4.hostname設置:
/etc/sysconfig/network: NETWORKING=yes
HOSTNAME=master/slave
/etc/hosts:192.168.200.10 master10
192.168.200.11 slave11
192.168.200.12 slave12
/etc/hostname:master10/slave11
logout----------->重新登錄root
5.檢查軟件:which yum 等
6.檢查jdk,卸載openjdk,安裝oracle jdk 1.8+
yum remove *-openjdk-*
下載oracle jdk 1.8.0_77+<用了_101:jdk-8u101-linux-x64.tar.gz>
/etc/profile 環境變量
7.數據庫會自動安裝,所以未提前安裝
8.ulimit -Sn
ulimit -Hn
如果<10000 ulimit -n 10000+
9.ssh配置:
master下:ssh-keygen
cat id_rsa.pub >> authorized_keys
chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys
slave下建立~/.ssh文件夾
scp </.ssh下的id_rsa.pub和authorized_keys【slave上的ambari agent安裝時需要訪問ambari server,所以需要無密碼訪問master,id_rsa也要拷貝】> root@<slave>:~/.ssh/
slave下: chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys
10.時間同步
1. NTP時鍾同步方式說明
NTP在linux下有兩種時鍾同步方式,分別為直接同步和平滑同步:
直接同步
使用ntpdate命令進行同步,直接進行時間變更。如果服務器上存在一個12點運行的任務,當前服務器時間是13點,但標准時間時11點,使用此命令可能會造成任務重復執行。因此使用ntpdate同步可能會引發風 險,因此該命令也多用於配置時鍾同步服務時第一次同步時間時使用。
平滑同步
使用ntpd進行時鍾同步,可以保證一個時間不經歷兩次,它每次同步時間的偏移量不會太陡,是慢慢來的,這正因為這樣,ntpd平滑同步可能耗費的時間比較長。
標准時鍾同步服務
http://www.pool.ntp.org/zone/cn網站包含全球的標准時間同步服務,也包括對中國時間的同步,對應的URL為cn.pool.ntp.org,在其中也描述了ntp配置文件中的建議寫法:
server 1.cn.pool.ntp.org
server 3.asia.pool.ntp.org
server 2.asia.pool.ntp.org
2. 環境情況
准備四台電腦,分別為:
IP 用途
192.168.11.212 ntpd服務器,用於與外部公共ntpd同步標准時間
172.16.248.129 ntpd客戶端,用於與ntpd同步時間
172.16.248.130 ntpd客戶端,用於與ntpd同步時間
172,16,248.131 ntpd客戶端,用於與ntpd同步時間
3. 檢查服務是否安裝
使用rpm檢查ntp包是否安裝
[root@localhost kevin]# rpm -q ntp
ntp-4.2.6p5-19.el7.CentOS.3.x86_64
如果已經安裝則略過此步,否則使用yum進行安裝,並設置系統開機自動啟動並啟動服務
[root@localhost kevin]# yum -y install ntp
[root@localhost kevin]# systemctl enable ntpd
[root@localhost kevin]# systemctl start ntpd
4. 設置ntp服務器: 192.168.11.212
配置前先使用命令:ntpdate -u cn.pool.ntp.org,同步服務器
修改/etc/ntp.conf文件,紅色字體是修改的內容
# For more information about this file, see the man pages
# ntp.conf(5), ntp_acc(5), ntp_auth(5), ntp_clock(5), ntp_misc(5), ntp_mon(5).
driftfile /var/lib/ntp/drift
# Permit time synchronization with our time source, but do not
# permit the source to query or modify the service on this system.
restrict default nomodify notrap nopeer noquery
# Permit all access over the loopback interface. This could
# be tightened as well, but to do so would effect some of
# the administrative functions.
restrict 127.0.0.1
restrict ::1
# Hosts on local network are less restricted.
#restrict 192.168.1.0 mask 255.255.255.0 nomodify notrap
restrict 172.16.248.0 mask 255.255.255.0 nomodify notrap
# Use public servers from the pool.ntp.org project.
# Please consider joining the pool (http://www.pool.ntp.org/join.html).
#server 0.centos.pool.ntp.org iburst
server 2.cn.pool.ntp.org
server 1.asia.pool.ntp.org
server 2.asia.pool.ntp.org
#broadcast 192.168.1.255 autokey # broadcast server
#broadcastclient # broadcast client
#broadcast 224.0.1.1 autokey # multicast server
#multicastclient 224.0.1.1 # multicast client
#manycastserver 239.255.254.254 # manycast server
#manycastclient 239.255.254.254 autokey # manycast client
# 允許上層時間服務器主動修改本機時間
restrict 2.cn.pool.ntp.org nomodify notrap noquery
restrict 1.asia.pool.ntp.org nomodify notrap noquery
restrict 2.asia.pool.ntp.org nomodify notrap noquery
server 127.0.0.1 # local clock
fudge 127.0.0.1 stratum 10
# Enable public key cryptography.
#crypto
includefile /etc/ntp/crypto/pw
# Key file containing the keys and key identifiers used when operating
# with symmetric key cryptography.
keys /etc/ntp/keys
# Specify the key identifiers which are trusted.
#trustedkey 4 8 42
# Specify the key identifier to use with the ntpdc utility.
#requestkey 8
# Specify the key identifier to use with the ntpq utility.
#controlkey 8
# Enable writing of statistics records.
#statistics clockstats cryptostats loopstats peerstats
# Disable the monitoring facility to prevent amplification attacks using ntpdc
# monlist command when default restrict does not include the noquery flag. See
# CVE-2013-5211 for more details.
# Note: Monitoring will not be disabled with the limited restriction flag.
disable monitor
修改完成后重啟ntpd服務systemctl restart ntpd
使用ntpq -p 查看網絡中的NTP服務器,同時顯示客戶端和每個服務器的關系
使用ntpstat 命令查看時間同步狀態,這個一般需要5-10分鍾后才能成功連接和同步。所以,服務器啟動后需要稍等下:
剛啟動的時候,一般是:
# ntpstat
unsynchronised
time server re-starting
polling server every 64 s
連接並同步后:
# ntpstat
synchronised to NTP server (202.112.10.36) at stratum 3
time correct to within 275 ms
polling server every 256 s
5. 設置ntp客戶端: 172.16.248.129|130|131
配置前先使用命令:ntpdate -u cn.pool.ntp.org,同步服務器
安裝ntp服務並設置為自動啟動,和前面的設置方式相同。然后編輯/etc/ntp.conf文件,紅色字體為變化的內容。
# For more information about this file, see the man pages
# ntp.conf(5), ntp_acc(5), ntp_auth(5), ntp_clock(5), ntp_misc(5), ntp_mon(5).
driftfile /var/lib/ntp/drift
# Permit time synchronization with our time source, but do not
# permit the source to query or modify the service on this system.
restrict default nomodify notrap nopeer noquery
# Permit all access over the loopback interface. This could
# be tightened as well, but to do so would effect some of
# the administrative functions.
restrict 127.0.0.1
restrict ::1
# Hosts on local network are less restricted.
#restrict 192.168.1.0 mask 255.255.255.0 nomodify notrap
# Use public servers from the pool.ntp.org project.
# Please consider joining the pool (http://www.pool.ntp.org/join.html).
#server 0.centos.pool.ntp.org iburst
server 172.16.248.1
restrict 172.16.248.1 nomodify notrap noquery
server 127.0.0.1
fudge 127.0.0.1 stratum 10
#broadcast 192.168.1.255 autokey # broadcast server
#broadcastclient # broadcast client
#broadcast 224.0.1.1 autokey # multicast server
#multicastclient 224.0.1.1 # multicast client
#manycastserver 239.255.254.254 # manycast server
#manycastclient 239.255.254.254 autokey # manycast client
# Enable public key cryptography.
#crypto
includefile /etc/ntp/crypto/pw
# Key file containing the keys and key identifiers used when operating
# with symmetric key cryptography.
keys /etc/ntp/keys
# Specify the key identifiers which are trusted.
#trustedkey 4 8 42
# Specify the key identifier to use with the ntpdc utility.
#requestkey 8
# Specify the key identifier to use with the ntpq utility.
#controlkey 8
# Enable writing of statistics records.
#statistics clockstats cryptostats loopstats peerstats
# Disable the monitoring facility to prevent amplification attacks using ntpdc
# monlist command when default restrict does not include the noquery flag. See
# CVE-2013-5211 for more details.
# Note: Monitoring will not be disabled with the limited restriction flag.
disable monitor
重啟ntpd服務
#systemctl restart ntpd
啟動后,查看同步情況
# ntpq -p
# ntpstat
*******注意這些坑*********
*******注意這些坑*********
*******注意這些坑*********
1.同步不了,可能是因為時間差距太大,而且這個能忍受的差距可能要比想象的小得多,所以才需要ntpdate;
2.配置不需要改動,有錯誤提示(127.0.0.1)也不影響,問題一般出在網絡上;
3.多重啟幾次;
11.DNS配置
1.hostname配置前面已完成
2.防火牆(iptables) systemctl disable firewalld
service stop firewalld.service
systemctl status firewalld
chkconfig --list
3.selinux
setenforce 0
/etc/selinux/config<<selinux=disabled //****************重啟后輸入setenforce、getenforce才會出現“selinux is disabled" 的提示
sestatus -v //檢查是否關閉成功
4.echo umask 0022 >> /etc/profile //*******用於配置新建文件和文件夾的默認權限(755)
12.配置yum倉庫
1.yum install httpd
安裝: httpd及依賴apr apr-util http-tools mailcap
systemctl start httpd
報錯:kill:cannot find process "",程序似乎在kill一個不存在的進程,可能跟默認配置有關,未研究,重啟解決
mkdir -p /var/www/html
2.從Hortonworks官方文檔中給出的下載地址下載tar文件:ambari、hdp2.5及對應的repo文件和gpgkey文件
3.解壓到apache http server根目錄ambari:/var/www/html hdp/hdputil:/var/www/html/hdp,並從網頁驗證是否安裝成功:http://master10/AMBARI-2.4.1.0/centos7 http://master10/...
4.修改repo文件並移動到/etc/yum.repo.d/
5.yum repolist
13.ambari server安裝
yum install ambari-server <前面配置完成后此時是從本地庫安裝> ambari-server /dependencies:postgresql/postgresql-libs/postgresql-server
ambari-server setup /用於安裝配置ambari server的數據庫、JDK、進程用戶。=============》》1.選擇用戶:直接回車,使用默認root;2.選擇jdk:3custom-->$JAVA_HOME;4.默認postgreSQL用戶名和密碼:【ambari/bigdata】
14.ambari Server 使用
ambari-server start
ambari-server status
ambari-server stop
***********與以往不同,此次並未在三台主機上分別安裝ambari-agent************
15.安裝HDP
1.選擇服務時:tez和slider與yarn有關,依賴安裝;smartsense作為日志很重要,被推薦安裝;
2.分配masters
3.分配clients
4.hive,smartsense和ambari metrics的密碼設為windey,用戶名分別為hive admin admin
16.安裝vncserver
yum install tigervnc-server -y
chkconfig vncserver on 或者 chkconfig vncserver --level 5 on
注:注冊時找錯ambari-server服務器位置:
主要問題出現在最后兩行。
上網查找原因,,可能的原因有:
1,
地址:http://stackoverflow.com/questions/27531215/ambari-1-7-0-cannot-register-datanodes-in-centos-cluster
說明:查看/etc/ambari-agent/conf/ambari-agent.ini文件,看hostname是否改成本機的主機名;
即,改成對應的master,slave1,slave2;
2,
地址:http://superuser.com/questions/825443/registration-with-the-server-failed-while-configuring-amabri-on-centos-7
說明:CentOS版本問題,使用CentOS7的話會失敗,使用CentOS6,5成功;
3,
地址:http://stackoverflow.com/questions/38423925/ambari-confirm-hosts-step-fails-registration-with-the-server-failed
說明:中文和英文操作系統版本問題,中文操作系統出錯,英文則成功;
Python 2.6 for UTF8 has a bug.
和
I think this is because I set a non-English language (i.e,. Trad. Chinese) as a default language when I installed CentOs 7. It would encounter a charset problem (UTF-8<->ascii) when confirming hosts. After changing the default language to English, This problem has been solved.
解決方法是將系統的默認語言改成英語。