cent os 6.5+ambari+HDP集群安裝

1. 搭建一個測試集群，集群有4台機器，配置集群中每一台機器的/etc/hosts文件：

[root@nn .ssh]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
172.28.3.40  nn  nn.hadoop.plat
172.28.3.41  dn1  dn0.hadoop.plat
172.28.3.42  dn2  dn1.hadoop.plat 
172.28.3.43  dn3  dn2.hadoop.plat

2. 配置namenode到datanode ssh免密碼登陸：

在nn上執行，ssh-keygen –t rsa

cd ~/.ssh

cat id_rsa.put >> authorized_keys

對集群中每一台data node執行:  ssh-copy-id root@dn1  ssh-copy-id root@dn2 ssh-copy-id root@dn13

這樣就可以保證，nn節點可以免密碼登陸到dn1, dn2, dn3

[root@nn .ssh]# ifconfig
eth0      Link encap:Ethernet  HWaddr 00:1A:4A:C6:6B:A0  
          inet addr:172.28.3.40  Bcast:172.28.7.255  Mask:255.255.248.0
          inet6 addr: fe80::21a:4aff:fec6:6ba0/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1064845 errors:0 dropped:0 overruns:0 frame:0
          TX packets:557212 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:1578655986 (1.4 GiB)  TX bytes:647178854 (617.1 MiB)

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:103276 errors:0 dropped:0 overruns:0 frame:0
          TX packets:103276 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:58108687 (55.4 MiB)  TX bytes:58108687 (55.4 MiB)

[root@nn .ssh]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
172.28.3.40  nn  nn.hadoop.plat
172.28.3.41  dn1  dn0.hadoop.plat
172.28.3.42  dn2  dn1.hadoop.plat 
172.28.3.43  dn3  dn2.hadoop.plat
[root@nn .ssh]# ssh dn1
SIOCADDRT: File exists

 

3. 關閉iptables

chkconfig iptables off

/etc/init.d/iptables stop

 

4. 關閉seLinux

查看selinux狀態：

[root@localhost ~]# /usr/sbin/sestatus –v

/usr/sbin/setenforce 0 #使SELinux工作模式變成permissive模式
/usr/sbin/setenforce 1 #使SELinux工作模式變成enforcing模式

這樣就可以實時控制SELinux的啟用和不啟用了。

三個參數介紹介紹

• enforcing — The SELinux security policy is enforced.

• permissive — The SELinux system prints warnings but does not enforce policy.

• disabled — SELinux is fully disabled. SELinux hooks are disengaged from the kernel and the pseudo-file system is unregistered.

永久關閉SELinux

編輯/etc/selinux/config，找到SELINUX 行修改成為：SELINUX=disabled:

# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
# enforcing - SELinux security policy is enforced.
# permissive - SELinux prints warnings instead of enforcing.
# disabled - No SELinux policy is loaded.
SELINUX=disabled
# SELINUXTYPE= can take one of these two values:
# targeted - Only targeted network daemons are protected.
# strict - Full SELinux protection.
SELINUXTYPE=targeted 

如果重啟系統，就會發現SELinux的狀態變成disabled

 

5. 關閉linux內核huge_page:

Add the following lines in /etc/rc.local and reboot the server:
echo never > /sys/kernel/mm/redhat_transparent_hugepage/enabled
echo never > /sys/kernel/mm/redhat_transparent_hugepage/defrag

 

6. 安裝java，配置JAVA_HOME

mkdir /usr/java
cd /usr/java
rz -be
/usr/java/jdk1.7.0_75
ln -s /usr/java/jdk1.7.0_75 /usr/java/default

vim /etc/profile

route add default gw 172.28.0.1
export JAVA_HOME=/usr/java/default
export PATH=$JAVA_HOME/bin:$PATH

編輯好/etc/profile，執行source /etc/profile使配置生效，保證每一台機器上的java版本都是一致的，並且JAVA_HOME環境變量是有效的：

 

7. 每台機器上安裝ntpd

rpm -aq | grep ntpd
yum install ntpd
chkconfig ntpd on
service ntpd start

確保每台機器的ntpd服務都處於運行狀態：

8. 確保機器上安裝了openssh-server，並且升級openssl到最新：

rpm -qa | grep ssh
yum install openssh-server
service sshd restart
chkconfig sshd on

確保openssl最新：

yum install openssl-devel-1.0.1e-42.el6.x86_64

 

9. yum源確保可以用，本次安裝采用了163的yum源，先將/etc/yum.repos.d/中，所有的文件都刪掉，然后新建文件CentOS6-Base-163.repo，填入如下內容：

# CentOS-Base.repo
#
# The mirror system uses the connecting IP address of the client and the
# update status of each mirror to pick mirrors that are updated to and
# geographically close to the client.  You should use this for CentOS updates
# unless you are manually picking other mirrors.
#
# If the mirrorlist= does not work for you, as a fall back you can try the
# remarked out baseurl= line instead.
#
#

[base]
name=CentOS-$releasever - Base - 163.com
baseurl=http://mirrors.163.com/centos/$releasever/os/$basearch/
#mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=os
gpgcheck=1
gpgkey=http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-6

#released updates
[updates]
name=CentOS-$releasever - Updates - 163.com
baseurl=http://mirrors.163.com/centos/$releasever/updates/$basearch/
#mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=updates
gpgcheck=1
gpgkey=http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-6

#additional packages that may be useful
[extras]
name=CentOS-$releasever - Extras - 163.com
baseurl=http://mirrors.163.com/centos/$releasever/extras/$basearch/
#mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=extras
gpgcheck=1
gpgkey=http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-6

#additional packages that extend functionality of existing packages
[centosplus]
name=CentOS-$releasever - Plus - 163.com
baseurl=http://mirrors.163.com/centos/$releasever/centosplus/$basearch/
#mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=centosplus
gpgcheck=1
enabled=0
gpgkey=http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-6

#contrib - packages by Centos Users
[contrib]
name=CentOS-$releasever - Contrib - 163.com
baseurl=http://mirrors.163.com/centos/$releasever/contrib/$basearch/
#mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=contrib
gpgcheck=1
enabled=0
gpgkey=http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-6

 

配置ambari的yum源，本配置源是部署在本地局域網中的一台apache服務器上的：

[root@hdp159 yum.repos.d]# cat ambari.repo 
[Updates-ambari-2.0.1]
name=ambari-2.0.1 - Updates
baseurl=http://172.28.4.159/ambari-test/centos6
gpgcheck=1
gpgkey=http://172.28.4.159/ambari-test/centos6/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1
[root@hdp159 yum.repos.d]#

 

將yum源配置好好了后，執行如下命令：

yum clean all
yum repolist

 

10. 在nn機器上，安裝ambari-servier，執行如下命令，之所以加—nogpgcheck參數，是因為此處安裝的是公司修改后的ambari，如果是安裝原生的ambari，不用加該選項：

yum install --nogpgcheck ambari-server

 

11.配置和啟動ambari-server, setup –j 配置ambari-server要使用的java環境：

 

ambari-server setup -j /usr/java/default
ambari-server start

 

12. 在瀏覽器中，輸入nn:8080進入ambari的登陸頁面，用戶名和密碼都是admin:

13. 配置HDP的redhat6的baseURL,此處使用的是本地局域網中的HDP安裝源：

http://172.28.4.159/HDP2.2.6/HDP/centos6/2.x/updates/2.2.6.0/

http://172.28.4.159/HDP2.2.6/HDP-UTILS-1.1.0.20/repos/centos6/

14. 將nn節點下，/root/.ssh/id_rsa文件上傳到ambari中，配置好Target Hosts:

 

15. 發現一個警告消息，在每一台機器上執行如下命令，消除警告：

[root@dn3 yum.repos.d]# python /usr/lib/python2.6/site-packages/ambari_agent/HostCleanup.py --silent --skip=users
INFO:HostCleanup:
Killing pid's: ['']

INFO:HostCleanup:Deleting packages: ['']

INFO:HostCleanup:
Deleting directories: ['']
INFO:HostCleanup:Path doesn't exists:
INFO:HostCleanup:
Deleting additional directories: ['']
INFO:HostCleanup:
Deleting repo files: []
INFO:HostCleanup:
Erasing alternatives:{'symlink_list': [''], 'target_list': ['']}

INFO:HostCleanup:Path doesn't exists:
INFO:HostCleanup:Clean-up completed. The output is at /var/lib/ambari-agent/data/hostcleanup.result

 

16. 分配slaves和clients:

設置好，hive和oozie數據庫的用戶名和密碼：

查看總結信息：

 

17. 進入安裝流程，最后安裝成功：