Cloudera Manager安裝前准備
（一）機器准備
192.168.80.148 clouderamanager01 （部署ClouderaManager-server和Mirror server） （部署Agent） （分配1G，我這里只是體驗過程，當然你可以分配更多或更小)
192.168.80.149 clouderamanager02 （部署Agent） （分配1G，我這里只是體驗過程，當然你可以分配更多或更小)
192.168.80.150 clouderamanager03 （部署Agent） （分配1G，我這里只是體驗過程，當然你可以分配更多或更小)
注意。我這里，因為安裝了兩種情況。所以在寫博客時，后面全部是用 clouderamanager01、 clouderamanager02和 clouderamanager03（注意是小寫）！！！
下面這種，也是可以的
192.168.80.148 clouderamanager01 （部署ClouderaManager-server和Mirror server） （分配1G，我這里只是體驗過程，當然你可以分配更多或更小)
192.168.80.149 clouderamanager02 （部署Agent） （分配1G，我這里只是體驗過程，當然你可以分配更多或更小)
192.168.80.150 clouderamanager03 （部署Agent） （分配1G，我這里只是體驗過程，當然你可以分配更多或更小)
192.168.80.151 clouderamanager04 （部署Agent） （分配1G，我這里只是體驗過程，當然你可以分配更多或更小)
說白了，就是在clouderamanager01上安裝了ClouderaManger。
就是在clouderamanager01上安裝了cdh單節點集群。
就是在clouderamanager01、clouderamanager02和clouderamanager03上安裝了cdh的3節點集群。
（二）機器的部署配置
這里我采用克隆的方式，當然大家也可以單獨搭建linux虛擬機。
VMware workstation 11 的下載
VMWare Workstation 11的安裝
CentOS 6.5的安裝詳解
CentOS 6.5安裝之后的網絡配置
CentOS 6.5靜態IP的設置（NAT和橋接聯網方式都適用）
CentOS常用命令、快照、克隆大揭秘
（1）修改主機名
現在的主機名是clouderaManager01，拿這個來說，其他的clouderaManager02、clouderaManager03和clouderaManager04是一樣的。當然我也可以輸入命令來查看一下主機名，我們輸入hostname,我們發現主機名就是clouderaManager01，那么接下來我們就來修改一下主機名。
常見的有三種方法，一種是臨時性的修改，但是當機器重啟之后就會恢復原來的值，比如我們試一下。
還有一種方法就是永久修改，我們直接修改配置文件，我們直接修改/etc/sysconfig/network
實際上還有一種方法，修改/etc/hosts,他對網絡中的主機名起作用，但是對系統本身（也就是本機）不起作用。他和第一種的效果是一樣的。實際上我們一般用這個來配置主機名和IP之間的對應關系。
我這里采用第二種方法。改完之后沒有變化，我們使用reboot命令重啟一下機器就可以了。
（2）Linux虛擬機的網絡配置和系統配置
我這里，僅拿clouderaManager01來說，其他的clouderaManager02、clouderaManager03和clouderaManager04是一樣的。
（1）ping回環地址：能ping通，說明虛擬機的網絡協議是好的。
(2)ping網關
點擊“編輯”——》“虛擬網絡編輯器”
但是這里邊要注意一個問題，因為我們是克隆的。所以我們還要改一下MAC地址（就是我們硬件的地址，用來定義網絡設備的位置），因為我們新的MAC地址和我們網卡配置里的MAC地址不一致，所以會導致網絡連接不上。
所以我們先來查看一下我們新的MAC地址
我們輸入vi /etc/udev/rules.d/70-persistent-net.rules(如果你是完全克隆，這個文件里應該包含eth0和eth1，我們將eth0配置刪除，只保留eth1,記錄該配置對應的MAC地址，並將eth1改名為eth0
接下來修改網卡vi /etc/sysconfig/network-scripts/ifcfg-eth0
你要是新搭建的Linux虛擬機的時候只需要把ONBOOT 的no改為yes就行，（開啟我們的網卡，我們才能夠看到我們的IP地址，BOOTPROTO=dhcp,指定獲取IP的方法，dhcp(動態主機配置協議)還有一種是static ，靜態的）
改回dhcp之后要重啟一下，否則還是查看不到IP地址。(或者用service network restart命令)
OK，前面的修改完之后，下面我們就輸入ifconfig來查看一下IP，然后我們ping IP
接下來我們再來ping www.baidu.com
OK,都能ping通，這就說明我們已經和外部網絡打通了。
（3）X-shell遠程連接
Xmanager Enterprise *安裝步驟
我這里，僅拿clouderaManager01來說，其他的clouderaManager02和clouderaManager03是一樣的。
（4）用戶設置：hadoop用戶
新建用戶組、用戶、用戶密碼、刪除用戶組、用戶（適合CentOS、Ubuntu）
我這里，是新建的hadoop用戶。
我這里，僅拿clouderaManager01來說，其他的clouderaManager02、clouderaManager03和clouderaManager04是一樣的。
（5）為hadoop用戶設置sudo權限
在root用戶下，輸入visudo打開文件
在文件末尾（按快捷鍵，Shift + g）添加hadoop ALL=(ALL) NOPASSWD:ALL
hadoop ALL=(ALL) NOPASSWD:ALL
我這里，僅拿clouderaManager01來說，其他的clouderaManager02、clouderaManager03和clouderaManager04是一樣的。
（6）配置靜態IP
現在的主機名是clouderaManager01，拿這個來說，其他的clouderaManager02和clouderaManager03是一樣的。
vi /etc/sysconfig/network-scripts/ifcfg-eth0
（7）查看並關閉防火牆
CentOS下的防火牆關閉
先查看一下防火牆狀態
service iptables status
如果防火牆在運行，就關閉，也有兩種方法。
一種是臨時關閉
service iptables stop
一種是永久關閉
chkconfig iptables off
這里我采用永久性關閉。
同理，現在的主機名是clouderaManager01，拿這個來說，其他的clouderaManager02和clouderaManager03是一樣的。
（8）關閉SELINUX
hadoop 50070 無法訪問問題解決匯總
sudo vi /etc/selinux/config
把SELINUX改為disabled
然后重啟reboot。
同理，現在的主機名是clouderaManager01，拿這個來說，其他的clouderaManager02和clouderaManager03是一樣的。
（9）相關服務的安裝
如果沒有安裝的話可以使用下面的命令安裝一下。
[root@clouderaManager01 hadoop]# yum -y install lrzsz
[root@clouderaManager01 hadoop]# yum install -y openssh-clients
同理，現在的主機名是clouderaManager01，拿這個來說，其他的clouderaManager02和clouderaManager03是一樣的。
（10）時鍾同步
[root@clouderaManager01 hadoop]# cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime cp: overwrite `/etc/localtime'? y [root@clouderaManager01 hadoop]# yum -y install ntp
[root@clouderaManager01 hadoop]# cd /usr/share/zoneinfo/Asia/ [root@clouderaManager01 Asia]# ntpdate pool.ntp.org 6 Apr 20:30:19 ntpdate[2938]: step time server 173.255.215.209 offset -86417.188942 sec [root@clouderaManager01 Asia]# date Thu Apr 6 20:30:23 CST 2017 [root@clouderaManager01 Asia]#
同理，現在的主機名是clouderaManager01，拿這個來說，其他的clouderaManager02和clouderaManager03是一樣的。
（11）修改hosts文件
vi /etc/hosts
在里面添加主機名和IP之間的對應關系
192.168.80.148 clouderaManager01 192.168.80.149 clouderaManager02 192.168.80.150 clouderaManager03
同理，現在的主機名是clouderaManager01，拿這個來說，其他的clouderaManager02和clouderaManager03是一樣的。
（12）配置ssh免密碼登錄 （對於這一步，其實可以不做，因為，在CM時會自動安裝，大家可以跳過）
關於這一步，請移步，先看看再說，細加小心。
hadoop-2.6.0.tar.gz的集群搭建（5節點）
這里我以clouderaManager01為例來配置ssh。拿這個來說，其他的clouderaManager02、clouderaManager03和clouderaManager04是一樣的。
[hadoop@clouderaManager01 ~]$ pwd /home/hadoop [hadoop@clouderaManager01 ~]$ mkdir .ssh [hadoop@clouderaManager01 ~]$ ssh-keygen -t rsa Generating public/private rsa key pair. Enter file in which to save the key (/home/hadoop/.ssh/id_rsa): Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /home/hadoop/.ssh/id_rsa. Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub. The key fingerprint is: cd:51:91:1e:95:47:b0:44:2e:fe:75:a5:ac:44:76:86 hadoop@clouderaManager01 The key's randomart image is: +--[ RSA 2048]----+ | o==+.| | .o+...| | ..E.=..| | o =.= ..| | S o o o o| | . o ..| | . . | | | | | +-----------------+ [hadoop@clouderaManager01 ~]$ cd .ssh [hadoop@clouderaManager01 .ssh]$ ls id_rsa id_rsa.pub [hadoop@clouderaManager01 .ssh]$ cat id_rsa.pub >> authorized_keys [hadoop@clouderaManager01 .ssh]$ ls authorized_keys id_rsa id_rsa.pub [hadoop@clouderaManager01 .ssh]$
[hadoop@clouderaManager01 .ssh]$ ls
authorized_keys id_rsa id_rsa.pub
[hadoop@clouderaManager01 .ssh]$ cd ..
[hadoop@clouderaManager01 ~]$ chmod 700 .ssh
[hadoop@clouderaManager01 ~]$ chmod 600 .ssh/*
[hadoop@clouderaManager01 ~]$ ssh clouderaManager01
The authenticity of host 'clouderamanager01 (192.168.80.148)' can't be established.
RSA key fingerprint is 99:42:d9:30:7a:ea:e5:db:9d:ae:53:42:ea:ea:1a:75.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'clouderamanager01,192.168.80.148' (RSA) to the list of known hosts.
[hadoop@clouderaManager01 ~]$ exit;
logout
Connection to clouderaManager01 closed.
[hadoop@clouderaManager01 ~]$ ssh clouderaManager01
Last login: Thu Apr 6 20:37:16 2017 from clouderamanager01
[hadoop@clouderaManager01 ~]$ exit;
logout
Connection to clouderaManager01 closed.
[hadoop@clouderaManager01 ~]$
將所有節點（clouderaManager02、clouderaManager03和clouderaManager04）中的共鑰id_ras.pub拷貝到clouderaManager01中的authorized_keys文件中。
[hadoop@clouderaManager02 ~]$ cat ~/.ssh/id_rsa.pub | ssh hadoop@clouderaManager01 'cat >> ~/.ssh/authorized_keys' The authenticity of host 'clouderamanager01 (192.168.80.148)' can't be established. RSA key fingerprint is 99:42:d9:30:7a:ea:e5:db:9d:ae:53:42:ea:ea:1a:75. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'clouderamanager01,192.168.80.148' (RSA) to the list of known hosts. hadoop@clouderamanager01's password: [hadoop@clouderaManager02 ~]$
[hadoop@clouderaManager03 ~]$ cat ~/.ssh/id_rsa.pub | ssh hadoop@clouderaManager01 'cat >> ~/.ssh/authorized_keys' The authenticity of host 'clouderamanager01 (192.168.80.148)' can't be established. RSA key fingerprint is 99:42:d9:30:7a:ea:e5:db:9d:ae:53:42:ea:ea:1a:75. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'clouderamanager01,192.168.80.148' (RSA) to the list of known hosts. hadoop@clouderamanager01's password: [hadoop@clouderaManager03 ~]$
[hadoop@clouderaManager04 ~]$ cat ~/.ssh/id_rsa.pub | ssh hadoop@clouderaManager01 'cat >> ~/.ssh/authorized_keys' The authenticity of host 'clouderamanager01 (192.168.80.148)' can't be established. RSA key fingerprint is 99:42:d9:30:7a:ea:e5:db:9d:ae:53:42:ea:ea:1a:75. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'clouderamanager01,192.168.80.148' (RSA) to the list of known hosts. hadoop@clouderamanager01's password: [hadoop@clouderaManager04 ~]$
得到
[hadoop@clouderaManager01 ~]$ cd .ssh [hadoop@clouderaManager01 .ssh]$ ls authorized_keys id_rsa id_rsa.pub known_hosts [hadoop@clouderaManager01 .ssh]$ cat authorized_keys ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAswjj3hTe9kqHOIA0vBu567OWLvEASOlum5NceSXrTFbL/9oG4mWZPcHJrgfLMEkHeJt+ZzT/7PtnDk2VjZS8cRBTbAxv3FQoqVLZtfZh9h+NfDr7H9RA8H8ksZL8qWB3zmhZ2nGCI+4ZVn9YtTKMgOoTiEzTWfv41k7WA4KWJYX7NqziOGzxqegpKOLK2Ec7vQuXq8zlSRW1iWMwyMdQtJK2mP66nWKrgVx5SQjipcJ+fDqcqghGYBj1kWfRV4xSeCQrdk/qIMO6PaLWKqUXSUnBP9HD/0+pDRrCCS/EaKpAs0YiiPu8MfGe3nLEqny4vbUtv61+zOlQ7OeKYWxq1Q== hadoop@clouderaManager01 ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAx2qIKmR564eQK+Lnp59oEvbB4HrBtxRxbTBNViO5tC4yDst1vwuPo23t9BlmMqsbCWSsAkEo6Vj3UA/qwdqKDFK1ssLRgzVG3OsqC7WrlEuSy7V27EpnfEFMkbXBiTrnRcbKNu3wdzgJZ7Mc8Yd0CqZ5c6UyPEzrgBou489fLyzKNSov+uzkahw1l7S77/lh03rbq7YQ/7gEK23tZgofsYFf+k8w6pmDpvl/mSv2156PkWMFQr9yLGen/q6kKsKqg6KT7B06a6zlB6khom+ou5p+dZDiR5AZkI3aNF0brZ7+OTckWjuh2xxgy9ZbRxy/mb6YNwjEl5mTlfFQkkQasQ== hadoop@clouderaManager02 ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAxYvoxWFWM9J/eyoe+O9EfA3wBe4oHQs6PZf7e5sQSoQJV+0gVufezZ+iWxGcL92uTNDE6PcrrfMWPv68Fl5mXsXrhVX8gJA/RKfeVnfLdNYI5AKmqniVq0h+qYv1wz/hOcaq/mNp3dHR3cyf+xjUc+gTcOIvJsJmBEVr5YtsfMCqVzjZZYjChEJ+jiRlf1fIwiogE0M8ncrqGDjS6aPDV3vsceHOspuuXD0APEdxb9YUVlAUXpXNADpmz3P/2V1aHbzNqFZWHcdyUtob7fABWQnuaCJPvaUXNGDBXt7Kv18d0Kbix4q1+o67nkx7B2nAQZT4D3EV3XnZLh6Gvym4Mw== hadoop@clouderaManager03 ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwGScqrw5oMaAj3MLS9wZLkFFLZH8+xXG2f5jM7SNFNcWTO4LAefvrZGS2uaWFPk0PTS+f2dfPX/r0etOFvIBobOmBfrcMByLj954sGFvlD/lU7sPjUFh/OwcXo3UxJyCHnwE4pkVpcmpmyubfzYqKyo8eRMBKLQQx9KlV9MXj1ADe/iwckGXMCG8bzz0/S92S9Y8FxhbTdXhUnVshtn1oJYPr+SxUk4lFed+k6hQah3lXfjd3iJ+uhRfSBkpV9PkSeDyD1h+8ymXNCkZcgcYRXclWziv8FR5f+J2NM5ogvDlGkkcjYKSa4NvCmNsrAPvDEKHnbhV2FbaWgcIC6Tgpw== hadoop@clouderaManager04 [hadoop@clouderaManager01 .ssh]$
然后將中clouderaManager01的authorized_keys文件分發到所有節點上面。
[hadoop@clouderaManager01 .ssh]$ scp -r authorized_keys hadoop@clouderaManager02:~/.ssh/ The authenticity of host 'clouderamanager02 (192.168.80.149)' can't be established. RSA key fingerprint is 1c:56:a2:f4:6e:5b:bc:50:cf:0c:c9:56:59:e6:63:0b. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'clouderamanager02,192.168.80.149' (RSA) to the list of known hosts. hadoop@clouderamanager02's password: authorized_keys 100% 1624 1.6KB/s 00:00 [hadoop@clouderaManager01 .ssh]$ scp -r authorized_keys hadoop@clouderaManager03:~/.ssh/ The authenticity of host 'clouderamanager03 (192.168.80.150)' can't be established. RSA key fingerprint is 59:0d:16:18:d0:2c:39:ff:8e:87:3a:31:92:55:86:f4. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'clouderamanager03,192.168.80.150' (RSA) to the list of known hosts. hadoop@clouderamanager03's password: authorized_keys 100% 1624 1.6KB/s 00:00 [hadoop@clouderaManager01 .ssh]$ scp -r authorized_keys hadoop@clouderaManager04:~/.ssh/ The authenticity of host 'clouderamanager04 (192.168.80.151)' can't be established. RSA key fingerprint is 02:48:fd:f0:15:25:71:a4:ae:89:5d:3d:18:e9:64:b9. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'clouderamanager04,192.168.80.151' (RSA) to the list of known hosts. hadoop@clouderamanager04's password: authorized_keys 100% 1624 1.6KB/s 00:00 [hadoop@clouderaManager01 .ssh]$
大家通過ssh 相互訪問，如果都能無密碼訪問，代表ssh配置成功。這里很簡單，我就不多贅述了。
（13）JDK的安裝（建議1.8）（對於這一步，其實可以不做，因為，在CM時會自動安裝，大家可以跳過）（但是要先卸載linux系統自帶的openjdk，因為自動也是安裝的是orcal公司的）
用文件上傳工具FileZila把jdk1.8的tar包上傳到我們指定的軟件安裝目錄下（比如：/home/hadoop/app或者/usr/local/jdk）,或者使用rz命令上傳也行，但是要切換到root用戶下，或者在hadoop用戶下使用sudo rz
jdk-8u60-linux-x64.tar.gz
注意，先記得。
Centos 6.5下的OPENJDK卸載和SUN的JDK安裝、環境變量配置
[root@clouderaManager01 local]# mkdir jdk
[root@clouderaManager01 local]# chown -R hadoop:hadoop jdk
[hadoop@clouderaManager01 jdk]$ pwd /usr/local/jdk [hadoop@clouderaManager01 jdk]$ ls [hadoop@clouderaManager01 jdk]$ rz [hadoop@clouderaManager01 jdk]$ ll total 176996 -rw-r--r-- 1 hadoop hadoop 181238643 Apr 7 09:17 jdk-8u60-linux-x64.tar.gz [hadoop@clouderaManager01 jdk]$
解壓jdk
刪除jdk壓縮包，並賦予用戶和用戶組。
[hadoop@clouderaManager01 jdk]$ pwd /usr/local/jdk [hadoop@clouderaManager01 jdk]$ ll total 177000 drwxr-xr-x 8 hadoop hadoop 4096 Aug 5 2015 jdk1.8.0_60 -rw-r--r-- 1 hadoop hadoop 181238643 Apr 7 09:17 jdk-8u60-linux-x64.tar.gz [hadoop@clouderaManager01 jdk]$ rm jdk-8u60-linux-x64.tar.gz [hadoop@clouderaManager01 jdk]$ ll total 4 drwxr-xr-x 8 hadoop hadoop 4096 Aug 5 2015 jdk1.8.0_60 [hadoop@clouderaManager01 jdk]$
配置jdk環境變量並使用source命令環境變量生效
#java
export JAVA_HOME=/usr/local/jdk/jdk1.8.0_60 export JRE_HOME=$JAVA_HOME/jre export CLASSPATH=.:$JAVA_HOME/lib:$JRE_HOME/lib export PATH=$PATH:$JAVA_HOME/bin
使用java -version命令查看JDK是否安裝成功
[root@clouderaManager01 jdk]# source /etc/profile
[root@clouderaManager01 jdk]# su hadoop
注意上面操作要在所有節點都執行
（14）以及，我們還可以設置文件打開數量和用於最大進程數
文件打開數量
ulimit -a
[hadoop@clouderaManager01 ~]$ ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 7685 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 10240 cpu time (seconds, -t) unlimited max user processes (-u) 1024 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited [hadoop@clouderaManager01 ~]$
默認是1024，為此，我們需要修改
用戶最大進程數
[hadoop@clouderaManager01 ~]$ ulimit -u 1024 [hadoop@clouderaManager01 ~]$
[hadoop@clouderaManager01 ~]$ sudo vi /etc/security/limits.conf
# /etc/security/limits.conf # #Each line describes a limit for a user in the form: # #<domain> <type> <item> <value> # #Where: #<domain> can be: # - an user name # - a group name, with @group syntax # - the wildcard *, for default entry # - the wildcard %, can be also used with %group syntax, # for maxlogin limit # #<type> can have the two values: # - "soft" for enforcing the soft limits # - "hard" for enforcing hard limits # #<item> can be one of the following: # - core - limits the core file size (KB) # - data - max data size (KB) # - fsize - maximum filesize (KB) # - memlock - max locked-in-memory address space (KB) # - nofile - max number of open files # - rss - max resident set size (KB) # - stack - max stack size (KB) # - cpu - max CPU time (MIN) # - nproc - max number of processes # - as - address space limit (KB) # - maxlogins - max number of logins for this user # - maxsyslogins - max number of logins on the system # - priority - the priority to run user process with # - locks - max number of file locks the user can hold # - sigpending - max number of pending signals # - msgqueue - max memory used by POSIX message queues (bytes) # - nice - max nice priority allowed to raise to values: [-20, 19] # - rtprio - max realtime priority # #<domain> <type> <item> <value> # #* soft core 0 #* hard rss 10000 #@student hard nproc 20 #@faculty soft nproc 20 #@faculty hard nproc 50 #ftp hard nproc 0 #@student - maxlogins 4 # End of file
* soft nofile 65535 * hard nofile 65535 * soft nproc 32000 * hard nproc 32000
同理，現在的主機名是clouderaManager01，拿這個來說，其他的clouderaManager02和clouderaManager03是一樣的。
OK，到此為止我們的准備工作基本就都做好了，接下來我們就先來部署時間服務器和時間客戶端工作。請移步，